crypto: api - Redo lookup on EEXIST

When two crypto algorithm lookups occur at the same time with
different names for the same algorithm, e.g., ctr(aes-generic)
and ctr(aes), they will both be instantiated.  However, only one
of them can be registered.  The second instantiation will fail
with EEXIST.

Avoid failing the second lookup by making it retry, but only once
because there are tricky names such as gcm_base(ctr(aes),ghash)
that will always fail, despite triggering instantiation and EEXIST.

Reported-by: Ingo Franzki <ifranzki@linux.ibm.com>
Fixes: 2825982d9d66 ("[CRYPTO] api: Added event notification")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Revert "crypto: testmgr - Add hash export format testing"

This reverts commit 18c438b228558e05ede7dccf947a6547516fc0c7.

The s390 hmac and sha3 algorithms are failing the test.  Revert
the change until they have been fixed.

Reported-by: Ingo Franzki <ifranzki@linux.ibm.com>
Link: https://lore.kernel.org/all/623a7fcb-b4cb-48e6-9833-57ad2b32a252@linux.ibm.com/
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: marvell/cesa - Do not chain submitted requests

This driver tries to chain requests together before submitting them
to hardware in order to reduce completion interrupts.

However, it even extends chains that have already been submitted
to hardware.  This is dangerous because there is no way of knowing
whether the hardware has already read the DMA memory in question
or not.

Fix this by splitting the chain list into two.  One for submitted
requests and one for requests that have not yet been submitted.
Only extend the latter.

Reported-by: Klaus Kudielka <klaus.kudielka@gmail.com>
Fixes: 85030c5168f1 ("crypto: marvell - Add support for chaining crypto requests in TDMA mode")
Cc: <stable@vger.kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: powerpc/poly1305 - add depends on BROKEN for now

As discussed in the thread containing
https://lore.kernel.org/linux-crypto/20250510053308.GB505731@sol/, the
Power10-optimized Poly1305 code is currently not safe to call in softirq
context.  Disable it for now.  It can be re-enabled once it is fixed.

Fixes: ba8f8624fde2 ("crypto: poly1305-p10 - Glue code for optmized Poly1305 implementation for ppc64le")
Cc: stable@vger.kernel.org
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Revert "crypto: powerpc/poly1305 - Add SIMD fallback"

This reverts commit c66d7ebbe2fa14e41913adb421090a7426f59786.

Link: https://lore.kernel.org/all/02c22854-eebf-4ad1-b89e-8c2b65ab8236@csgroup.eu/
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: ccp - Add missing tee info reg for teev2

The tee info reg for teev2 is the same as teev1.

Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: ccp - Add missing bootloader info reg for pspv5

The bootloader info reg for pspv5 is the same as pspv4.

Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: sun8i-ce - move fallback ahash_request to the end of the struct

'struct ahash_request' has a flexible array at the end, so it must be the
last member in a struct, to avoid overwriting other struct members.

Therefore, move 'fallback_req' to the end of the 'sun8i_ce_hash_reqctx'
struct.

Fixes: 56f6d5aee88d ("crypto: sun8i-ce - support hash algorithms")
Signed-off-by: Ovidiu Panait <ovidiu.panait.oss@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: octeontx2 - Use dynamic allocated memory region for lmtst

Current driver uses static LMTST region allocated by firmware.
Firmware allocated memory for LMTST is available in PF/VF BAR2.
Using this memory have performance impact as this is mapped as
device memory. There is another option to allocate contiguous
memory at run time and map this in LMT MAP table with the
help of AF driver. With this patch dynamic allocated memory
is used for LMTST.

Also add myself as maintainer for crypto octeontx2 driver

Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: octeontx2 - Initialize cptlfs device info once

Function otx2_cptlf_set_dev_info() initializes common
fields of cptlfs data-struct. This function is called
every time a cptlf is initialized but this needs be done
once for a cptlf block. So this initialization is moved
to early device probe code to avoid redundant initialization.

Signed-off-by: Bharat Bhushan <bbhushan2@marvell.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: xts - Only add ecb if it is not already there

Only add ecb to the cipher name if it isn't already ecb.

Also use memcmp instead of strncmp since these strings are all
stored in an array of length CRYPTO_MAX_ALG_NAME.

Fixes: f1c131b45410 ("crypto: xts - Convert to skcipher")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: lrw - Only add ecb if it is not already there

Only add ecb to the cipher name if it isn't already ecb.

Also use memcmp instead of strncmp since these strings are all
stored in an array of length CRYPTO_MAX_ALG_NAME.

Fixes: 700cb3f5fe75 ("crypto: lrw - Convert to skcipher")
Reported-by: kernel test robot <oliver.sang@intel.com>
Closes: https://lore.kernel.org/oe-lkp/202505151503.d8a6cf10-lkp@intel.com
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: testmgr - Add hash export format testing

Ensure that the hash state can be exported to and imported from
the generic algorithm.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: testmgr - Use ahash for generic tfm

As shash is being phased out, use ahash for the generic tfm.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: hmac - Add ahash support

Add ahash support to hmac so that drivers that can't do hmac in
hardware do not have to implement duplicate copies of hmac.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: testmgr - Ignore EEXIST on shash allocation

Soon hmac will support ahash.  For compatibility hmac still supports
shash so it is possible for two hmac algorithms to be registered at
the same time.  The shash algorithm will have the driver name
"hmac-shash(XXX-driver)".  Due to a quirk in the API, there is no way
to locate the shash algorithm using the name "hmac(XXX-driver)".  It
has to be addressed as either "hmac(XXX)" or "hmac-shash(XXX-driver)".

Looking it up with "hmac(XXX-driver)" will simply trigger the creation
of another instance, and on the second instantiation this will fail
with EEXIST.

Catch the error EEXIST along with ENOENT since it is expected.

If a real shash algorithm came this way, it would be addressed using
the proper name "hmac-shash(XXX-driver)".

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: algapi - Add driver template support to crypto_inst_setname

Add support to crypto_inst_setname for having a driver template
name that differs from the algorithm template name.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: shash - Set reqsize in shash_alg

Make reqsize static for shash algorithms.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: hmac - Add export_core and import_core

Add export_import and import_core so that hmac can be used as a
fallback by block-only drivers.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: hmac - Zero shash desc in setkey

The shash desc needs to be zeroed after use in setkey as it is
not finalised (finalisation automatically zeroes it).

Also remove the final function as it's been superseded by finup.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: ahash - Handle partial blocks in API

Provide an option to handle the partial blocks in the ahash API.
Almost every hash algorithm has a block size and are only able
to hash partial blocks on finalisation.

As a first step disable virtual address support for algorithms
with state sizes larger than HASH_MAX_STATESIZE.  This is OK as
virtual addresses are currently only used on synchronous fallbacks.

This means ahash_do_req_chain only needs to handle synchronous
fallbacks, removing the complexities of saving the request state.

Also move the saved request state into the ahash_request object
as nesting is no longer possible.

Add a scatterlist to ahash_request to store the partial block.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: hash - Add export_core and import_core hooks

Add export_core and import_core hooks.  These are intended to be
used by algorithms which are wrappers around block-only algorithms,
but are not themselves block-only, e.g., hmac.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: hash - Move core export and import into internel/hash.h

The core export and import functions are targeted at implementors
so move them into internal/hash.h.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

dt-bindings: crypto: Convert Marvell CESA to DT schema

Convert the Marvell CESA binding to DT schema format. The
marvell-cesa.txt and mv_cesa.txt are duplicate bindings.

The clock properties are quite varied for each platform hence the
if/then schemas. The old binding was fairly accurate with reality.

The original binding didn't mention there is 1 interrupt per CESA
engine. Based on users, there's a maximum of 2 engines.

Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

dt-bindings: crypto: Convert img,hash-accelerator to DT schema

Convert the Imagination Technologies hardware hash accelerator binding
to DT schema format. It's a straight forward conversion.

Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

dt-bindings: crypto: Convert hisilicon,hip0{6,7}-sec to DT schema

Convert the HiSilicon HIP06/7 Security Accelerator binding to DT schema
format. It's a straight forward conversion.

Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

dt-bindings: crypto: Convert brcm,spum-crypto to DT schema

Convert the Broadcom SPUM/SPU2 binding to DT schema format. It's a
straight forward conversion.

Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

dt-bindings: crypto: Convert axis,artpec6-crypto to DT schema

Convert the Axis Crypto engine binding to DT schema format. It's a
straight forward conversion.

Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

dt-bindings: crypto: Convert amd,ccp-seattle-v1a to DT schema

Convert the AMD Cryptographic Coprocessor binding to DT schema format.
It's a straight forward conversion.

Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

dt-bindings: crypto: Drop obsolete mediatek,eip97-crypto

The mediatek,eip97-crypto binding is half abandoned. The driver was
dropped in 2020 as the Mediatek platforms use InsideSecure block and
the driver for it. All the platforms except MT7623 were updated. A
patch to update it was submitted, but never addressed the review
comments.

Link: https://lore.kernel.org/all/20210303080923.16761-1-vic.wu@mediatek.com/
Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

dt-bindings: crypto: fsl,sec-v4.0: Add fsl,sec-v6.0

The fsl,sec-v6.0 binding is the same as the fsl,sec-v4.0 binding, so add
it to the existing schema and drop the old .txt binding.

The compatibles in the .txt binding don't match the 1 user. Follow the
user for the schema.

Signed-off-by: Rob Herring (Arm) <robh@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: qat - enable reporting of error counters for GEN6 devices

Enable the reporting of error counters through sysfs for QAT GEN6
devices and update the ABI documentation.

This enables the reporting of the following:
   - errors_correctable - hardware correctable errors that allow the
     system to recover without data loss.
   - errors_nonfatal: errors that can be isolated to specific in-flight
     requests.
   - errors_fatal: errors that cannot be contained to a request,
     requiring a Function Level Reset (FLR) upon occurrence.

Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: qat - enable RAS support for GEN6 devices

Enable the reporting and handling of errors for QAT GEN6 devices.

Errors are categorized as correctable, non-fatal, or fatal. Error
handling involves reading the error source registers (ERRSOU0 to ERRSOU3)
to determine the source of the error and then decoding the actual source
reading specific registers.

The action taken depends on the error type:
   - Correctable and Non-Fatal errors. These error are logged, cleared and
     the corresponding counter is incremented.
   - Fatal errors. These errors are logged, cleared and a Function Level
     Reset (FLR) is scheduled.

This reports and handles the following errors:
   - Accelerator engine (AE) correctable errors
   - Accelerator engine (AE) uncorrectable errors
   - Chassis push-pull (CPP) errors
   - Host interface (HI) parity errors
   - Internal memory parity errors
   - Receive interface (RI) errors
   - Transmit interface (TI) errors
   - Interface for system-on-chip (SoC) fabric (IOSF) primary command
     parity errors
   - Shared RAM and slice module (SSM) errors

Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: ccp - Add support for PCI device 0x17D8

Add a new CCP/PSP PCI device ID.

Signed-off-by: John Allen <john.allen@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: marvell/cesa - Avoid empty transfer descriptor

The user may set req->src even if req->nbytes == 0.  If there
is no data to hash from req->src, do not generate an empty TDMA
descriptor.

Fixes: db509a45339f ("crypto: marvell/cesa - add TDMA support")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: marvell/cesa - Handle zero-length skcipher requests

Do not access random memory for zero-length skcipher requests.
Just return 0.

Fixes: f63601fd616a ("crypto: marvell/cesa - add a new driver for Marvell's CESA")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: powerpc/poly1305 - Add SIMD fallback

Add a SIMD fallback path for poly1305-p10 by converting the 2^64
based hash state into a 2^44 base.  In order to ensure that the
generic fallback is actually 2^44, add ARCH_SUPPORTS_INT128 to
powerpc and make poly1305-p10 depend on it.

Fixes: ba8f8624fde2 ("crypto: poly1305-p10 - Glue code for optmized Poly1305 implementation for ppc64le")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: iaa - Optimize rebalance_wq_table()

The function opencodes for_each_cpu() by using a plain for-loop. The
loop calls cpumask_weight() inside the conditional section. Because
cpumask_weight() is O(1), the overall complexity of the function is
O(node * node_cpus^2). Also, cpumask_nth() internally calls hweight(),
which, if not hardware accelerated, is slower than cpumask_next() in
for_each_cpu().

If switched to the dedicated for_each_cpu(), the rebalance_wq_table()
can drop calling cpumask_weight(), together with some housekeeping code.
This makes the overall complexity O(node * node_cpus), or simply speaking
O(nr_cpu_ids).

While there, fix opencoded for_each_possible_cpu() too.

Signed-off-by: Yury Norov <yury.norov@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: qat/qat_6xxx - Fix NULL vs IS_ERR() check in adf_probe()

The pcim_iomap_region() returns error pointers.  It doesn't return NULL
pointers.  Update the check to match.

Fixes: 17fd7514ae68 ("crypto: qat - add qat_6xxx driver")
Signed-off-by: Dan Carpenter <dan.carpenter@linaro.org>
Acked-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: testmgr - enable CRYPTO_MANAGER when CRYPTO_SELFTESTS

crypto/testmgr.c is compiled only when CRYPTO_MANAGER is enabled.  To
make CRYPTO_SELFTESTS work as expected when CRYPTO_MANAGER doesn't get
enabled for another reason, automatically set CRYPTO_MANAGER to the
value of CRYPTO_ALGAPI when CRYPTO_SELFTESTS is enabled.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: Kconfig - make CRYPTO_MANAGER a hidden symbol

There is no reason for people configuring the kernel to be asked about
CRYPTO_MANAGER, so make it a hidden symbol.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: testmgr - rename noextratests to noslowtests

Rename the noextratests module parameter to noslowtests, and replace
other remaining mentions of "extra" in the code with "slow".  This
addresses confusion regarding the word "extra" like that seen at
https://lore.kernel.org/r/6cecf2de-9aa0-f6ea-0c2d-8e974a1a820b@huawei.com/.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: testmgr - make it easier to enable the full set of tests

Currently the full set of crypto self-tests requires
CONFIG_CRYPTO_MANAGER_EXTRA_TESTS=y.  This is problematic in two ways.
First, developers regularly overlook this option.  Second, the
description of the tests as "extra" sometimes gives the impression that
it is not required that all algorithms pass these tests.

Given that the main use case for the crypto self-tests is for
developers, make enabling CONFIG_CRYPTO_SELFTESTS=y just enable the full
set of crypto self-tests by default.

The slow tests can still be disabled by adding the command-line
parameter cryptomgr.noextratests=1, soon to be renamed to
cryptomgr.noslowtests=1.  The only known use case for doing this is for
people trying to use the crypto self-tests to satisfy the FIPS 140-3
pre-operational self-testing requirements when the kernel is being
validated as a FIPS 140-3 cryptographic module.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: testmgr - replace CRYPTO_MANAGER_DISABLE_TESTS with CRYPTO_SELFTESTS

The negative-sense of CRYPTO_MANAGER_DISABLE_TESTS is a longstanding
mistake that regularly causes confusion.  Especially bad is that you can
have CRYPTO=n && CRYPTO_MANAGER_DISABLE_TESTS=n, which is ambiguous.

Replace CRYPTO_MANAGER_DISABLE_TESTS with CRYPTO_SELFTESTS which has the
expected behavior.

The tests continue to be disabled by default.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: testmgr - remove panic_on_fail

The cryptomgr.panic_on_fail=1 kernel command-line parameter is not very
useful now that the tests have been fixed to WARN on failure, since
developers can just use panic_on_warn=1 instead.  There's no need for a
special option just for the crypto self-tests.  Remove it.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: tcrypt - rename CRYPTO_TEST to CRYPTO_BENCHMARK

tcrypt is actually a benchmarking module and not the actual tests.  This
regularly causes confusion.  Update the kconfig option name and help
text accordingly.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> # m68k
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: null - use memcpy_sglist()

Make null_skcipher_crypt() use memcpy_sglist() instead of the
skcipher_walk API, as this is simpler.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: null - merge CRYPTO_NULL2 into CRYPTO_NULL

There is no reason to have separate CRYPTO_NULL2 and CRYPTO_NULL
options.  Just merge them into CRYPTO_NULL.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: null - remove the default null skcipher

crypto_get_default_null_skcipher() and
crypto_put_default_null_skcipher() are no longer used, so remove them.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: krb5enc - do not select CRYPTO_NULL

The krb5enc code does not use any of the so-called "null algorithms", so
it does not need to select CRYPTO_NULL.  Presumably this unused
dependency got copied from one of the other kconfig options.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: geniv - use memcpy_sglist() instead of null skcipher

For copying data between two scatterlists, just use memcpy_sglist()
instead of the so-called "null skcipher".  This is much simpler.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: gcm - use memcpy_sglist() instead of null skcipher

For copying data between two scatterlists, just use memcpy_sglist()
instead of the so-called "null skcipher".  This is much simpler.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: authenc - use memcpy_sglist() instead of null skcipher

For copying data between two scatterlists, just use memcpy_sglist()
instead of the so-called "null skcipher".  This is much simpler.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: algif_aead - use memcpy_sglist() instead of null skcipher

For copying data between two scatterlists, just use memcpy_sglist()
instead of the so-called "null skcipher".  This is much simpler.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: lib/chacha - add array bounds to function prototypes

Add explicit array bounds to the function prototypes for the parameters
that didn't already get handled by the conversion to use chacha_state:

- chacha_block_*():
  Change 'u8 *out' or 'u8 *stream' to u8 out[CHACHA_BLOCK_SIZE].

- hchacha_block_*():
  Change 'u32 *out' or 'u32 *stream' to u32 out[HCHACHA_OUT_WORDS].

- chacha_init():
  Change 'const u32 *key' to 'const u32 key[CHACHA_KEY_WORDS]'.
  Change 'const u8 *iv' to 'const u8 iv[CHACHA_IV_SIZE]'.

No functional changes.  This just makes it clear when fixed-size arrays
are expected.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: lib/chacha - add strongly-typed state zeroization

Now that the ChaCha state matrix is strongly-typed, add a helper
function chacha_zeroize_state() which zeroizes it.  Then convert all
applicable callers to use it instead of direct memzero_explicit.  No
functional changes.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: lib/chacha - use struct assignment to copy state

Use struct assignment instead of memcpy() in lib/crypto/chacha.c where
appropriate.  No functional change.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: lib/chacha - strongly type the ChaCha state

The ChaCha state matrix is 16 32-bit words.  Currently it is represented
in the code as a raw u32 array, or even just a pointer to u32.  This
weak typing is error-prone.  Instead, introduce struct chacha_state:

    struct chacha_state {
            u32 x[16];
    };

Convert all ChaCha and HChaCha functions to use struct chacha_state.
No functional changes.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Acked-by: Kent Overstreet <kent.overstreet@linux.dev>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: crypto4xx - Remove ahash-related code

The hash implementation in crypto4xx has been disabled since 2009.
As nobody has tried to fix this remove all the dead code.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: powerpc/poly1305 - Add poly1305_emit_arch wrapper

Add poly1305_emit_arch with fallback instead of calling assembly
directly.  This is because the state format differs between p10
and that of the generic implementation.

Reported-by: Venkat Rao Bagalkote <venkat88@linux.ibm.com>
Reported-by: Eric Biggers <ebiggers@google.com>
Fixes: 14d31979145d ("crypto: powerpc/poly1305 - Add block-only interface")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: arm64/sha256 - fix build when CONFIG_PREEMPT_VOLUNTARY=y

Fix the build of sha256-ce.S when CONFIG_PREEMPT_VOLUNTARY=y by passing
the correct label to the cond_yield macro.  Also adjust the code to
execute only one branch instruction when CONFIG_PREEMPT_VOLUNTARY=n.

Fixes: 6e36be511d28 ("crypto: arm64/sha256 - implement library instead of shash")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202505071811.yYpLUbav-lkp@intel.com/
Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: powerpc/poly1305 - Add missing poly1305_emit_arch

Rename poly1305_emit_64 to poly1305_emit_arch to conform with
the expectation of the poly1305 library.

Reported-by: Thorsten Leemhuis <linux@leemhuis.info>
Fixes: 14d31979145d ("crypto: powerpc/poly1305 - Add block-only interface")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Tested-by: Thorsten Leemhuis <linux@leemhuis.info>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

um: Include linux/types.h in asm/fpu/api.h

Include linux/types.h before using bool.

Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202505070045.vWc04ygs-lkp@intel.com/
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Acked-by: Johannes Berg <johannes@sipsolutions.net>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: lib/poly1305 - Build main library on LIB_POLY1305 and split generic code out

Split the lib poly1305 code just as was done with sha256.  Make
the main library code conditional on LIB_POLY1305 instead of
LIB_POLY1305_GENERIC.

Reported-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Fixes: 10a6d72ea355 ("crypto: lib/poly1305 - Use block-only interface")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: s390/sha512 - Initialise upper counter to zero for sha384

Initialise the high bit counter to zero in sha384_init.

Also change the state initialisation to use ctx->sha512.state
instead of ctx->state for consistency.

Fixes: 572b5c4682c7 ("crypto: s390/sha512 - Use API partial block handling")
Reported-by: Ingo Franzki <ifranzki@linux.ibm.com>
Reported-by: Harald Freudenberger <freude@linux.ibm.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: padlock-sha - Use core import and export for fallback

As padlock-sha is block-only, it needs to use core import and
export on the fallback.

Also call sha256_block_init instead of sha256_init although this
is harmless as sha256_init doesn't write into the partial block
area.

Fixes: 63dc06cd12f9 ("crypto: padlock-sha - Use API partial block handling")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: ahash - Add HASH_REQUEST_ZERO

Add a helper to zero hash stack requests that were never cloned
off the stack.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: ahash - Add core export and import

Add crypto_ahash_export_core and crypto_ahash_import_core.  For
now they only differ from the normal export/import functions when
going through shash.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: ahash - Enforce MAX_SYNC_HASH_REQSIZE for sync ahash

As sync ahash algorithms (currently there are none) are used without
a fallback, ensure that they obey the MAX_SYNC_HASH_REQSIZE rule
just like shash algorithms.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: shash - Mark shash algorithms as REQ_VIRT

Mark shash algorithms with the REQ_VIRT bit as they can handle
virtual addresses as is.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: shash - Cap state size to HASH_MAX_STATESIZE

Now that all shash algorithms have converted over to the generic
export format, limit the shash state size to HASH_MAX_STATESIZE.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: zynqmp-sha - Fix partial block implementation

The zynqmp-sha partial block was based on an old design of the
partial block API where the leftover calculation was done in the
Crypto API.  As the leftover calculation is now done by the
algorithm, fix this by passing the partial blocks to the fallback.

Also zero the stack descriptors.

Fixes: 201e9ec3b621 ("crypto: zynqmp-sha - Use API partial block handling")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: hisilicon/qm - replace devm_kzalloc with devm_kcalloc

Replace devm_kzalloc that has an internal multiplication with
devm_kcalloc to improve code readability and safety from overflows.

Signed-off-by: Ethan Carter Edwards <ethan@ethancedwards.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: hisilicon/qm - remove sizeof(char)

`sizeof(char)` evaluates to 1. Remove the churn.

Signed-off-by: Ethan Carter Edwards <ethan@ethancedwards.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: s390/hmac - Use API partial block handling

Use the Crypto API partial block handling.

Also switch to the generic export format.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: sha256 - Use the partial block API

Use the shash partial block API by default.  Add a separate set
of lib shash algorithms to preserve testing coverage until lib/sha256
has its own tests.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: lib/sha256 - Use generic block helper

Use the BLOCK_HASH_UPDATE_BLOCKS helper instead of duplicating
partial block handling.

Also remove the unused lib/sha256 force-generic interface.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: x86/sha256 - Add simd block function

Add CRYPTO_ARCH_HAVE_LIB_SHA256_SIMD and a SIMD block function
so that the caller can decide whether to use SIMD.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: riscv/sha256 - Add simd block function

Add CRYPTO_ARCH_HAVE_LIB_SHA256_SIMD and a SIMD block function
so that the caller can decide whether to use SIMD.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: arm64/sha256 - Add simd block function

Add CRYPTO_ARCH_HAVE_LIB_SHA256_SIMD and a SIMD block function
so that the caller can decide whether to use SIMD.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: arm/sha256 - Add simd block function

Add CRYPTO_ARCH_HAVE_LIB_SHA256_SIMD and a SIMD block function
so that the caller can decide whether to use SIMD.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: arch/sha256 - Export block functions as GPL only

Export the block functions as GPL only, there is no reason
to let arbitrary modules use these internal functions.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: sha256 - Use the partial block API for generic

The shash interface already handles partial blocks, use it for
sha224-generic and sha256-generic instead of going through the
lib/sha256 interface.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: lib/sha256 - Add helpers for block-based shash

Add an internal sha256_finup helper and move the finalisation code
from __sha256_final into it.

Also add sha256_choose_blocks and CRYPTO_ARCH_HAVE_LIB_SHA256_SIMD
so that the Crypto API can use the SIMD block function unconditionally.
The Crypto API must not be used in hard IRQs and there is no reason
to have a fallback path for hardirqs.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: sun8i-ce-hash - use pm_runtime_resume_and_get()

Replace pm_runtime_get_sync() usage with pm_runtime_resume_and_get() to
simplify error handling.

This is recommended in the documentation of pm_runtime_get_sync().

Signed-off-by: Ovidiu Panait <ovidiu.panait.oss@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: sun8i-ce - undo runtime PM changes during driver removal

The pm_runtime_use_autosuspend() call must be undone with
pm_runtime_dont_use_autosuspend() at driver exit, but this is not
currently handled in the driver.

To fix this issue and at the same time simplify error handling, switch
to devm_pm_runtime_enable(). It will call both pm_runtime_disable() and
pm_runtime_dont_use_autosuspend() during driver removal.

Fixes: 06f751b61329 ("crypto: allwinner - Add sun8i-ce Crypto Engine")
Signed-off-by: Ovidiu Panait <ovidiu.panait.oss@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: api - Rename CRYPTO_ALG_REQ_CHAIN to CRYPTO_ALG_REQ_VIRT

As chaining has been removed, all that remains of REQ_CHAIN is
just virtual address support.  Rename it before the reintroduction
of batching creates confusion.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: acomp - Clone folios properly

The folios contain references to the request itself so they must
be setup again in the cloned request.

Fixes: 5f3437e9c89e ("crypto: acomp - Simplify folio handling")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

hwrng: rockchip - add support for RK3576's RNG

The Rockchip RK3576 SoC uses a new hardware random number generator IP.
It's also used on the Rockchip RK3562 and the Rockchip RK3528.

It has several modes of operation and self-checking features that are
not implemented here. For starters, it has a DRNG output, which is an
AES-CTR pseudo-random number generator that can be reseeded from the
true entropy regularly.

However, it also allows for access of the true entropy generator
directly. This entropy is generated from an oscillator.

There are several configuration registers which we don't touch here. The
oscillator can be switched between a "CRO" and "STR" oscillator, and the
length of the oscillator can be configured.

The hardware also supports some automatic continuous entropy quality
checking, which is also not implemented in this driver for the time
being.

The output as-is has been deemed sufficient to be useful:

  rngtest: starting FIPS tests...
  rngtest: bits received from input: 20000032
  rngtest: FIPS 140-2 successes: 997
  rngtest: FIPS 140-2 failures: 3
  rngtest: FIPS 140-2(2001-10-10) Monobit: 0
  rngtest: FIPS 140-2(2001-10-10) Poker: 1
  rngtest: FIPS 140-2(2001-10-10) Runs: 1
  rngtest: FIPS 140-2(2001-10-10) Long run: 1
  rngtest: FIPS 140-2(2001-10-10) Continuous run: 0
  rngtest: input channel speed: (min=17.050; avg=1897.272;
           max=19531250.000)Kibits/s
  rngtest: FIPS tests speed: (min=44.773; avg=71.179; max=96.820)Mibits/s
  rngtest: Program run time: 11760715 microseconds
  rngtest: bits received from input: 40000032
  rngtest: FIPS 140-2 successes: 1997
  rngtest: FIPS 140-2 failures: 3
  rngtest: FIPS 140-2(2001-10-10) Monobit: 0
  rngtest: FIPS 140-2(2001-10-10) Poker: 1
  rngtest: FIPS 140-2(2001-10-10) Runs: 1
  rngtest: FIPS 140-2(2001-10-10) Long run: 1
  rngtest: FIPS 140-2(2001-10-10) Continuous run: 0
  rngtest: input channel speed: (min=17.050; avg=1798.618;
           max=19531250.000)Kibits/s
  rngtest: FIPS tests speed: (min=44.773; avg=64.561; max=96.820)Mibits/s
  rngtest: Program run time: 23507723 microseconds

Stretching the entropy can then be left up to Linux's actual entropy
pool.

Signed-off-by: Nicolas Frattaroli <nicolas.frattaroli@collabora.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

dt-bindings: rng: rockchip,rk3588-rng: add rk3576-rng compatible

The RK3576 SoC contains another standalone TRNG implementation. While
the register map and hardware is different, it has the same
clocks/interrupts/resets as the RK3588's TRNG, so can go in the same
binding.

Add the compatible and generalise the title/description of the binding
some more.

Signed-off-by: Nicolas Frattaroli <nicolas.frattaroli@collabora.com>
Acked-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: rng - fix documentation for crypto_rng_alg()

Current documentation states that crypto_rng_alg() returns the cra_name of
the rng algorithm, but it actually returns a 'struct rng_alg' pointer from
a RNG handle.

Update documentation to reflect this.

Signed-off-by: Ovidiu Panait <ovidiu.panait.oss@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: qat - add qat_6xxx driver

Add a new driver, qat_6xxx, to support QAT GEN6 devices.
QAT GEN6 devices are a follow-on generation of GEN4 devices and
differently from the previous generation, they can support all three
services (symmetric, asymmetric, and data compression) concurrently.

In order to have the qat_6xxx driver to reuse some of the GEN4 logic,
a new abstraction layer has been introduced to bridge the two
implementations. This allows to avoid code duplication and to keep the
qat_6xxx driver isolated from the GEN4 logic. This approach has been
used for the PF to VF logic and the HW CSR access logic.

Signed-off-by: Laurent M Coquerel <laurent.m.coquerel@intel.com>
Co-developed-by: George Abraham P <george.abraham.p@intel.com>
Signed-off-by: George Abraham P <george.abraham.p@intel.com>
Co-developed-by: Karthikeyan Gopal <karthikeyan.gopal@intel.com>
Signed-off-by: Karthikeyan Gopal <karthikeyan.gopal@intel.com>
Co-developed-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: qat - add firmware headers for GEN6 devices

Add firmware headers related to compression that define macros for
building the hardware configuration word, along with bitfields related
to algorithm settings.

Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: qat - update firmware api

Update the firmware API to have partial decomp as an argument.
Modify the firmware descriptor to support auto-select best and partial
decompress.
Define the maximal auto-select best value.
Define the mask and bit position for the partial decompress field in the
firmware descriptor.

Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: qat - export adf_init_admin_pm()

Export the function adf_init_admin_pm() as it will be used by the
qat_6xxx driver to send the power management initialization messages
to the firmware.

Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: qat - expose configuration functions

The functions related to compression and crypto configurations were
previously declared static, restricting the visibility to the defining
source file. Remove the static qualifier, allowing it to be used in other
files as needed. This is necessary for sharing this configuration functions
with other QAT generations.

Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: qat - export adf_get_service_mask()

Export the function adf_get_service_mask() as it will be used by the
qat_6xxx driver to configure the device.

Signed-off-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: qat - add GEN6 firmware loader

Add support for the QAT GEN6 devices in the firmware loader.
This includes handling firmware images signed with the RSA 3K and the
XMSS algorithms.

Co-developed-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Signed-off-by: Jack Xu <jack.xu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: qat - refactor FW signing algorithm

The current implementation is designed to support single FW signing
authentication only.
Refactor the implementation to support other FW signing methods.

This does not include any functional change.

Co-developed-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Signed-off-by: Jack Xu <jack.xu@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

crypto: qat - use pr_fmt() in qat uclo.c

Add pr_fmt() to qat uclo.c logging and update the debug and error messages
to utilize it accordingly.

This does not introduce any functional changes.

Signed-off-by: Suman Kumar Chakraborty <suman.kumar.chakraborty@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>