u-boot/lib/rsa
Heiko Stuebner 49d0ea3746 lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()
When calculating rrtmp/rr rsa_gen_key_prop() tries to make
(((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and
(((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[]
with rlen being num_bits * 2

On a 4096bit key this comes down to to 257 uint32_t elements
in rr and 256 elements in rrtmp but with the current allocation
rr and rrtmp only have 129 uint32_t elements.

On 2048bit keys this works by chance as the defined max_rsa_size=4096
allocates a suitable number of elements, but with an actual 4096bit key
this results in other memory parts getting overwritten.

So as suggested by Heinrich Schuchardt just use the actual bit-size
of the key as base for the size calculation, in turn making the code
compatible to any future keysizes.

Suggested-by: Heinrich Schuchardt <xypron.debian@gmx.de>
Signed-off-by: Heiko Stuebner <heiko.stuebner@theobroma-systems.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

rrtmp needs 2 + (((*prop)->num_bits * 2) >> 5) array elements.

Reviewed-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
2020-07-08 17:21:46 -04:00
..
Kconfig lib: rsa: generate additional parameters for public key 2020-03-12 08:20:39 -04:00
Makefile lib: rsa: take spl/non-spl into account when building rsa_verify_with_pkey() 2020-07-08 17:21:46 -04:00
rsa-checksum.c SPDX: Convert all of our single license tags to Linux Kernel style 2018-05-07 09:34:12 -04:00
rsa-keyprop.c lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop() 2020-07-08 17:21:46 -04:00
rsa-mod-exp.c common: Drop log.h from common header 2020-05-18 21:19:18 -04:00
rsa-sign.c lib: rsa: avoid overriding the object name when already specified 2020-05-15 14:47:35 -04:00
rsa-verify.c lib: rsa: take spl/non-spl into account when building rsa_verify_with_pkey() 2020-07-08 17:21:46 -04:00