dc(1) fails with "big number failure" on 2^64

[Jan Mikkelsen]

Hi,

Great, the test case is very useful.

I have applied the following patch to crypto/bn/bn_word.c, which fixes the problem for me.

--- //depot/vendor/freebsd/9.1-local/src/crypto/openssl/crypto/bn/bn_word.c	2012-08-13 00:32:35.000000000 1000
+++ /data/scratch/janm/p4/freebsd-image-std-2011.1/FreeBSD/src/crypto/openssl/crypto/bn/bn_word.c	2012-08-13 00:32:35.000000000 1000
@@ -145,9 +145,11 @@
 		return(i);
 		}
 	/* Only expand (and risk failing) if it's possibly necessary */
-	if (((BN_ULONG)(a->d[a->top - 1] + 1) == 0) &&
-			(bn_wexpand(a,a->top+1) == NULL))
-		return(0);
+	if (
+		(((a->top == 1) && (BN_MASK2 - w < a->d[0])) ||
+		 ((a->top > 1) && ((BN_ULONG)(a->d[a->top - 1] + 1) == 0))) &&
+		(bn_wexpand(a,a->top+1) == NULL))
+			return(0);
 	i=0;
 	for (;;)
 		{

This is a heap overflow in BN_add_word. I don't know what other problems this could cause beyond bc and dc returning crap ...

Regards,

Jan.



On 09/11/2012, at 6:39 AM, Michiel Boland <michiel at boland.org> wrote:

> On 11/08/2012 19:32, Michiel Boland wrote:
> [...]
>> No fix, but I see a problem in the BN_add_word function in
>> /usr/src/crypto/openssl/crypto/bn/bn_word.c
> 
> Small test case:-
> 
> #include <openssl/bn.h>
> #include <limits.h>
> 
> int main()
> {
>        BIGNUM *n;
> 
>        n = BN_new();
>        BN_set_word(n, ULONG_MAX - 1);
>        BN_add_word(n, 2);
>        BN_free(n);
>        return 0;
> }
> 
> 
> $ gcc x.c -lcrypto
> $ valgrind ./a.out
> ==30682== Memcheck, a memory error detector
> ==30682== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
> ==30682== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
> ==30682== Command: ./a.out
> ==30682==
> ==30682== Invalid write of size 8
> ==30682==    at 0x1328EA8: BN_add_word (bn_word.c:158)
> ==30682==    by 0x40076E: main (in /usr/home/boland/a.out)
> ==30682==  Address 0x18fc0a8 is 0 bytes after a block of size 8 alloc'd
> ==30682==    at 0x100410B: malloc (in /usr/local/lib/valgrind/vgpreload_memcheck-amd64-freebsd.so)
> ==30682==    by 0x1331B82: CRYPTO_malloc (mem.c:328)
> ==30682==    by 0x1330F76: ??? (bn_lib.c:317)
> ==30682==    by 0x13310C7: bn_expand2 (bn_lib.c:432)
> ==30682==    by 0x133121C: BN_set_word (bn_lib.c:570)
> ==30682==    by 0x400760: main (in /usr/home/boland/a.out)
> 
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"