kern/102795: {SHA1, RIPEMD160}_Update() produce wrong results for large buffers

[Pavel Gorshkov]

>Number:         102795
>Category:       kern
>Synopsis:       {SHA1,RIPEMD160}_Update() produce wrong results for large buffers
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Sep 02 17:00:35 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator:     Pavel Gorshkov
>Release:        6.1-STABLE
>Organization:
>Environment:
FreeBSD localhost 6.1-STABLE FreeBSD 6.1-STABLE #8: Tue May 16 14:22:42 MSD 2006     root at localhost:/usr/obj/usr/src/sys/DESKTOP  i386
>Description:
asm-optimized SHA1_Update() and RIPEMD160_Update() as provided by
libmd.a handle buffers larger than approx. 930M incorrectly
(this is known to happen on 7.0-CURRENT/i386 as well)
>How-To-Repeat:
compile the test program (included at the bottom):

    gcc mdtest.c -o sha1test.shared -lmd
    gcc mdtest.c -o sha1test.static -lmd -static
    gcc -DDO_RMD mdtest.c -o rmd160test.shared -lmd
    gcc -DDO_RMD mdtest.c -o rmd160test.static -lmd -static

create a large test file:

    dd if=/dev/zero bs=32m count=32 of=testfile

run the test program, then verify the results with either openssl or gpg:

    $ ./sha1test.shared testfile
    2a492f15396a6768bcbca016993f4b4c8b0b5307
    $ ./sha1test.static testfile
    0d6ee6083bf8b6368cb80d323e82164e5540e296 <======= WRONG!!!
    $ openssl dgst -sha1 <testfile
    2a492f15396a6768bcbca016993f4b4c8b0b5307
    $ gpg --print-md sha1 <testfile | tr -d ' '
    2A492F15396A6768BCBCA016993F4B4C8B0B5307

    $ ./rmd160test.shared testfile
    eb712b817a49164045b6c7039409e89764bb2b28
    $ ./rmd160test.static testfile
    08890f37f17e739121de6c38c15532c8ff9142a0 <======= WRONG!!!
    $ openssl dgst -ripemd160 <testfile
    eb712b817a49164045b6c7039409e89764bb2b28
    $ gpg --print-md ripemd160 <testfile | tr -d ' '
    EB712B817A49164045B6C7039409E89764BB2B28


/*** mdtest.c ***/
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>

#ifdef DO_RMD
#include <ripemd.h>
RIPEMD160_CTX ctx;
#define MD_INIT     RIPEMD160_Init
#define MD_UPDATE   RIPEMD160_Update
#define MD_FINAL    RIPEMD160_Final
#else
#include <sha.h>
SHA_CTX ctx;
#define MD_INIT     SHA1_Init
#define MD_UPDATE   SHA1_Update
#define MD_FINAL    SHA1_Final
#endif

int main(int argc, char **argv) {
    int fd, i;
    struct stat st;
    unsigned char *buf, digest[20];
    char hexdigest[41];

    if (argc < 2 ||
            stat(argv[1], &st) == -1 ||
            (fd=open(argv[1], O_RDONLY)) == -1 ||
            (buf=mmap(NULL, st.st_size, PROT_READ,
                      MAP_PRIVATE, fd, 0)) == MAP_FAILED)
        return 1;

    MD_INIT(&ctx);
    MD_UPDATE(&ctx, buf, st.st_size);
    MD_FINAL(digest, &ctx);

    for (i = 0; i < 20; ++i)
        sprintf(hexdigest + 2*i, "%02x", digest[i]);
    puts(hexdigest);

    if (st.st_size)
        munmap(buf, st.st_size);
    close(fd);
    return 0;
}
/*** end of mdtest.c ***/

>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted: