Serious 'tr' bug, patch for review included

[Tim Robbins]

On Fri, Aug 01, 2003 at 04:44:08AM +0400, Andrey Chernov wrote:

> @@ -208,10 +210,18 @@
>  		if ((func)(cnt))
>  			*p++ = cnt;
>  	*p = OOBCH;
> +	n = p - cp->set;
>  
>  	s->cnt = 0;
> -	s->state = SET;
>  	s->set = cp->set;
> +	if (strcmp(s->str, "upper") == 0)
> +		s->state = SET_UPPER;
> +	else if (strcmp(s->str, "lower") == 0) {
> +		s->state = SET_LOWER;
> +	} else
> +		s->state = SET;
> +	if ((s->state == SET_LOWER || s->state == SET_UPPER) && n > 1)
> +		mergesort(s->set, n, sizeof(*(s->set)), charcoll);
>  }
>  
>  static int

I haven't tested the patch yet, but I don't think it's safe to use
charcoll() to sort "set", which is a char array; charcoll() casts its
arguments to int *, dereferences them, then discards all but the low
8 bits by casting to char. Using charcoll() to sort char arrays may
work on little endian machines, but may not on big endian machines.

Also, watch out for this warning in qsort(3):
     The qsort() and heapsort() functions sort an array of nmemb objects, the
     initial member of which is pointed to by base.  The size of each object
     is specified by size.  Mergesort() behaves similarly, but requires that
     size be greater than ``sizeof(void *) / 2''.
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


Tim