kern/144307: ENOENT set unnecessarily under certain circumstances when malloc is called / fails

[Bruce Evans]

On Mon, 28 Jun 2010, Jaakko Heinonen wrote:

> On 2010-06-28, Garrett Cooper wrote:
> >     Or the malloc(3) call could be fixed with the couple of lines I
> > noted (well, adlibbed of course...
> >
> >     Which I agree with, but shouldn't we fix malloc(3) (and any other
> > function calls that depend on malloc(3) for sensible results)?
>
> It's not required for POSIX compliance at least. Did you actually read
> the quotes from POSIX?
>
> "The value of errno should only be examined when it is indicated to be
> valid by a function's return value."
>
> "The setting of errno after a successful call to a function is
> unspecified unless the description of that function specifies that errno
> shall not be modified."
>
> In other words the value of errno is undefined and shouldn't be
                                        unspecified
> examined unless malloc(3) returns NULL.

Not quite even then.  malloc(0) may return NULL, so errno shouldn't
be examined unless malloc() returns NULL and its arg (when converted
to a size_t) is nonzero.

Maybe more of these bugs could be found by setting errno to EDOOFUS in
malloc() and other commonly used library functions :-).  This is easier
to recognize than say ENOTTY from isatty() in stream initialization on
non-ttys.

Bruce