Conventions for system headers

[Bruce Evans]

On Mon, 7 May 2007, Maciej Sobczak wrote:

> David Schultz wrote:
>
>>> I would like to propose a campaign of bringing all FreeBSD system headers 
>>> in line with the guarantee that C99 provides for its standard headers, for 
>>> the benefit and convenience of both developers and port maintainers.
>> 
>> Which headers did you have in mind?
>
> Everything that we consider "system" headers, but <net/if.h> was a motivating 
> example that has led me to investigate the whole issue.
> In general, I would like to see consistent conventions applied to all headers 
> from /usr/include.
>
> Additional note: it looks to me that IEEE Std 1003.1 says that <net/if.h> is 
> enough for its functions to work, which is also consistent with FreeBSD's own 
> man pages, for example:
>
> http://www.freebsd.org/cgi/man.cgi?query=if_nametoindex

That is a bug in if_indextoname.3.  I fixed it in rev.1.2, but the fix
got clobbered in rev.1.8 by overwriting the file with a foreign version.
Rev.1.2 was the result of run a script to check for consistency of
includes in man pages.  Not fixing rev.1.8 is partly the result of not
running (or at least using the output) the script for > 5 years.

> But in reality <net/if.h> is not complete.

Network headers generally have intricate, poorly documented, but very
"BSD-standard" prerequisites (see Stevens for how BSD-standard they are).
I have another #include-checking script which I haven't run for > 7 years
that checks for regressions in historical prerequisites in netorking and
other very standard headers.  It takes about 1000 lines to list the
historical bogus prerequisites, and stopped running because it doesn't
list new bogus prerequisites.

>> This sounds like a laudable
>> goal, but I'd try to limit it to headers where the requisite
>> changes won't be too obscene. For example, a lot of the networking
>> headers have sensible and standard dependencies. (POSIX says that
>> they may include certain headers that they depend on, but we
>> don't, often resulting in a compile-time error; maybe that should
>> be changed.)
>
> Exactly.

The "may" bug in POSIX should certainly be changed :-).

>> Generally try to avoid namespace pollution
>
> Interestingly, namespace pollution cannot get worse by solving this problem - 
> it can even be potentially reduced.

Yes, but reducing it is difficult.  Everything that is used in multiple
headers needs to be declared in a common header, perhaps with underscores
to limit its visibility.

For <net/if.h>, this isn't very difficult -- only struct sockaddr,
struct sockaddr_storage, struct timeval and a few standard scalar types
need to declared in common headers; the sockaddr structs are fairly
simple, and struct timeval and the scalar types are already declared
with underscores in common headers.  The sockaddr structs are nonstandard,
at least in draft POSIX.1-2001, so underscored versions would be needed.

<net/if.h> contains very little in POSIX.1-2001, so it doesn't need
struct sockaddr or have a "may" clause allowing it to be polluted with
<sys/socket.h> or anything else.  OTOH, FreeBSD's <net/if.h> is already
polluted with all the symbols in <sys/time.h>, which is in turn polluted
with all the symbols in <time.h>.  Other nested pollution (at least
when <sys/socket.h> is included and __BSD_VISIBLE is 1) includes all
the symbols in <sys/param.h> and <sys/select.h>.  The pollution with
<sys/time.h> is just to get the declarations of time_t and struct timeval.
This is not permitted by POSIX but has been there since rev.1.1, so
fixing it is not just a matter of using the central headers that declare
__time_t and struct __timeval.

> Consider (again, <net/if.h> is a good example, but let's keep the discussion 
> general) that a user needs to #include header X.h. If it's not complete, then 
> he has to also #include some additional header before:
>
> #include <Y.h>  /* required by X.h */
> #include <X.h>
>
> If the user omits the first #include, his code will not compile.
> Now it is clear that in his program the user gets in the global namespace 
> everything from both X.h and Y.h, even though probably only part of Y.h is 
> needed for X.h to work.

This clarity is the only thing that requiring the application to #include
everything does -- it is clear that _all_ the symbols in Y.h are visible,
while if Y.h is included in X.h, then the programmer has to read the or
remember the documentation to know what is included, and the documentation
has to be correct.  In practice, the documentation is rarely correct, and
never in a form that makes the complete pollution clear (due to nested
includes and intricate namespace rules).

> A simple fix, just to keep the convention that allows arbitrary order, is to 
> #include Y.h from X.h. This way X.h becomes complete - but the namespace 
> pollution does not increase; in fact, the sequence of tokens that the 
> compiler sees is exactly the same. It is just more convenient for the user 
> and consistent with the standards.

It is a bug for standard to allow this, but even POSIX only allows it for
a few cases.

> More complete solution would be of course to refactor common definitions from 
> Y.h and #include them internally from both X.h and Y.h, so that X.h is 
> complete with respect to its own definitions and the additional stuff from 
> Y.h is not necessarily pulled if not needed.
> This solution would require creating more headers in the system, which is 
> probably more intrusive than it needs to be.

This solution is used in almost all cases that have been fixed so far.
It is mainly simpler cases that have been fixed -- most things in C99
(which doesn't permit the nexted includes), things near <sys/types.h>
and some networking headers (sys/socket.h and netinet/in.h...).  There
is still a lot of undocumented pollution but hopefully no nested
includes in the useual case where __BSD_VISIBLE is 1.  Pollution generally
involves going outide of the namespace reserved for a header, and not
ducumenting it involves never mentioning of namespaces in FreeBSD man
pages (unlike in standards).

> I would like to propose to fix the problem the simple way (for the moment) - 
> by just forcibly #include'ing the required headers from their dependants. It 
> will not lead to proliferation of new headers and the namespace pollution 
> will remain the same as it is now.

I think this would result in namespace pollution soon growing much larger,
since it removes a limit on pullution for new interfaces.

Bruce