Re: Header symbols that shouldn't be visible to ports?

In message <CAOtMX2g0TkQvM6N0yf_fr667XpAFupyY6auf8_8H6VJt7MqEkA@mail.gmail.c
om>
, Alan Somers writes:
> On Sat, Sep 3, 2022 at 11:10 PM Konstantin Belousov <kostikbel@gmail.com> wro
> te:
> >
> > On Sat, Sep 03, 2022 at 10:19:12AM -0600, Alan Somers wrote:
> > > Our /usr/include headers define a lot of symbols that are used by
> > > critical utilities in the base system like ps and ifconfig, but aren't
> > > stable across major releases.  Since they aren't stable, utilities
> > > built for older releases won't run correctly on newer ones.  Would it
> > > make sense to guard these symbols so they can't be used by programs in
> > > the ports tree?  There is some precedent for that, for example
> > > _WANT_SOCKET and _WANT_MNTOPTNAMES.
> > _WANT_SOCKET is clearly about exposing parts of the kernel definitions
> > for userspace code that wants to dig into kernel structures.  Similarly
> > for _WANT_MNTOPTNAMES, but in fact this thing is quite stable.  The
> > definitions are guarded by additional defines not due to their instability,
> > but because using them in userspace requires (much) more preparation from
> > userspace environment, which is either not trivial (_WANT_SOCKET) or
> > contradicts to standartized use of the header (_WANT_MNTOPTNAMES +
> > sys/mount.h).
> >
> > >
> > > I'm particular, I'm thinking about symbols like the following:
> > > MINCORE_SUPER
> > Why this symbol should be hidden?  It is implementation-defined and
> > intended to be exposed to userspace.  All MINCORE_* not only MINCORE_SUPER
> > are under BSD_VISIBLE braces, because POSIX does not define the symbols.
>
> Because it isn't stable.  It changed for example in rev 847ab36bf22
> for 13.0.  Programs using the older value (including virtually every
> Rust program) won't work on 13.0 and later.
>
> >
> > > TDF_*
> > These symbols coming from non-standard header sys/proc.h.  If userspace
> > includes the header, it is already outside any formal standard, and I
> > do not see a reason to make the implementation more convoluted there.
> >
> > > PRI_MAX*
> > > PRI_MIN*
> > > PI_*, PRIBIO, PVFS, etc
> > > IFCAP_*
> > These are all implementation-specific and come from non-standard headers,
> > unless I am mistaken, then please correct me.
> >
> > > RLIM_NLIMITS
> > > IFF_*
> > Same.
> >
> > > *_MAXID
> > This is too broad.
>
> I'm talking about symbols like IPV6CTL_MAXID, which record the size of
> sysctl lists.  Obviously, these symbols can't be stable, and probably
> aren't useful outside of the base system.
>
> >
> > >
> > > Clearly delineating private symbols like this would ease the
> > > maintenance burden on languages that rely on FFI, like Ruby and Rust.
> > > FFI basically assumes that symbols once defined will never change.
> >
> > Why e.g. sys/proc.h is ever consumed by FFI wrappers?
>
> I should add a little detail.  Rust uses FFI to access C functions,
> and #define'd constants are redefined in the Rust bindings.  For most
> Rust programs, the build process doesn't check the contents of
> /usr/include in any way.  Instead, all of that stuff is hard-coded in
> the Rust bindings.  That makes cross-compiling a breeze!  But it does
> cause problems when the C library changes.  Adding a new symbol, like
> copy_file_range, isn't so bad.  If your Rust program doesn't use it,
> then the Rust binding will become an unused symbol and get eliminated
> by the linker.  If your Rust program does use it OTOH, then it will be
> resolved by the dynamic linker at runtime - if you're running on
> FreeBSD 13 or newer.  Otherwise, your program will fail to run.  A
> bigger problem is with symbols that change.  For example, the 64-bit
> inode stuff.  Rust programs still use a FreeBSD 11 ABI (we're working
> on that).  But other symbols change more frequently.  Things like
> PRI_MAX_REALTIME can change between any two releases.  That creates a
> big maintenance burden to keep track of them in the FFI bindings.  And
> they also aren't very useful in cross-compiled programs targeting a
> FreeBSD 11 ABI.  Instead, they really need to have bindings
> automatically generated at build time.  That's possible, but it's not
> the default.

This is exactly what happened with DMD D. When 64-bit statfs was introduced 
all DMD D compiled programs failed to run and recompiling didn't help. The 
DMD upstream failed to understand the problem. Eventually the port had to 
be removed.

>
> So what the Rust community really needs is a way to know which symbols
> will be stable across releases, and which might vary.  Are you
> suggesting that anything from a non-POSIX header file should be
> considered variable?
>

Rust and every other community.


-- 
Cheers,
Cy Schubert <Cy.Schubert@cschubert.com>
FreeBSD UNIX:  <cy@FreeBSD.org>   Web:  http://www.FreeBSD.org
NTP:           <cy@nwtime.org>    Web:  https://nwtime.org

			e^(i*pi)+1=0