Dummynet AQM v0.1- CoDel and FQ-CoDel for FreeBSD's ipfw/dummynet

[Don Lewis]

On 10 Mar, Rasool Al-Saadi wrote:
> 
> 
> On Wednesday, 9 March 2016, Don Lewis wrote:
>> 
>> On 26 Feb, Rasool Al-Saadi wrote:
>> > Dear all,
>> >
>> > I would like to announce that we (myself and Grenville Armitage)
>> > released
>> Dummynet AQM v0.1, which is an independent implementation of CoDel
>> and FQ-CoDel for FreeBSD's ipfw/dummynet framework, based on the IETF
>> CoDel [1] and FQ-CoDel [2] Internet-Drafts.
>> > We prepared patches for FreeBSD11-CURRENT-r295345  and FreeBSD 10.x-
>> RELEASE (10.0, 10.1, 10.2), and a technical report  of our
>> implementation.
>> >
>> > Patches and documentation can be found in:
>> > http://caia.swin.edu.au/freebsd/aqm
>> 
>> Without the patch below, the dummynet module fails to load
>> 
>> # kldload dummynet.ko
>> kldload: can't load dummynet.ko: No such file or directory
>> 
>> and the following is printed to /var/log/messages:
>> 
>> link_elf: symbol sysctl__net_inet_ip_dummynet_children undefined
> 
> Thanks again for testing the patch and for providing feedback.
> 
> It seems that this error (and the compilation error in your previous
> email) appears in i386 versions of FreeBSD.

I was testing on FreeBSD 10.3-PRERELEASE and would have been willing to
bet that the difference in behavior was due to a recent change in
FreeBSD.  After testing, this second bug is also a difference between
i386 and amd64.

Digging into the problem, this line in ip_dn_io.c:

  static SYSCTL_NODE(_net_inet_ip, OID_AUTO, dummynet, CTLFLAG_RW,
  0, "Dummynet");

gets translated into:

  static struct sysctl_oid_list sysctl__net_inet_ip_dummynet_children; static struct sysctl_oid sysctl___net_inet_ip_dummynet = { &sysctl__net_inet_ip_children, { ((void *)0) }, (-1), 1|((0x80000000|0x40000000)), (void*)&sysctl__net_inet_ip_dummynet_children, 0, "dummynet", 0, "N", 0, 0, "Dummynet" }; __asm__(".globl " "__start_set_sysctl_set"); __asm__(".globl " "__stop_set_sysctl_set"); static void const * const __set_sysctl_set_sym_sysctl___net_inet_ip_dummynet __attribute__((__section__("set_" "sysctl_set"))) __attribute__((__used__)) = &sysctl___net_inet_ip_dummynet; _Static_assert((((0x80000000|0x40000000)) & 0xf) == 0 || (((0x80000000|0x40000000)) & 0) == 1, "compile-time assertion failed");

by the C preprocessor, and it shows up in the symbol table of the .o
file (on amd64):

  0000000000000140 b sysctl__net_inet_ip_dummynet_children

where "b" indicates that it is a local symbol in the uninitialized data
section.

The dn_aqm_codel.c and dn_sched_fq_codel.c files also what access to
this SYSCTL_NODE, which they indicated by using:
  SYSCTL_DECL(_net_inet_ip_dummynet);
which gets pre-processed to:
  extern struct sysctl_oid_list sysctl__net_inet_ip_dummynet_children;
which results in an undefined symbol in the compiled .o file:
                   U sysctl__net_inet_ip_dummynet_children

A symbol declared as an extern in one compilation until should not
get linked to a matching symbol defined as a static in another
compilation unit.

Looking at the symbol table for the module dummynet.ko, I see both the
static and undefined versions of the symbol:

00000000000001b0 b sysctl__net_inet_ip_dummynet_children
                 U sysctl__net_inet_ip_dummynet_children

For some reason, the kernel linker on amd64 accepts this module, whereas
on i386 it is (correctly in my opinion) rejected.  I'll try to put
together a simple example that I can use to file a FreeBSD bug report.

With the patch (on i386), the symbol is defined as global in ip_dn_io.o:
  000000b8 B sysctl__net_inet_ip_dummynet_children
and there are no undefined versions of this symbol in the .ko file:
  00010e4c b sysctl__net_inet_ip_dummynet_children
however I don't know why it is no longer a global in the .ko file.