inpcb lookups in udp_send -> inpcb laddr and flowtable

[Adrian Chadd]

Hi!

I found this whilst doing UDP RSS testing.

root at adrian-hackbox:/home/adrian/work/freebsd/head/src/sys/net #
sysctl debug.lock.prof.stats | head -2 ; sysctl debug.lock.prof.stats
| sort -nk4 | tail -5
debug.lock.prof.stats:
     max  wait_max       total  wait_total       count    avg wait_avg
cnt_hold cnt_lock name
      40        11      114900       55276      359106      0      0
0  20962 /usr/home/adrian/work/freebsd/head/src/sys/dev/ixgbe/ixgbe.c:1550
(sleep mutex:ix0:tx(6))
      21        11      144253       71959      454169      0      0
0  27506 /usr/home/adrian/work/freebsd/head/src/sys/dev/ixgbe/ixgbe.c:1550
(sleep mutex:ix0:tx(1))
       9        10     3132066       85999    42942422      0      0
0  35943 /usr/home/adrian/work/freebsd/head/src/sys/netinet/udp_usrreq.c:369
(sleep mutex:so_rcv)
     676      1713      805307      169654     7076283      0      0
0  69741 /usr/home/adrian/work/freebsd/head/src/sys/netinet/in_pcb.c:963
(sleep mutex:rtentry)
     674      4426     2111324     6415012     7076284      0      0
0 489745 /usr/home/adrian/work/freebsd/head/src/sys/net/route.c:420
(sleep mutex:rtentry)


then...

root at adrian-hackbox:/home/adrian/git/github/erikarn/freebsd-rss/rss-udp-srv
# dtrace -n 'fbt::rtalloc1_fib:entry { @[stack()] = count(); }'
dtrace: description 'fbt::rtalloc1_fib:entry ' matched 1 probe
^[[A^C


              kernel`rtalloc_ign_fib+0x98
              kernel`in_pcbladdr+0x7d
              kernel`in_pcbconnect_setup+0x1af
              kernel`udp_send+0x4af
              kernel`sosend_dgram+0x30b
              kernel`kern_sendit+0x20b
              kernel`sendit+0x129
              kernel`sys_sendto+0x4d
              kernel`amd64_syscall+0x25a
              kernel`0xffffffff80cd178b
          1848628


It turns out that if a local address isn't bound (ie, it's INADDR_ANY)
then in_pcbladdr() will do a route table lookup to find the interface
to transmit from.

The v4/v6 laddr paths don't know about flowtable; they just do a
routing table lookup.

I'm currently experimenting with a kernel fix to use flowtable here
for IPv4 UDP sockets but I'm not sure about the IPv6 path (the source
interface/address selection is much crazier with all the various scope
bits going on.)

I'd like some help with the IPv6 source selection please!

Thanks!


-a