ngX connected hosts not receiving replies from non-kernel IP services.

[Zaphod Beeblebrox]

I have a network of computers at home.  The gateway/firewall is FreeBSD 9.2
running mpd5.  The host requesting the service is FreeBSD 9.2.  The
misbehaving host is FreeBSD 10.0p6 running mpd5.  So the details:

ssh is listening (output of netstat -an)

tcp4       0      0 *.22                   *.*                    LISTEN
tcp6       0      0 *.22                   *.*                    LISTEN

named is listening (installed from bind99 port)

tcp4       0      0 xx.yy.30.99.53         *.*                    LISTEN
udp4       0      0 xx.yy.30.99.53         *.*

mpd 5 on the server is up:

[2:35:335]root at owl:~> ifconfig ng29
ng29: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0
mtu 1436
        inet xx.yy.31.6 --> xx.yy.16.50 netmask 0xffffffff
        inet6 fe80::219:b9ff:fef9:b9e7%ng29 prefixlen 64 scopeid 0x23
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

ping works:

[1:71:137]root at virtual:/vr2/backup/nozfs/ox/local-etc> ping xx.yy.16.3
PING xx.yy.16.3 (xx.yy.16.3): 56 data bytes
64 bytes from xx.yy.16.3: icmp_seq=0 ttl=63 time=7.439 ms
64 bytes from xx.yy.16.3: icmp_seq=1 ttl=63 time=6.756 ms

now tcpdumping from the FreeBSD 10.0p6 server host while I ssh:

[2:29:329]root at owl:~> tcpdump -nvi ng29 host xx.yy.16.3
tcpdump: listening on ng29, link-type NULL (BSD loopback), capture size
65535 bytes
capability mode sandbox enabled
18:14:36.276578 IP (tos 0x0, ttl 63, id 3249, offset 0, flags [none], proto
TCP (6), length 60)
    xx.yy.20.52.39218 > xx.yy.16.3.22: Flags [S], cksum 0x4aa1 (correct),
seq 3433340283, win 65535, options [mss 1396,nop,wscale 6,sackOK,TS val
435369805 ecr 0], length 0
18:14:39.290104 IP (tos 0x0, ttl 63, id 4999, offset 0, flags [none], proto
TCP (6), length 60)
    xx.yy.20.52.39218 > xx.yy.16.3.22: Flags [S], cksum 0x3ee9 (correct),
seq 3433340283, win 65535, options [mss 1396,nop,wscale 6,sackOK,TS val
435372805 ecr 0], length 0
18:14:42.502893 IP (tos 0x0, ttl 63, id 6832, offset 0, flags [none], proto
TCP (6), length 60)
    xx.yy.20.52.39218 > xx.yy.16.3.22: Flags [S], cksum 0x3269 (correct),
seq 3433340283, win 65535, options [mss 1396,nop,wscale 6,sackOK,TS val
435376005 ecr 0], length 0

Similarly tcpdumping from the server while running "dig google.ca
@xx.yy.30.99"

[2:37:337]root at owl:~> tcpdump -nvi ng29 host xx.yy.30.99
tcpdump: listening on ng29, link-type NULL (BSD loopback), capture size
65535 bytes
capability mode sandbox enabled
18:36:02.841942 IP (tos 0x0, ttl 63, id 30407, offset 0, flags [none],
proto UDP (17), length 66)
    xx.yy.20.52.27400 > xx.yy.30.99.53: 40608+ [1au] A? google.ca. (38)
18:36:07.838721 IP (tos 0x0, ttl 63, id 33612, offset 0, flags [none],
proto UDP (17), length 66)
    xx.yy.20.52.27400 > xx.yy.30.99.53: 40608+ [1au] A? google.ca. (38)

Frustratingly, ssh and bind work just fine from hosts that are on the lan
with the server.  It's like some portion of the packet routing machinery is
broken with ngX.

Before y'all ask, too, ip.forwarding is 1.  The ng-connected hosts can use
the rest of the internet ... just not non-kernel services on the host that
breaks up their l2tp.