kern/103464: jail networking failures to 127.0.0.1 only

[Matt Simerson]

>Number:         103464
>Category:       kern
>Synopsis:       jail networking failures to 127.0.0.1 only
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Sep 22 00:40:21 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator:     Matt Simerson
>Release:        FreeBSD 6.1-RELEASE-p6
>Organization:
tnpi
>Environment:
FreeBSD jail11 6.1-RELEASE-p6 FreeBSD 6.1-RELEASE-p6 #1: Sun Sep 17 19:00:32 CDT 2006     root at jails.cadillac.net:/usr/obj/usr/src/sys/SMP  i386
>Description:
DNS requests sent from a jail to the host (which is running dnscache) fail.

Details:
   The FreeBSD host has two interfaces of concern:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        inet 10.0.1.219 netmask 0xffffff00 broadcast 10.0.1.255
        inet 10.0.1.160 netmask 0xffffffff broadcast 10.0.1.160
        inet 10.0.1.161 netmask 0xffffffff broadcast 10.0.1.161
        <snip>
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
        inet 127.0.0.1 netmask 0xff000000 
        inet 127.0.0.2 netmask 0xffffffff
        inet 127.0.0.3 netmask 0xffffffff
        <snip>

I configured dnscache on 127.0.0.1 and permitted all hosts on the 127 network to access it. DNS queries from host OS work perfectly as expected:

host-os# dig www.freebsd.org. @127.0.0.1
;; ANSWER SECTION:
www.freebsd.org.        3590    IN      A       216.136.204.117

..but queries from a jail running on 127.0.0.2, fail. 

mysql# dig www.freebsd.org. @127.0.0.1

; <<>> DiG 9.3.2 <<>> www.freebsd.org. @127.0.0.1
; (1 server found)
;; connection timed out; no servers could be reached

..so then I tried by creating another jail on one of the 10. addresses. I get exactly the same results, no DNS queries work.

So I moved the DNS resolver from 127.0.0.1 to 10.0.1.219. Once listening on a 10 net address, all jails could resolve queries using it. Then, thinking it was something specific to the loopback interface, moved the resolver to 127.0.0.2, but it still works! So, the only address that causes this problem is the special 127.0.0.1. 

Then, just for grins, I decided to see what was happening to the requests. 10.0.1.161 is the jailed host sending the DNS request.

host-os# tcpdump -i lo0 port 53
19:29:15.021769 IP localhost.cadillac.net.64402 > localhost.cadillac.net.domain:  34780+ PTR? 161.1.0.10.in-addr.arpa. (41)
19:29:15.022086 IP localhost.cadillac.net.domain > localhost.cadillac.net.64402:  34780 NXDomain* 0/0/0 (41)
19:29:19.204934 IP 10.0.1.161.51344 > 10.0.1.161.domain:  40192+ A? www.freebsd.org. (33)
19:29:24.205913 IP 10.0.1.161.51344 > 10.0.1.161.domain:  40192+ A? www.freebsd.org. (33)


 ...and dnscache actually gets the request

2006-09-21 19:29:15.021908500 query 9 7f000001:fb92:87dc 12 161.1.0.10.in-addr.arpa.
19:29:14.204174 IP 10.0.1.161.51344 > 10.0.1.161.domain:  40192+ A? www.freebsd.org. (33)
2006-09-21 19:29:15.022088500 cached nxdomain 161.1.0.10.in-addr.arpa.
2006-09-21 19:29:15.022211500 sent 9 41

  ...but the DNS client never receives the answer. 

So, the request actually does make it from the jail to the host, but when I ran tcpdump on em0 (the interface the jail is on), there is no response going back to the jail. 
>How-To-Repeat:
1. Install FreeBSD 6.1 - std install
2. install a DNS resolver (BIND or dnscache) on 127.0.0.1
3. create a jail on any interface, using any IP on the box
4. log into the jail and attempt to resolve DNS queries using the DNS cache on 127.0.0.1:  (dig www.freebsd.org. @127.0.0.1)
5. witness the failure.

6. Move the DNS cache to any other IP.
7. Witnenss it work.

>Fix:
Fix the code, or document the limitation.

A workaround is not to run services (perhaps only DNS?) on 127.0.0.1 but on another IP, such as 127.0.0.2.
>Release-Note:
>Audit-Trail:
>Unformatted: