Router on 6.0-stable fails to route tcp packets due to NAT?? malfunction

Oleg Tarasov subscriber at osk.com.ua
Mon Dec 26 03:44:40 PST 2005


Hello, all

SYSTEM DESCRIPTION

I have built a production system based on FreeBSD 6.0-stable. The main
Internet connection is established using mpd 3.18 which is started by
attached script "mpd". It is rcorder'ed similar to ppp-user.

mpd configuration is attached in mpd.conf and mpd.links. Shortly, ng0
is a PPPoE connection on rl1 interface.

By the way user ppp failed to work with PPPoE connection correctly
usually causing "No buffer space available" error which caused all
network connections to stop working. Manual restart of ppp helped but
it is quite unacceptable for production system. I attach ppp.conf

Firewall is configured to manually divert packets to natd. I attach
rc.firewall which was simplifyed to a minimum of functions for test
purposes.

natd is configured using the following config file:
===============================================================
log no
use_sockets yes
same_ports yes
interface ng0
unregistered_only yes
log_ipfw_denied yes
log_denied yes
===============================================================

I attach kernel configuration file used to compile it.

Here is output of ifconfig:
===============================================================
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=8<VLAN_MTU>
        inet 192.168.82.253 netmask 0xffffff00 broadcast 192.168.82.255
        ether 00:30:4f:1c:ed:19
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
rl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=8<VLAN_MTU>
        ether 00:30:4f:1c:ed:17
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
        inet 127.0.0.1 netmask 0xff000000
ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> mtu 1492
        inet my.ip.add.ress --> prov.ip.add.ress netmask 0xffffffff
===============================================================

Here is output of netstat -rn:
===============================================================
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            prov.ip.add.ress   UGS         0   512334    ng0
my.ip.add.ress     lo0                UHS         0     2426    lo0
127.0.0.1          127.0.0.1          UH          0    21881    lo0
192.168.82         link#1             UC          0        0    rl0
192.168.82.253     00:30:4f:1c:ed:19  UHLW        1     1162    lo0
prov.ip.add.ress   my.ip.add.ress     UH          1        0    ng0
===============================================================

Windows client configuration:
===============================================================
inet 192.168.82.111 netmask 255.255.255.0 192.168.82.253
===============================================================

Windows client routing table
===============================================================
          0.0.0.0          0.0.0.0   192.168.82.253  192.168.82.111       30
        127.0.0.0        255.0.0.0        127.0.0.1       127.0.0.1       1
      192.168.0.0    255.255.255.0   192.168.82.204  192.168.82.111       1
     192.168.82.0    255.255.255.0   192.168.82.111  192.168.82.111       30
   192.168.82.111  255.255.255.255        127.0.0.1       127.0.0.1       30
   192.168.82.255  255.255.255.255   192.168.82.111  192.168.82.111       30
        224.0.0.0        240.0.0.0   192.168.82.111  192.168.82.111       30
  255.255.255.255  255.255.255.255   192.168.82.111  192.168.82.111       1
Default gateway:      192.168.82.253
===============================================================

The system has SQUID, mail, ftp systems and usually direct packet
routing was not used so the problem was located after a month of usage
of the system.

PROBLEM DESCRIPTION

I have a number of Windows XP clients in the network which are
configured to use This machine as a default gateway. Any icmp packets
to Internet work quite normal. Web worked normally too but when using
proxy, so packet routing is not used for that.

The problem was first encountered when trying to play online game
which did not use proxy. Later it was confirmed when trying to serf
the Web with usage of proxy turned off.

Problem is that almost all data is not transmitted normally using tcp
connections. For example trying to open www.gnome.org fails completely
but packet flow seems to be normal. The most strange thing is that
this problem occurs only on some clients when other ones work quite
fine!!! From malfunctioning machines some sites can be opened too!!!
Some sites can be opened partitially - some parts like pictures can
fail to open.

You can say - "How can we be sure that you client machines are
configured normally?" - I am system administrator for some years and
have plenty of servers and clients confugured by my hands. Also I have
a production system based on 5.4p5 which is configured similarly to
this one but using kernel ppp for internet connection - but that one
had no problems.

Everything in the LAN works perfectly. Also everything going through
proxy also works fine. Any connection made directly from server has no
problems. This makes me think the problem is in routing or NAT.

For test purposes I have reinstalled my own client machine (which also
has the problem described above) from scratch - no result. I changed
network card, changed IP address - no positive result.

From all above I make a conclusion that possible reason is in the NAT
malfunction. Or I dont know what...

Here is the dump on both interfaces ng0 and rl0 which are Internet and
LAN interfaces. I try to open www.gnome.org and I see this:

tcpdump on ng0
===============================================================
09:55:13.757127 IP (tos 0x0, ttl 127, id 56112, offset 0, flags [DF], proto: TCP (6), length: 48) piramida.com.ua.1140 > window.gnome.org.http: S, cksum 0x2b0b (correct), 687058407:687058407(0) win 16384 <mss 1460,nop,nop,sackOK>
09:55:13.982233 IP (tos 0x0, ttl  47, id 0, offset 0, flags [DF], proto: TCP (6), length: 48) window.gnome.org.http > piramida.com.ua.1140: S, cksum 0x6f48 (correct), 3785163588:3785163588(0) ack 687058408 win 5840 <mss 1460,nop,nop,sackOK>
09:55:13.982616 IP (tos 0x0, ttl 127, id 56115, offset 0, flags [DF], proto: TCP (6), length: 40) piramida.com.ua.1140 > window.gnome.org.http: ., cksum 0x6e6c (correct), ack 1 win 17520
09:55:13.982774 IP (tos 0x0, ttl 127, id 56116, offset 0, flags [DF], proto: TCP (6), length: 322) piramida.com.ua.1140 > window.gnome.org.http: P 1:283(282) ack 1 win 17520
09:55:14.219491 IP (tos 0x0, ttl  47, id 58466, offset 0, flags [DF], proto: TCP (6), length: 40) window.gnome.org.http > piramida.com.ua.1140: ., cksum 0x98a2 (correct), ack 283 win 6432
09:55:59.300589 IP (tos 0x0, ttl 127, id 62999, offset 0, flags [DF], proto: TCP (6), length: 40) piramida.com.ua.1140 > window.gnome.org.http: R, cksum 0xb1be (correct), 283:283(0) ack 1 win 0
09:55:59.417698 IP (tos 0x0, ttl  64, id 36993, offset 0, flags [none], proto: TCP (6), length: 40) 192.168.82.111.1140 > window.gnome.org.http: ., cksum 0x58ec (correct), ack 3785163589 win 0
                                                                ^^^^^^                              ^^^^^^^^^^^^^^^^^^
                                                                !!!!!!                              !!!!!!!!!!!!!!!!!!
===============================================================

tcpdump on rl0
===============================================================
09:55:13.756938 IP (tos 0x0, ttl 128, id 56112, offset 0, flags [DF], proto: TCP (6), length: 48) 192.168.82.111.1140 > window.gnome.org.http: S, cksum 0xd233 (correct), 687058407:687058407(0) win 16384 <mss 1460,nop,nop,sackOK>
09:55:13.982399 IP (tos 0x0, ttl  46, id 0, offset 0, flags [DF], proto: TCP (6), length: 48) window.gnome.org.http > 192.168.82.111.1140: S, cksum 0x1671 (correct), 3785163588:3785163588(0) ack 687058408 win 5840 <mss 1460,nop,nop,sackOK>
09:55:13.982538 IP (tos 0x0, ttl 128, id 56115, offset 0, flags [DF], proto: TCP (6), length: 40) 192.168.82.111.1140 > window.gnome.org.http: ., cksum 0x1595 (correct), ack 1 win 17520
09:55:13.982719 IP (tos 0x0, ttl 128, id 56116, offset 0, flags [DF], proto: TCP (6), length: 322) 192.168.82.111.1140 > window.gnome.org.http: P 1:283(282) ack 1 win 17520
09:55:14.219666 IP (tos 0x0, ttl  46, id 58466, offset 0, flags [DF], proto: TCP (6), length: 40) window.gnome.org.http > 192.168.82.111.1140: ., cksum 0x3fcb (correct), ack 283 win 6432
09:55:59.300444 IP (tos 0x0, ttl 128, id 62999, offset 0, flags [DF], proto: TCP (6), length: 40) 192.168.82.111.1140 > window.gnome.org.http: R, cksum 0x58e7 (correct), 283:283(0) ack 1 win 0
09:55:59.417786 IP (tos 0x0, ttl  64, id 36994, offset 0, flags [none], proto: TCP (6), length: 40) window.gnome.org.http > 192.168.82.111.1140: ., cksum 0x58ec (correct), ack 283 win 0
===============================================================
I am not sure what the hell is happening.
The same problem occurs when trying to connect to ftp server - ftp
commands work fine but when I'm trying to download file and massive
tcp connection forms connection hangs.

I would appriciate any useful information on this topic and
information on how can I debug this more deeply.

-- 
Best regards,
 Oleg Tarasov                          mailto:subscriber at osk.com.ua
-------------- next part --------------
Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
	FreeBSD 6.0-RELEASE #0: Tue Nov 29 15:32:53 EET 2005
	    root at gandalf.piramida.com.ua:/usr/obj/usr/src/sys/PIRAMIDA
	    WARNING: debug.mpsafenet forced to 0 as ipsec requires Giant
	    WARNING: MPSAFE network stack disabled, expect reduced performance.
	    Timecounter "i8254" frequency 1193182 Hz quality 0
	    CPU: Intel(R) Celeron(TM) CPU                1100MHz (1093.90-MHz 686-class CPU)
	      Origin = "GenuineIntel"  Id = 0x6b1  Stepping = 1
	        Features=0x383f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
		real memory  = 402587648 (383 MB)
		avail memory = 384339968 (366 MB)
		npx0: [FAST]
		npx0: <math processor> on motherboard
		npx0: INT 16 interface
		acpi0: <IntelR AWRDACPI> on motherboard
		acpi0: Power Button (fixed)
		pci_link0: <ACPI PCI Link LNKA> irq 9 on acpi0
		pci_link1: <ACPI PCI Link LNKB> irq 11 on acpi0
		pci_link2: <ACPI PCI Link LNKC> irq 11 on acpi0
		pci_link3: <ACPI PCI Link LNKD> irq 5 on acpi0
		pci_link4: <ACPI PCI Link LNKE> irq 0 on acpi0
		pci_link5: <ACPI PCI Link LNKF> irq 0 on acpi0
		pci_link6: <ACPI PCI Link LNK0> irq 0 on acpi0
		pci_link7: <ACPI PCI Link LNK1> irq 11 on acpi0
		Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
		acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
		cpu0: <ACPI CPU> on acpi0
		acpi_throttle0: <ACPI CPU Throttling> on cpu0
		acpi_button0: <Power Button> on acpi0
		acpi_button1: <Sleep Button> on acpi0
		pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
		pci0: <ACPI PCI bus> on pcib0
		agp0: <Intel 82815 (i815 GMCH) host to PCI bridge> mem 0xe8000000-0xebffffff at device 0.0 on pci0
		pcib1: <PCI-PCI bridge> at device 1.0 on pci0
		pci1: <PCI bus> on pcib1
		pci1: <display, VGA> at device 0.0 (no driver attached)
		pcib2: <ACPI PCI-PCI bridge> at device 30.0 on pci0
		pci2: <ACPI PCI bus> on pcib2
		rl0: <RealTek 8139 10/100BaseTX> port 0xc000-0xc0ff mem 0xee000000-0xee0000ff irq 11 at device 2.0 on pci2
		miibus0: <MII bus> on rl0
		rlphy0: <RealTek internal media interface> on miibus0
		rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
		rl0: Ethernet address: 00:30:4f:1c:ed:19
		rl0: [GIANT-LOCKED]
		rl1: <RealTek 8139 10/100BaseTX> port 0xc400-0xc4ff mem 0xee001000-0xee0010ff irq 5 at device 3.0 on pci2
		miibus1: <MII bus> on rl1
		rlphy1: <RealTek internal media interface> on miibus1
		rlphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
		rl1: Ethernet address: 00:30:4f:1c:ed:17
		rl1: [GIANT-LOCKED]
		isab0: <PCI-ISA bridge> at device 31.0 on pci0
		isa0: <ISA bus> on isab0
		atapci0: <Intel ICH2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 31.1 on pci0
		ata0: <ATA channel 0> on atapci0
		ata1: <ATA channel 1> on atapci0
		uhci0: <Intel 82801BA/BAM (ICH2) USB controller USB-A> port 0xd000-0xd01f irq 5 at device 31.2 on pci0
		uhci0: [GIANT-LOCKED]
		usb0: <Intel 82801BA/BAM (ICH2) USB controller USB-A> on uhci0
		usb0: USB revision 1.0
		uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
		uhub0: 2 ports with 2 removable, self powered
		pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
		uhci1: <Intel 82801BA/BAM (ICH2) USB controller USB-B> port 0xd800-0xd81f irq 11 at device 31.4 on pci0
		uhci1: [GIANT-LOCKED]
		usb1: <Intel 82801BA/BAM (ICH2) USB controller USB-B> on uhci1
		usb1: USB revision 1.0
		uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
		uhub1: 2 ports with 2 removable, self powered
		pci0: <multimedia, audio> at device 31.5 (no driver attached)
		acpi_tz0: <Thermal Zone> on acpi0
		fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
		fdc0: [FAST]
		fd0: <1440-KB 3.5" drive> on fdc0 drive 0
		sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
		sio0: type 16550A
		sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
		sio1: type 16550A
		ppc0: <Standard parallel printer port> port 0x378-0x37f irq 7 on acpi0
		ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
		ppbus0: <Parallel port bus> on ppc0
		plip0: <PLIP network interface> on ppbus0
		lpt0: <Printer> on ppbus0
		lpt0: Interrupt-driven port
		ppi0: <Parallel I/O> on ppbus0
		atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
		atkbd0: <AT Keyboard> irq 1 on atkbdc0
		kbd0 at atkbd0
		atkbd0: [GIANT-LOCKED]
		psm0: <PS/2 Mouse> irq 12 on atkbdc0
		psm0: [GIANT-LOCKED]
		psm0: model NetMouse/NetScroll Optical, device ID 0
		pmtimer0 on isa0
		sc0: <System console> at flags 0x100 on isa0
		sc0: VGA <16 virtual consoles, flags=0x300>
		vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
		Timecounter "TSC" frequency 1093902442 Hz quality 800
		Timecounters tick every 1.000 msec
		IPsec: Initialized Security Association Processing.
		ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding enabled, default to accept, logging limited to 300 packets/entry by default
		ad0: 38204MB <SAMSUNG SP0411N TW100-08> at ata0-master UDMA100
		acd0: CDROM <ASUS CD-S520/A4/1.2> at ata1-master UDMA33
		Trying to mount root from ufs:/dev/ad0s1a
		rl1: link state changed to UP
		
-------------- next part --------------
A non-text attachment was scrubbed...
Name: KERNEL
Type: application/octet-stream
Size: 10757 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20051226/347c28b9/KERNEL.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpd
Type: application/octet-stream
Size: 775 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20051226/347c28b9/mpd.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpd.conf
Type: application/octet-stream
Size: 594 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20051226/347c28b9/mpd-0001.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpd.links
Type: application/octet-stream
Size: 154 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20051226/347c28b9/mpd-0002.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ppp.conf
Type: application/octet-stream
Size: 750 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20051226/347c28b9/ppp.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rc.firewall
Type: application/octet-stream
Size: 8150 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20051226/347c28b9/rc.obj


More information about the freebsd-net mailing list