8.0 network problem
David Warren
davideugenewarren at gmail.com
Mon Jul 5 22:49:48 UTC 2010
Hi all,
As several people have noted, I should have been more specific about
the problem. As far as I can tell, it's limited to the wired portion of my
network involving the em0 interface. Samba and scp transfers to and from
computers on the ral0 interface seem unaffected. Even at the relatively
slow speeds of wireless, those transfers are substantially faster than the
degraded speeds I'm seeing on the wired network. I should also have noted
that the main computer on the wired network is a Windows box, while I use
the wireless with my MacBook. I'm using pf for my firewall on the FreeBSD
box, and having tried a bunch of the suggested tests I'm thinking that's the
most likely culprit. I'm going to try a few things out to test that idea.
However, for completeness, here's are the results of the various tests and
diagnostics that were requested.
My pciconf output:
> pciconf -lvc
nfe0 at pci0:0:20:0: class=0x068000 card=0x816a1043 chip=0x026910de
rev=0xa3 hdr=0x00
vendor = 'Nvidia Corp'
device = 'MCP51 Ethernet Controller (2A34103C)'
class = bridge
cap 01[44] = powerspec 2 supports D0 D1 D2 D3 current D0
em0 at pci0:4:8:0: class=0x020000 card=0x13768086 chip=0x107c8086 rev=0x05
hdr=0x00
vendor = 'Intel Corporation'
device = 'Gigabit Ethernet Controller (Copper) rev 5 (82541PI)'
class = network
subclass = ethernet
cap 01[dc] = powerspec 2 supports D0 D3 current D0
cap 07[e4] = PCI-X supports 2048 burst read, 1 split transaction
ral0 at pci0:4:9:0: class=0x028000 card=0x71281432 chip=0x03011814
rev=0x00 hdr=0x00
vendor = 'Ralink Technology, Corp'
device = 'Edimax 54 MBit WLan 802.11g rt 2500 (b8341462)'
class = network
cap 01[40] = powerspec 2 supports D0 D3 current D0
dmesg isn't reporting anything new when this happens. Disabling rxcsum
and txcsum doesn't seem to have an effect. Here's a quick overview of what
I've tried:
pscp works from .13 to .1 (pscp because I'm using key-based identification).
samba fast then slow from .1 to .13
samba fast then slow from .13 to .1
netcat fast then slow from .13 to .1
netcat not working from .1 to .13 (I need to figure out how to get netcat
input into the Windows box on .13)
While the problem was evident, I collected the following netstat:
> netstat -i
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs
Coll
em0 1500 <Link#1> 00:0e:0c:b7:71:44 4840841 0 3791658
0 0
em0 1500 192.168.0.0 192.168.0.1 5496050 - 3687095
- -
ral0 2290 <Link#2> 00:1f:1f:3f:76:f3 0 0 965271
1156 0
nfe0 1500 <Link#3> 00:01:29:d4:2d:6b 349488 0 83833
0 0
nfe0 1500 173.19.224.0/ 173-19-224-254.cl 3260 - 7725
- -
plip0 1500 <Link#4> 0 0 0
0 0
lo0 16384 <Link#5> 758 0 758
0 0
lo0 16384 fe80:5::1 fe80:5::1 0 - 0
- -
lo0 16384 localhost ::1 0 - 0
- -
lo0 16384 your-net localhost 74 - 758
- -
wlan0 1500 <Link#6> 00:1f:1f:3f:76:f3 873552 64 950672
7 0
wlan0 1500 192.168.1.0 192.168.1.1 142745 - 937938
- -
pflog 33152 <Link#7> 0 0 242
0 0
which appears to be free of errors for em0. I also got the following
vmstat -i:
> vmstat -i
interrupt total rate
irq1: atkbd0 8 0
irq6: fdc0 7 0
irq15: ata1 144 0
irq16: em0 2456436 52
irq17: ral0 3782116 80
irq20: atapci2 217325 4
irq21: hdac0 ohci0 11 0
irq22: nfe0 ehci0 436597 9
irq23: atapci1 206722 4
cpu0: timer 94022466 2000
cpu1: timer 94022098 2000
Total 195143930 4151
And the following netstat:
> sudo netstat -I em0 -indb
Name Mtu Network Address Ipkts Ierrs Ibytes
Opkts Oerrs Obytes Coll Drop
em0 1500 <Link#1> 00:0e:0c:b7:71:44 5023101 0 3770654334
3978181 0 925661555 0 0
em0 1500 192.168.0.0/2 192.168.0.1 5677843 - 4600314279
3873061 - 739695560 - -
After starting a netcat transfer from .13 to .1, .13's system monitor
indicates that transfer over the local (outbound) interface starts out very
fast for a few seconds and then bottoms out. This iostat captures the same
pattern. ad4, ad6, ad8, and ad10 are the four disks sharing the base system
(geom mirror) and ZFS bulk filesystem.
> iostat -n 5 -w 1 -c 30
tty ad4 ad6 ad8
ad10 da0 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps
MB/s KB/t tps MB/s us ni sy in id
1 237 27.06 2 0.06 27.06 2 0.06 27.84 2 0.07 28.29 2
0.06 61.23 0 0.03 0 0 1 0 99
0 671 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0
0.00 0.00 0 0.00 0 0 7 0 93
0 224 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0
0.00 0.00 0 0.00 2 0 70 0 29
0 224 0.00 0 0.00 0.00 0 0.00 64.00 1 0.06 0.00 0
0.00 0.00 0 0.00 2 0 63 0 36
0 224 64.00 1 0.06 0.00 0 0.00 0.00 0 0.00 64.00 1
0.06 0.00 0 0.00 2 0 57 0 41
0 224 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0
0.00 0.00 0 0.00 3 0 55 0 42
0 224 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0
0.00 0.00 0 0.00 2 0 51 0 48
0 224 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0
0.00 0.00 0 0.00 5 0 50 0 45
0 224 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0
0.00 0.00 0 0.00 4 0 57 0 39
0 224 0.50 14 0.01 0.50 15 0.01 0.50 29 0.01 0.00 0
0.00 0.00 0 0.00 1 0 4 0 95
0 224 1.23 87 0.10 0.50 84 0.04 0.50 158 0.08 0.50 5
0.00 0.00 0 0.00 2 0 7 0 91
0 224 3.18 84 0.26 3.03 89 0.26 1.64 160 0.26 18.33 15
0.27 0.00 0 0.00 2 0 12 1 85
0 224 42.66 1167 48.63 42.70 1198 49.97 42.69 1870 77.96 42.66 1238
51.59 0.00 0 0.00 0 0 46 3 51
0 232 37.66 798 29.36 38.95 737 28.01 1.60 99 0.15 38.41 703
26.35 0.00 0 0.00 1 0 20 1 78
0 224 59.19 13 0.75 58.79 12 0.69 64.00 17 1.06 59.23 13
0.75 0.00 0 0.00 4 0 13 0 82
0 224 45.14 11 0.48 40.94 9 0.36 49.18 14 0.67 46.71 12
0.55 0.00 0 0.00 3 0 17 0 79
0 224 64.00 15 0.94 64.00 20 1.25 64.00 27 1.68 64.00 13
0.81 0.00 0 0.00 1 0 13 0 85
0 224 64.00 18 1.12 64.00 16 1.00 64.00 27 1.68 64.00 14
0.87 0.00 0 0.00 3 0 13 0 84
0 224 64.00 14 0.87 64.00 13 0.81 64.00 21 1.31 64.00 16
1.00 0.00 0 0.00 3 0 6 0 91
0 224 64.00 11 0.69 64.00 16 1.00 64.00 20 1.25 64.00 14
0.87 0.00 0 0.00 2 0 11 0 86
0 224 59.77 15 0.87 56.94 9 0.50 60.83 20 1.19 59.77 15
0.87 0.00 0 0.00 2 0 11 0 87
0 224 64.00 12 0.75 64.00 9 0.56 64.00 15 0.94 64.00 9
0.56 0.00 0 0.00 2 0 14 0 83
0 671 64.00 8 0.50 64.00 8 0.50 64.00 15 0.94 64.00 8
0.50 0.00 0 0.00 2 0 13 0 84
0 224 64.00 12 0.75 64.00 13 0.81 64.00 28 1.75 64.00 14
0.87 0.00 0 0.00 3 0 12 0 85
0 224 64.00 15 0.94 64.00 15 0.94 64.00 26 1.62 64.00 13
0.81 0.00 0 0.00 2 0 12 0 85
0 224 59.42 18 1.04 61.38 8 0.48 59.85 20 1.17 57.04 12
0.67 0.00 0 0.00 3 0 10 0 87
0 224 64.00 16 1.00 64.00 15 0.94 64.00 22 1.37 64.00 14
0.87 0.00 0 0.00 1 0 10 0 89
0 224 64.00 14 0.87 64.00 9 0.56 64.00 28 1.75 64.00 15
0.94 0.00 0 0.00 4 0 10 0 86
0 224 64.00 11 0.69 64.00 6 0.37 64.00 23 1.43 64.00 10
0.62 0.00 0 0.00 2 0 9 0 89
0 224 64.00 11 0.69 64.00 8 0.50 64.00 20 1.25 64.00 11
0.69 0.00 0 0.00 2 0 12 0 86
Here's the iostat output during the transfer of a file from the ZFS
volume to an external HDD (da0):
> iostat -n 5 -w 1 -c 30
tty ad4 ad6 ad8
ad10 da0 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps
MB/s KB/t tps MB/s us ni sy in id
1 241 29.20 3 0.08 29.21 3 0.08 30.03 3 0.09 30.35 3
0.08 61.97 1 0.06 0 0 1 0 99
10 2167 64.00 2 0.12 0.00 0 0.00 64.00 1 0.06 0.00 0
0.00 0.00 0 0.00 1 0 7 0 92
0 224 0.00 0 0.00 64.00 2 0.12 0.00 0 0.00 64.00 3
0.19 0.00 0 0.00 2 0 13 0 85
12 961 44.45 144 6.24 43.24 150 6.32 42.81 152 6.34 45.06 143
6.28 62.21 382 23.22 2 0 19 1 78
0 335 45.67 163 7.26 43.89 174 7.44 43.28 175 7.38 46.06 170
7.63 63.07 453 27.91 3 0 32 1 64
0 335 32.70 209 6.66 32.09 213 6.66 32.66 221 7.03 32.85 211
6.75 63.10 398 24.53 6 0 24 2 68
0 335 45.22 167 7.36 43.16 172 7.23 45.68 170 7.57 43.88 176
7.53 63.08 457 28.15 4 0 38 1 57
0 335 45.25 157 6.93 43.95 166 7.11 45.94 163 7.30 43.69 167
7.11 62.88 433 26.61 4 0 35 1 60
0 335 44.04 171 7.34 45.36 174 7.69 46.46 169 7.65 43.66 172
7.32 63.09 462 28.47 2 0 36 3 60
0 335 43.38 179 7.57 44.68 170 7.40 45.68 170 7.57 43.39 178
7.53 63.08 455 28.03 3 0 44 1 52
0 335 43.53 173 7.34 45.01 164 7.19 44.18 169 7.28 44.73 165
7.19 63.10 465 28.66 1 0 24 4 70
0 335 43.26 178 7.50 45.27 172 7.59 43.85 181 7.73 45.40 172
7.61 62.98 468 28.78 7 0 37 1 55
0 335 45.18 170 7.49 44.54 171 7.42 44.66 182 7.92 45.71 175
7.80 62.94 455 27.97 2 0 30 2 66
0 335 44.25 171 7.38 44.52 169 7.33 45.94 174 7.79 43.35 178
7.52 63.07 449 27.66 3 0 37 1 59
0 335 45.77 165 7.36 43.42 171 7.24 45.62 166 7.38 43.86 179
7.65 63.08 458 28.22 4 0 36 1 59
0 349 46.11 99 4.45 43.89 105 4.49 47.30 106 4.89 44.82 109
4.76 63.17 287 17.73 3 0 35 1 61
0 335 64.00 5 0.31 64.00 9 0.56 64.00 8 0.50 64.00 7
0.44 0.00 0 0.00 3 0 29 0 69
0 335 64.00 10 0.62 64.00 3 0.19 64.00 15 0.94 64.00 12
0.75 0.00 0 0.00 3 0 22 0 75
0 335 64.00 2 0.12 64.00 4 0.25 64.00 6 0.37 64.00 6
0.37 0.00 0 0.00 6 0 30 0 64
0 335 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 64.00 1
0.06 0.00 0 0.00 4 0 36 0 60
0 335 64.00 2 0.12 64.00 4 0.25 64.00 10 0.62 64.00 6
0.37 0.00 0 0.00 4 0 20 0 75
and from the external HDD to the ZFS pool:
> iostat -n 5 -w 1 -c 40
tty ad4 ad6 ad8
ad10 da0 cpu
tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps
MB/s KB/t tps MB/s us ni sy in id
1 241 28.86 3 0.07 28.87 3 0.07 29.70 3 0.08 30.03 3
0.08 61.79 1 0.05 0 0 1 0 99
16 2391 64.00 6 0.37 64.00 7 0.44 64.00 8 0.50 64.00 6
0.37 0.00 0 0.00 2 0 14 0 84
8 277 64.00 2 0.12 0.00 0 0.00 64.00 3 0.19 64.00 2
0.12 58.15 13 0.74 6 0 17 0 76
10 957 30.35 139 4.11 31.93 138 4.29 32.62 141 4.48 32.65 138
4.39 63.03 246 15.17 2 0 13 2 84
0 335 7.17 29 0.20 4.82 31 0.15 10.24 33 0.33 11.64 33
0.37 63.04 248 15.30 2 0 30 0 67
0 335 64.00 7 0.44 64.00 5 0.31 64.00 10 0.62 64.00 8
0.50 63.04 249 15.30 5 0 23 0 72
0 335 43.29 317 13.42 43.22 316 13.35 43.47 323 13.73 43.34 321
13.60 63.00 241 14.80 2 0 31 2 65
0 335 64.00 4 0.25 58.79 12 0.69 58.79 12 0.69 58.79 12
0.69 63.04 250 15.36 5 0 27 0 68
0 335 64.00 5 0.31 64.00 5 0.31 64.00 11 0.69 64.00 8
0.50 63.04 249 15.30 3 0 28 0 69
0 335 64.00 13 0.81 64.00 4 0.25 64.00 14 0.87 64.00 11
0.69 63.27 248 15.29 3 0 24 0 73
0 335 64.00 2 0.12 53.58 6 0.31 55.00 7 0.38 55.00 7
0.38 62.78 250 15.30 6 0 26 0 68
0 335 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0
0.00 63.04 248 15.30 6 0 29 0 64
0 335 64.00 2 0.12 64.00 2 0.12 64.00 2 0.12 64.00 1
0.06 63.04 250 15.36 6 0 23 0 71
0 335 48.38 4 0.19 64.00 4 0.25 43.00 3 0.13 48.25 4
0.19 63.04 248 15.30 3 0 32 0 65
0 335 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0
0.00 63.04 250 15.36 5 0 29 0 66
0 335 64.00 1 0.06 64.00 2 0.12 64.00 4 0.25 64.00 3
0.19 63.28 248 15.35 4 0 27 0 68
0 335 64.00 5 0.31 64.00 6 0.37 64.00 6 0.37 64.00 5
0.31 63.04 249 15.30 4 0 29 0 67
0 335 0.00 0 0.00 64.00 1 0.06 64.00 1 0.06 64.00 1
0.06 63.04 250 15.36 2 0 38 0 60
0 335 64.00 1 0.06 0.00 0 0.00 64.00 3 0.19 64.00 2
0.12 62.78 249 15.24 5 0 32 0 62
0 335 21.42 6 0.13 35.61 9 0.31 35.61 9 0.31 32.06 8
0.25 63.04 250 15.36 3 0 34 0 62
0 335 55.07 7 0.38 64.00 1 0.06 56.19 8 0.44 43.17 3
0.13 63.04 248 15.30 6 0 28 0 67
0 335 51.50 10 0.50 64.00 13 0.81 58.32 22 1.25 61.14 22
1.31 63.28 249 15.36 3 0 26 0 71
0 1006 55.91 11 0.60 54.00 15 0.79 59.32 19 1.10 56.50 20
1.10 63.04 250 15.36 4 0 27 0 69
0 335 60.25 10 0.59 61.66 16 0.96 62.46 24 1.46 62.39 23
1.40 63.04 249 15.30 1 0 27 0 72
0 335 57.65 10 0.56 57.65 10 0.56 60.26 17 1.00 60.03 16
0.94 63.04 250 15.36 3 0 24 0 73
0 335 64.00 8 0.50 64.00 7 0.44 64.00 14 0.87 64.00 16
1.00 63.04 249 15.30 3 0 29 0 68
0 335 64.00 10 0.62 64.00 8 0.50 64.00 16 1.00 64.00 15
0.94 62.78 249 15.30 1 0 29 0 70
0 335 64.00 6 0.37 64.00 14 0.87 64.00 21 1.31 64.00 20
1.25 63.28 248 15.35 3 0 25 0 72
0 335 64.00 10 0.62 64.00 8 0.50 64.00 17 1.06 64.00 17
1.06 63.04 250 15.36 1 0 27 0 71
0 335 64.00 9 0.56 64.00 13 0.81 64.00 23 1.43 64.00 22
1.37 63.04 249 15.30 4 0 24 0 71
0 335 64.00 9 0.56 64.00 11 0.69 64.00 20 1.25 64.00 21
1.31 63.04 250 15.36 2 0 25 0 72
0 419 64.00 9 0.56 64.00 7 0.44 64.00 15 0.94 64.00 13
0.81 62.68 197 12.03 3 0 23 0 74
0 335 59.54 14 0.81 64.00 3 0.19 60.32 17 1.00 64.00 18
1.12 0.00 0 0.00 4 0 25 0 71
0 335 44.50 63 2.73 45.35 75 3.31 46.15 76 3.42 44.82 69
3.01 0.00 0 0.00 4 0 48 0 48
0 335 42.74 1560 65.11 42.75 1651 68.93 42.74 2466 102.95 42.74 1634
68.20 0.00 0 0.00 1 0 53 5 40
0 350 42.69 1519 63.32 42.40 1431 59.25 41.90 623 25.48 42.24 1468
60.55 0.00 0 0.00 0 0 27 4 69
0 344 11.20 88 0.96 11.31 67 0.74 18.38 82 1.47 19.02 81
1.50 0.00 0 0.00 3 0 12 0 85
0 335 64.00 7 0.44 64.00 7 0.44 64.00 12 0.75 64.00 11
0.69 0.00 0 0.00 3 0 23 0 73
0 335 64.00 7 0.44 64.00 9 0.56 64.00 14 0.87 64.00 14
0.87 0.00 0 0.00 3 0 23 0 74
0 335 64.00 8 0.50 64.00 6 0.37 64.00 12 0.75 64.00 7
0.44 0.00 0 0.00 2 0 19 1 78
Many thanks for all the ideas, and I'll report back after a bit more
testing.
Dave
On Mon, Jul 5, 2010 at 12:51 AM, Jeremy Chadwick
<freebsd at jdc.parodius.com>wrote:
> On Sun, Jul 04, 2010 at 07:52:11PM -0500, David Warren wrote:
> > I've got a persistent problem with my LAN. I'm running a FreeBSD
> 8.0
> > box as a home server performing the following functions for wired and
> > wireless networks: router; firewall; DHCP server; and file server. For
> what
> > it's worth, I've got ZFS up and running as the main filesystem. The
> > recurring issue is that file transfers from the FreeBSD box to computers
> on
> > the wired network (gigabit) start out fast and then become agonizingly
> > slow. I'm sharing home directories over Samba, and those transfers work
> > briefly and then tail off to a few kilobytes per second. The failure is
> > somewhat predicatable in that it tends to happen once a few hundred
> > megabytes have been transferred.
>
> Your system has 3 different network interfaces on it (em, ral, nfe), but
> you don't tell us across which interface the slow transfers happen. You
> also don't tell us which firewalling stack you're using (ipfw, ipfilter,
> pf). Let us know.
>
> I'm going to make the assumption that based on your "...on the wired
> network (gigabit)..." statement that the transfers are going across the
> em0 interface, but again, I'm not sure.
>
> Relevant interfaces (wlan0 is tied to ral0):
>
> > em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> > options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
> > ether 00:0e:0c:b7:71:44
> > inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
> > media: Ethernet autoselect (1000baseT <full-duplex>)
> > status: active
> > ral0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 2290
> > ether 00:1f:1f:3f:76:f3
> > media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap>
> > status: running
> > nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
> > options=8<VLAN_MTU>
> > ether 00:01:29:d4:2d:6b
> > inet XXX.XXX.XXX.XXX netmask 0xfffffc00 broadcast 255.255.255.255
> > media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>)
> > status: active
> > [...]
> > wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
> > ether 00:1f:1f:3f:76:f3
> > inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
> > media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap>
> > status: running
> > ssid FreeBSD_AP channel 7 (2442 Mhz 11g) bssid 00:1f:1f:3f:76:f3
> > country US authmode WPA privacy MIXED deftxkey 2 TKIP 2:128-bit
> > txpower 0 scanvalid 60 protmode CTS dtimperiod 1 -dfs
>
> First and foremost: is the problem specific to Samba? Can you reproduce
> the problem when using the FTP protocol?
>
> Are there any indications of problems in "dmesg" when the issue is
> happening?
>
> Can you provide output from "vmstat -i" while the problem is happening?
>
> Can you provide output from "pciconf -lvc"? Only interested in the
> sections relevant to the above devices.
>
> Can you provide contents of /etc/make.conf, /etc/sysctl.conf, and
> /boot/loader.conf?
>
> Have you looked at "netstat -I <iface> -indb" output during the slow
> transfers to see if there's any indication of problems, or some sort of
> "common rate" (transfer, etc.)
>
> Does disabling the firewalling stack improve things at all?
>
> Can the slowness be reproduced using benchmarks/netperf or only when
> using something that involves actual disk I/O? (To use netperf you'll
> need two FreeBSD boxes). If only disk I/O, then ZFS analysis might be
> needed (there are some performance adjustments that are often required).
>
> Focusing more on em0:
>
> Have you tried disabling rxcsum and txcsum (using ifconfig) to see if
> there's any improvement? I don't see TSO used by your interface, so
> that should rule out any problems with that feature.
>
> --
> | Jeremy Chadwick jdc at parodius.com |
> | Parodius Networking http://www.parodius.com/ |
> | UNIX Systems Administrator Mountain View, CA, USA |
> | Making life hard for others since 1977. PGP: 4BD6C0CB |
>
>
--
Post-doctoral Fellow
Neurology Department
University of Iowa Hospitals and Clinics
davideugenewarren at gmail.com
More information about the freebsd-stable
mailing list