HELP. FreeBSD 8.1 polling issue
pyunyh at gmail.com
Thu Aug 26 19:54:00 UTC 2010
On Thu, Aug 26, 2010 at 01:34:45PM +0800, MAI JIN wrote:
> I got a freeBSD 8.1 polling issue on my PC. It is a dual-core Intel
> Pentium x86 PC (2.8GHz each core). The Ethernet interface is Broadcom
> NetXtreme 57xx Gigabit Ethernet interface.
> I set the following options (enable polling and zero-buffer copy) and
> rebuilt the kernel:
> # To make an SMP kernel, the next two lines are needed
> options SMP # Symmetric MultiProcessor
> device apic # I/O APIC
> options DEVICE_POLLING # Open Polling
> options HZ=1000
> options ZERO_COPY_SOCKETS
> The following were appended to the /etc/sysctl.conf
> # increase BPF buffer to 10M
> After installed and rebooted the system, kern.polling.enable was not
> found in MIB so I had to ignore this error. Looks like
> kern.polling.enable is removed from FreeBSD v8.1?
> Everything looked good so build my application to received data from
> another HP server. I wrote the application using libpcap-1.1.1 with BFP
> zero-copy turned on (I found the #define HAVE_ZEROCOPY_BPF 1 in
> config.h). Attached please find the source code of my application.
> Before running the application, I set the following parameters:
> ifconfig bge0 polling # This will turn on the polling of the
> Broadcom driver.
> sysctl -w net.bpf.bufsize=10485760
> sysctl -w net.bpf.maxbufsize=10485760
> sysctl -w kern.polling.idle_poll=1
> sysctl -w kern.polling.burst_max=1000
> sysctl -w kern.polling.each_burst=128
> sysctl -w net.inet.ip.intr_queue_maxlen=256
> Then I ran the application to receive data from the HP server. I ran
> multiple iperf on the HP server to send around 133Mbits/s UDP load to
> the PC under test. The UDP payload size was 47 bytes. The entire IP
> packet size is 76 bytes.
> First of all, the receiving application worked well and received around
> 205K packets/second without packet losing (I checked the receiving
> status using pcap_stats). However, after 2 minutes, the application can
> not received data any more. The packets/second is 0. I ran the ping from
> the PC under test and found that the ping reporting timeout and
> destination unreachable (the ping from HP to the PC also failed). Looked
> like the link between the HP server and PC was broken so the application
> could receive data. No packet was dropped. Then I restart the bge0
> interface using: ifconfig bge0 down && ifconfig bge0 up
> And then I re-ran the application and it continued receiving data. But
> after 1 or 2 minutes, the link broke again. I think it was my
> application that caused the bge0 interface down. I started the tcpdump
> and it worked well without breaking the link.
> I tried to increase the kern.polling.each_burst from 128 to 500 but the
> application would cause the bge0 down within 1 minute. No packet was
> dropped before the link was down.
> I checked the CPU usage of the PC. The sys used is around 90% (might be
> caused by kern.polling.idle_poll=1), user land is 13%.
> I don't understand why the application would break the bge0.
> I tried changing the parameters:
> options HZ=2000
> sysctl -w net.bpf.bufsize=20485760
> sysctl -w net.bpf.maxbufsize=20485760
> sysctl -w kern.polling.idle_poll=1
> sysctl -w kern.polling.burst_max=10000
> sysctl -w kern.polling.each_burst=5000
> The performance was better: I got 307K packet/second (the HP server
> sended around 250Mbits/s, my PC got 200Mbits/s). But after 2 minutes,
> the bge0 was down again.
I'm not a fan of polling(4) especially for intelligent controllers
like bge(4) but it seems bge(4) was dead under high network load.
Would you show me the output of both verbose dmesg and
> Could anybody have a look at this issue? How can <<cap.cpp>> I optimize
> the performance of the polling?
More information about the freebsd-net