em(4) receive part wedging randomly at moderate load

[Benjamin Rosenblum]

i believe one of the problems that im having is the size of the onboard 
memory.  the new intel gig cards (82547 and higher) only have 40K 
onboard instead of the original 64K onboard.  i have a few servers that 
i built with the older cards that do not seem to have the same 
problems.  unfortunately my network card is onboard which prevents me 
from swaping it with one of the older ones for testing.  i have tried to 
go into the code and manually adjust the buffer sizes since i have read 
on many sites that was a major problem every combination that i use has 
proven fatal.  i have also tried to adjust my mtu size anywhere from 
1500 to 15000 and while different sizes augments the time it takes for 
the crash to occur it still occurs.  smaller mtu sizes seem to take 
longer to crash but the crash is inevitable.  while i do know C very 
well i am not very versed in writing drivers (lack of experience) so my 
attempts to debug this problem are comming up short.  i have heard that 
the drivers in 6.0 have been updated and may infact fix this problem but 
im still very weary of running a beta os on a production computer.  when 
6.0 is released fully ill upgrade to it and see if that fixes the 
problem but until then i would like some sort of fix for this.

Petri Helenius wrote:

> Benjamin Rosenblum wrote:
>
>> the em driver in itself is extremly buggy.  many people, myself 
>> included, are hitting some major problems with this driver that are 
>> causeing some serious issues.  i cant transfer any large files to my 
>> server because the em driver panics and drops the connection for 
>> 15-20 seconds.  its a real pain in the butt when this happens too 
>> cause this is my primary network storage server.  i have had to 
>> resort to the backup systems lately because of this problem.  i think 
>> the entire em network driver needs to get reworked and all these bugs 
>> really need to be taken care of since this is one of the top like 3 
>> network cards used in the field today for gig transfer.
>>
> Does anyone have the programming data for the chipsets so the driver 
> could be taken further? I've been unable to obtain them from Intel 
> despite of repeated attempts.
>
> Pete
>
>> Gleb Smirnoff wrote:
>>
>>>  Colleagues,
>>>
>>>  during last month we are experiencing a nasty problem with em(4)
>>> driver. Several times a day the receive path of the driver wedges
>>> for a minute or two. During wedge the transmit part works with
>>> no problems. The latter fact makes this problem very nasty, because
>>> the problematic router can't be backed up with help of CARP.
>>>
>>> Some details: during the wedge all incoming packets are lost and
>>> counted as "Missed packets". I've checked this using
>>> `sysctl dev.em.0.stats=1`. The `dmesg` output is the following:
>>>
>>> em0: Excessive collisions = 0
>>> em0: Symbol errors = 0
>>> em0: Sequence errors = 0
>>> em0: Defer count = 0
>>> em0: Missed Packets = 1266
>>> em0: Receive No Buffers = 220
>>> em0: Receive length errors = 0
>>> em0: Receive errors = 0
>>> em0: Crc errors = 0
>>> em0: Alignment errors = 0
>>> em0: Carrier extension errors = 0
>>> em0: XON Rcvd = 0
>>> em0: XON Xmtd = 0
>>> em0: XOFF Rcvd = 0
>>> em0: XOFF Xmtd = 0
>>> em0: Good Packets Rcvd = 28347789
>>> em0: Good Packets Xmtd = 30911959
>>>
>>> There is a clear evidence that command `sysctl dev.em.0.stats=1` itself
>>> can trigger the wedge. It is important, that the stats are printed
>>> to a 9600 baud serial console, and this takes about a second. I have
>>> suspicion, that the wedge happens when kernel doesn't service NIC
>>> interrupts for some period of time. Yes, some packets should be lost in
>>> this case, but the wedge must not continue for minutes!
>>>
>>> The box is serving 8 - 15 kpps, 70 - 100 MBps. It runs stateful pf(4)
>>> firewall, with 50k - 80k states. The IP fastforwarding is enabled. The
>>> average state insert/removal ratio is 300 states per second, however
>>> sometimes several thousands of states can be removed in one pass. The
>>> state removal locks the network code for quite a long time, so I guess
>>> that wedge happens exactly when a lot of states are removed. The NIC
>>> interrupts aren't serviced for some time and it wedges.
>>>
>>> The hardware is Supermicro server, with two onboard NICs:
>>> dev.em.0.%pnpinfo: vendor=0x8086 device=0x1075 subvendor=0x8086 
>>> subdevice=0x1075 class=0x020000
>>> dev.em.1.%pnpinfo: vendor=0x8086 device=0x1076 subvendor=0x8086 
>>> subdevice=0x1076 class=0x020000
>>>
>>> The NIC is plugged in Cisco Catalyst 6509 gigabit ethernet port. No
>>> errors are counted on switch port.
>>>
>>> To workaround the problem, I have made the following patch:
>>>
>>> @@ -1650,12 +1651,18 @@
>>>        struct ifnet   *ifp;
>>>        struct adapter * adapter = arg;
>>>        ifp = adapter->ifp;
>>> +       uint64_t        ompc;
>>>
>>>        EM_LOCK(adapter);
>>>
>>>        em_check_for_link(&adapter->hw);
>>>        em_print_link_status(adapter);
>>> -       em_update_stats_counters(adapter);   +       ompc = 
>>> adapter->stats.mpc;
>>> +       em_update_stats_counters(adapter);
>>> +       if (adapter->stats.mpc > ompc) {
>>> +               printf("em watchdog: mpc %lld->%lld\n", ompc, 
>>> adapter->stats.mpc);
>>> +               em_init_locked(adapter);
>>> +       }
>>>        if (em_display_debug_stats && ifp->if_drv_flags & 
>>> IFF_DRV_RUNNING) {
>>>                em_print_hw_stats(adapter);
>>>        }
>>>
>>> It helps to reduce downtime from few minutes to 2 seconds, but this
>>> is very dirty approach to the problem. Sample prints during runtime
>>> with patch:
>>>
>>> em watchdog: mpc 1767->2739
>>> em watchdog: mpc 2739->4724
>>> em watchdog: mpc 4724->7794
>>> em watchdog: mpc 7794->10729
>>>
>>> Every time this is printed, the network wedges for 2 seconds and then
>>> it revives.
>>>
>>> I am asking developers, who work in Intel, to pay attention to this 
>>> problem.
>>>
>>>> From my side I can offer any help in testing and debugging.
>>>
>>>
>>>
>>>  
>>>
>>
>>
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>
>
>
>