Intel em0: watchdog timeout

Tue Feb 23 00:29:47 UTC 2010

Is your driver static, ie builtin, to the kernel, or do you load/unload it
as a module?
I ask because perhaps we could try a later driver, and being a module makes
that
easier.

Jack

On Mon, Feb 22, 2010 at 3:37 PM, Kirk Davis <kirk.davis at epsb.ca> wrote:

>  OK.  I have the following in /boot/loader.conf (and rebooted)
> hw.em.rxd=1024
> hw.em.txd=1024
>
> Should this be hw.em2.rxd?  Is it set per interface or across all
> interfaces?
>
> nmbcluster=262144
>
> # sysctl dev.em.2.stats=1
> Feb 22 16:29:57 inet-gw kernel: em2: Defer count = 20
> Feb 22 16:29:57 inet-gw kernel: em2: Missed Packets = 119947
> Feb 22 16:29:57 inet-gw kernel: em2: Receive No Buffers = 276762
> Feb 22 16:29:57 inet-gw kernel: em2: Receive Length Errors = 0
> Feb 22 16:29:57 inet-gw kernel: em2: Receive errors = 0
> Feb 22 16:29:57 inet-gw kernel: em2: Crc errors = 0
> Feb 22 16:29:57 inet-gw kernel: em2: Alignment errors = 0
> Feb 22 16:29:57 inet-gw kernel: em2: Collision/Carrier extension errors = 0
> Feb 22 16:29:57 inet-gw kernel: em2: RX overruns = 21
> Feb 22 16:29:57 inet-gw kernel: em2: watchdog timeouts = 47
> Feb 22 16:29:57 inet-gw kernel: em2: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK
> MSIX IRQ = 0
> Feb 22 16:29:57 inet-gw kernel: em2: XON Rcvd = 22
> Feb 22 16:29:57 inet-gw kernel: em2: XON Xmtd = 8349
> Feb 22 16:29:57 inet-gw kernel: em2: XOFF Rcvd = 31
> Feb 22 16:29:57 inet-gw kernel: em2: XOFF Xmtd = 15779
> Feb 22 16:29:57 inet-gw kernel: em2: Good Packets Rcvd = 966101852
> Feb 22 16:29:57 inet-gw kernel: em2: Good Packets Xmtd = 755993237
> Feb 22 16:29:57 inet-gw kernel: em2: TSO Contexts Xmtd = 0
> Feb 22 16:29:57 inet-gw kernel: em2: TSO Contexts Failed = 0
>
> still seeing the watchdog timer and link up/down messages.
>
> Should I try going higher than 1024 on the hw.em.rxd?  I'm not sure the
> next time I can schedule another reboot on this production server.
>
> ---- Kirk
>
>
> *Kirk Davis***
> *Senior Network Analyst, ITS*
> *Edmonton Public Schools*
> *One Kingsway Ave. *
> *Edmonton, Alberta, Canada*
> *T5H 4G9*
> *phone: 1-780-429-8308*
>
>
>  ------------------------------
> *From:* Jack Vogel [mailto:jfvogel at gmail.com]
> *Sent:* Monday, February 22, 2010 3:45 PM
> *To:* Kirk Davis
> *Cc:* Mike Tancsa; freebsd-net at freebsd.org
>
> *Subject:* Re: Intel em0: watchdog timeout
>
> OK, so you are still failing to get mbufs in the RX side, increase the
> nmbcluster
> value, and then what size is your RX ring (number of rx descriptors)?
>
> If you havent already done so, change that to 1024.
>
> I am developing a change in the RX code right now that will help
> this situation, but am doing so in the 10G driver, once its solid there
> I will be backporting it into the 1G drivers, it will make discards
> almost unnecessary.
>
> Jack
>
> On Mon, Feb 22, 2010 at 1:43 PM, Kirk Davis <kirk.davis at epsb.ca> wrote:
>
>>
>>
>> > -----Original Message-----
>> > From: Mike Tancsa [mailto:mike at sentex.net]
>> > Subject: Re: Intel em0: watchdog timeout
>> >
>> > At 03:46 PM 2/22/2010, Kirk Davis wrote:
>> > >Does this need to be done in loader.conf?  It doesn't seem
>> > to take from
>> > >the command line.
>> > ># sysctl dev.em.2.stats=1
>> > >dev.em.2.stats: -1 -> -1
>> > >
>> > ># sysctl dev.em.2.stats
>> > >dev.em.2.stats: -1
>> >
>> > Hi,
>> >          After you issue those commands, the driver will spit out a
>> > lot of useful stats to syslog. It will report something like the
>> > following in /var/log/messages
>> >
>> > Feb 22 16:06:31 offsite kernel: em0: Excessive collisions = 0
>> > Feb 22 16:06:31 offsite kernel: em0: Sequence errors = 0
>> > Feb 22 16:06:31 offsite kernel: em0: Defer count = 0
>> > Feb 22 16:06:31 offsite kernel: em0: Missed Packets = 0
>> > Feb 22 16:06:31 offsite kernel: em0: Receive No Buffers = 0
>> > Feb 22 16:06:31 offsite kernel: em0: Receive Length Errors = 0
>> > Feb 22 16:06:31 offsite kernel: em0: Receive errors = 0
>> > Feb 22 16:06:31 offsite kernel: em0: Crc errors = 0
>> > Feb 22 16:06:31 offsite kernel: em0: Alignment errors = 0
>> > Feb 22 16:06:31 offsite kernel: em0: Collision/Carrier
>> > extension errors = 0
>> > Feb 22 16:06:31 offsite kernel: em0: RX overruns = 0
>> > Feb 22 16:06:31 offsite kernel: em0: watchdog timeouts = 0
>> > Feb 22 16:06:31 offsite kernel: em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0
>> > LINK MSIX IRQ = 0
>> > Feb 22 16:06:31 offsite kernel: em0: XON Rcvd = 0
>> > Feb 22 16:06:31 offsite kernel: em0: XON Xmtd = 0
>> > Feb 22 16:06:31 offsite kernel: em0: XOFF Rcvd = 0
>> > Feb 22 16:06:31 offsite kernel: em0: XOFF Xmtd = 0
>> > Feb 22 16:06:31 offsite kernel: em0: Good Packets Rcvd = 2559032551
>> > Feb 22 16:06:31 offsite kernel: em0: Good Packets Xmtd = 1568751141
>> > Feb 22 16:06:31 offsite kernel: em0: TSO Contexts Xmtd = 0
>> > Feb 22 16:06:31 offsite kernel: em0: TSO Contexts Failed = 0
>>
>> Thanks Mike and Jack.  I don't know why I didn'ty notice the output in
>> /var/log/messages
>>
>> Here is the output for the two interfaces that are causing this issue.
>>
>> Feb 22 13:33:52 inet-gw kernel: em0: Excessive collisions = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: Sequence errors = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: Defer count = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: Missed Packets = 24296
>> Feb 22 13:33:52 inet-gw kernel: em0: Receive No Buffers = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: Receive Length Errors = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: Receive errors = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: Crc errors = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: Alignment errors = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: Collision/Carrier extension errors
>> = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: RX overruns = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: watchdog timeouts = 6
>> Feb 22 13:33:52 inet-gw kernel: em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0
>> LINK MSIX IRQ = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: XON Rcvd = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: XON Xmtd = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: XOFF Rcvd = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: XOFF Xmtd = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: Good Packets Rcvd = 424303810
>> Feb 22 13:33:52 inet-gw kernel: em0: Good Packets Xmtd = 576529136
>> Feb 22 13:33:52 inet-gw kernel: em0: TSO Contexts Xmtd = 0
>> Feb 22 13:33:52 inet-gw kernel: em0: TSO Contexts Failed = 0
>> Feb 22 13:34:12 inet-gw kernel: em2: Excessive collisions = 0
>> Feb 22 13:34:12 inet-gw kernel: em2: Sequence errors = 0
>> Feb 22 13:34:12 inet-gw kernel: em2: Defer count = 20
>> Feb 22 13:34:12 inet-gw kernel: em2: Missed Packets = 68059
>> Feb 22 13:34:12 inet-gw kernel: em2: Receive No Buffers = 275612
>> Feb 22 13:34:12 inet-gw kernel: em2: Receive Length Errors = 0
>> Feb 22 13:34:12 inet-gw kernel: em2: Receive errors = 0
>> Feb 22 13:34:12 inet-gw kernel: em2: Crc errors = 0
>> Feb 22 13:34:12 inet-gw kernel: em2: Alignment errors = 0
>> Feb 22 13:34:12 inet-gw kernel: em2: Collision/Carrier extension errors
>> = 0
>> Feb 22 13:34:12 inet-gw kernel: em2: RX overruns = 17
>> Feb 22 13:34:12 inet-gw kernel: em2: watchdog timeouts = 38
>> Feb 22 13:34:12 inet-gw kernel: em2: RX MSIX IRQ = 0 TX MSIX IRQ = 0
>> LINK MSIX IRQ = 0
>> Feb 22 13:34:12 inet-gw kernel: em2: XON Rcvd = 21
>> Feb 22 13:34:12 inet-gw kernel: em2: XON Xmtd = 8344
>> Feb 22 13:34:12 inet-gw kernel: em2: XOFF Rcvd = 30
>> Feb 22 13:34:12 inet-gw kernel: em2: XOFF Xmtd = 9159
>> Feb 22 13:34:12 inet-gw kernel: em2: Good Packets Rcvd = 713607509
>> Feb 22 13:34:12 inet-gw kernel: em2: Good Packets Xmtd = 569694020
>> Feb 22 13:34:12 inet-gw kernel: em2: TSO Contexts Xmtd = 0
>> Feb 22 13:34:12 inet-gw kernel: em2: TSO Contexts Failed = 0
>> Feb 22 13:35:10 inet-gw kernel: em2: Excessive collisions = 0
>> Feb 22 13:35:10 inet-gw kernel: em2: Sequence errors = 0
>> Feb 22 13:35:10 inet-gw kernel: em2: Defer count = 20
>> Feb 22 13:35:10 inet-gw kernel: em2: Missed Packets = 68059
>> Feb 22 13:35:10 inet-gw kernel: em2: Receive No Buffers = 275612
>> Feb 22 13:35:10 inet-gw kernel: em2: Receive Length Errors = 0
>> Feb 22 13:35:10 inet-gw kernel: em2: Receive errors = 0
>> Feb 22 13:35:10 inet-gw kernel: em2: Crc errors = 0
>> Feb 22 13:35:10 inet-gw kernel: em2: Alignment errors = 0
>> Feb 22 13:35:10 inet-gw kernel: em2: Collision/Carrier extension errors
>> = 0
>> Feb 22 13:35:10 inet-gw kernel: em2: RX overruns = 17
>> Feb 22 13:35:10 inet-gw kernel: em2: watchdog timeouts = 38
>> Feb 22 13:35:10 inet-gw kernel: em2: RX MSIX IRQ = 0 TX MSIX IRQ = 0
>> LINK MSIX IRQ = 0
>> Feb 22 13:35:10 inet-gw kernel: em2: XON Rcvd = 21
>> Feb 22 13:35:10 inet-gw kernel: em2: XON Xmtd = 8344
>> Feb 22 13:35:10 inet-gw kernel: em2: XOFF Rcvd = 30
>> Feb 22 13:35:10 inet-gw kernel: em2: XOFF Xmtd = 9159
>> Feb 22 13:35:10 inet-gw kernel: em2: Good Packets Rcvd = 715555016
>> Feb 22 13:35:10 inet-gw kernel: em2: Good Packets Xmtd = 571157561
>> Feb 22 13:35:10 inet-gw kernel: em2: TSO Contexts Xmtd = 0
>> Feb 22 13:35:10 inet-gw kernel: em2: TSO Contexts Failed = 0
>> Feb 22 13:39:12 inet-gw kernel: em2: Excessive collisions = 0
>> Feb 22 13:39:12 inet-gw kernel: em2: Sequence errors = 0
>> Feb 22 13:39:12 inet-gw kernel: em2: Defer count = 20
>> Feb 22 13:39:12 inet-gw kernel: em2: Missed Packets = 68059
>> Feb 22 13:39:12 inet-gw kernel: em2: Receive No Buffers = 275612
>> Feb 22 13:39:12 inet-gw kernel: em2: Receive Length Errors = 0
>> Feb 22 13:39:12 inet-gw kernel: em2: Receive errors = 0
>> Feb 22 13:39:12 inet-gw kernel: em2: Crc errors = 0
>> Feb 22 13:39:12 inet-gw kernel: em2: Alignment errors = 0
>> Feb 22 13:39:12 inet-gw kernel: em2: Collision/Carrier extension errors
>> = 0
>> Feb 22 13:39:12 inet-gw kernel: em2: RX overruns = 17
>> Feb 22 13:39:12 inet-gw kernel: em2: watchdog timeouts = 38
>> Feb 22 13:39:12 inet-gw kernel: em2: RX MSIX IRQ = 0 TX MSIX IRQ = 0
>> LINK MSIX IRQ = 0
>> Feb 22 13:39:12 inet-gw kernel: em2: XON Rcvd = 21
>> Feb 22 13:39:12 inet-gw kernel: em2: XON Xmtd = 8344
>> Feb 22 13:39:12 inet-gw kernel: em2: XOFF Rcvd = 30
>> Feb 22 13:39:12 inet-gw kernel: em2: XOFF Xmtd = 9159
>> Feb 22 13:39:12 inet-gw kernel: em2: Good Packets Rcvd = 723521981
>> Feb 22 13:39:12 inet-gw kernel: em2: Good Packets Xmtd = 577211431
>> Feb 22 13:39:12 inet-gw kernel: em2: TSO Contexts Xmtd = 0
>> Feb 22 13:39:12 inet-gw kernel: em2: TSO Contexts Failed = 0
>>
>>
>> Can this be the problem? "Receive No Buffers = 275612"
>>
>> ---- Kirk
>> Kirk Davis
>> Senior Network Analyst, ITS
>> Edmonton Public Schools
>> One Kingsway Ave.
>> Edmonton, Alberta, Canada
>> T5H 4G9
>> phone: 1-780-429-8308
>>
>>
>>
>