Intel em0: watchdog timeout

Tue Feb 23 15:18:13 UTC 2010

The driver is static right now. I have another Dell 2950 for backup so
I'm going to split the mirror on the production server and put one of
the drives into the development server. Then I can do the upgrade to 7.2
and test.  I don't think I will be able to get a window to switch out
the production box until this weekend at the earliest.

How far behind is the driver in 7.2-RELEASE with any patches applied?
Would this get me close enough to the latest driver?  I'm not sure if
I'm ready to go to 7.3-RC1 on my production server.

----- Kirk

________________________________

	From: Jack Vogel [mailto:jfvogel at gmail.com] 
	Sent: Monday, February 22, 2010 5:30 PM
	To: Kirk Davis
	Cc: Mike Tancsa; freebsd-net at freebsd.org
	Subject: Re: Intel em0: watchdog timeout

	Is your driver static, ie builtin, to the kernel, or do you
load/unload it as a module?
	I ask because perhaps we could try a later driver, and being a
module makes that
	easier. 

	Jack

	On Mon, Feb 22, 2010 at 3:37 PM, Kirk Davis <kirk.davis at epsb.ca>
wrote:

		OK.  I have the following in /boot/loader.conf (and
rebooted)
		hw.em.rxd=1024
		hw.em.txd=1024

		Should this be hw.em2.rxd?  Is it set per interface or
across all interfaces?

		nmbcluster=262144

		# sysctl dev.em.2.stats=1
		Feb 22 16:29:57 inet-gw kernel: em2: Defer count = 20
		Feb 22 16:29:57 inet-gw kernel: em2: Missed Packets =
119947   
		Feb 22 16:29:57 inet-gw kernel: em2: Receive No Buffers
= 276762
		Feb 22 16:29:57 inet-gw kernel: em2: Receive Length
Errors = 0 
		Feb 22 16:29:57 inet-gw kernel: em2: Receive errors = 0
		Feb 22 16:29:57 inet-gw kernel: em2: Crc errors = 0
		Feb 22 16:29:57 inet-gw kernel: em2: Alignment errors =
0
		Feb 22 16:29:57 inet-gw kernel: em2: Collision/Carrier
extension errors = 0
		Feb 22 16:29:57 inet-gw kernel: em2: RX overruns = 21
		Feb 22 16:29:57 inet-gw kernel: em2: watchdog timeouts =
47
		Feb 22 16:29:57 inet-gw kernel: em2: RX MSIX IRQ = 0 TX
MSIX IRQ = 0 LINK MSIX IRQ = 0
		Feb 22 16:29:57 inet-gw kernel: em2: XON Rcvd = 22
		Feb 22 16:29:57 inet-gw kernel: em2: XON Xmtd = 8349
		Feb 22 16:29:57 inet-gw kernel: em2: XOFF Rcvd = 31
		Feb 22 16:29:57 inet-gw kernel: em2: XOFF Xmtd = 15779
		Feb 22 16:29:57 inet-gw kernel: em2: Good Packets Rcvd =
966101852
		Feb 22 16:29:57 inet-gw kernel: em2: Good Packets Xmtd =
755993237
		Feb 22 16:29:57 inet-gw kernel: em2: TSO Contexts Xmtd =
0
		Feb 22 16:29:57 inet-gw kernel: em2: TSO Contexts Failed
= 0

		still seeing the watchdog timer and link up/down
messages.

		Should I try going higher than 1024 on the hw.em.rxd?
I'm not sure the next time I can schedule another reboot on this
production server.

		---- Kirk

		Kirk Davis 
		Senior Network Analyst, ITS 
		Edmonton Public Schools 
		One Kingsway Ave. 
		Edmonton, Alberta, Canada 
		T5H 4G9 
		phone: 1-780-429-8308 

________________________________

			From: Jack Vogel [mailto:jfvogel at gmail.com] 

			Sent: Monday, February 22, 2010 3:45 PM
			To: Kirk Davis
			Cc: Mike Tancsa; freebsd-net at freebsd.org 

			Subject: Re: Intel em0: watchdog timeout

			OK, so you are still failing to get mbufs in the
RX side, increase the nmbcluster
			value, and then what size is your RX ring
(number of rx descriptors)?

			If you havent already done so, change that to
1024. 

			I am developing a change in the RX code right
now that will help
			this situation, but am doing so in the 10G
driver, once its solid there
			I will be backporting it into the 1G drivers, it
will make discards
			almost unnecessary.

			Jack

			On Mon, Feb 22, 2010 at 1:43 PM, Kirk Davis
<kirk.davis at epsb.ca> wrote:

				> -----Original Message-----
				> From: Mike Tancsa
[mailto:mike at sentex.net]
				> Subject: Re: Intel em0: watchdog
timeout
				>
				> At 03:46 PM 2/22/2010, Kirk Davis
wrote:
				> >Does this need to be done in
loader.conf?  It doesn't seem
				> to take from
				> >the command line.
				> ># sysctl dev.em.2.stats=1
				> >dev.em.2.stats: -1 -> -1
				> >
				> ># sysctl dev.em.2.stats
				> >dev.em.2.stats: -1
				>
				> Hi,
				>          After you issue those
commands, the driver will spit out a
				> lot of useful stats to syslog. It will
report something like the
				> following in /var/log/messages
				>
				> Feb 22 16:06:31 offsite kernel: em0:
Excessive collisions = 0
				> Feb 22 16:06:31 offsite kernel: em0:
Sequence errors = 0
				> Feb 22 16:06:31 offsite kernel: em0:
Defer count = 0
				> Feb 22 16:06:31 offsite kernel: em0:
Missed Packets = 0
				> Feb 22 16:06:31 offsite kernel: em0:
Receive No Buffers = 0
				> Feb 22 16:06:31 offsite kernel: em0:
Receive Length Errors = 0
				> Feb 22 16:06:31 offsite kernel: em0:
Receive errors = 0
				> Feb 22 16:06:31 offsite kernel: em0:
Crc errors = 0
				> Feb 22 16:06:31 offsite kernel: em0:
Alignment errors = 0
				> Feb 22 16:06:31 offsite kernel: em0:
Collision/Carrier
				> extension errors = 0
				> Feb 22 16:06:31 offsite kernel: em0:
RX overruns = 0
				> Feb 22 16:06:31 offsite kernel: em0:
watchdog timeouts = 0
				> Feb 22 16:06:31 offsite kernel: em0:
RX MSIX IRQ = 0 TX MSIX IRQ = 0
				> LINK MSIX IRQ = 0
				> Feb 22 16:06:31 offsite kernel: em0:
XON Rcvd = 0
				> Feb 22 16:06:31 offsite kernel: em0:
XON Xmtd = 0
				> Feb 22 16:06:31 offsite kernel: em0:
XOFF Rcvd = 0
				> Feb 22 16:06:31 offsite kernel: em0:
XOFF Xmtd = 0
				> Feb 22 16:06:31 offsite kernel: em0:
Good Packets Rcvd = 2559032551
				> Feb 22 16:06:31 offsite kernel: em0:
Good Packets Xmtd = 1568751141
				> Feb 22 16:06:31 offsite kernel: em0:
TSO Contexts Xmtd = 0
				> Feb 22 16:06:31 offsite kernel: em0:
TSO Contexts Failed = 0

				Thanks Mike and Jack.  I don't know why
I didn'ty notice the output in
				/var/log/messages

				Here is the output for the two
interfaces that are causing this issue.

				Feb 22 13:33:52 inet-gw kernel: em0:
Excessive collisions = 0
				Feb 22 13:33:52 inet-gw kernel: em0:
Sequence errors = 0
				Feb 22 13:33:52 inet-gw kernel: em0:
Defer count = 0
				Feb 22 13:33:52 inet-gw kernel: em0:
Missed Packets = 24296
				Feb 22 13:33:52 inet-gw kernel: em0:
Receive No Buffers = 0
				Feb 22 13:33:52 inet-gw kernel: em0:
Receive Length Errors = 0
				Feb 22 13:33:52 inet-gw kernel: em0:
Receive errors = 0
				Feb 22 13:33:52 inet-gw kernel: em0: Crc
errors = 0
				Feb 22 13:33:52 inet-gw kernel: em0:
Alignment errors = 0
				Feb 22 13:33:52 inet-gw kernel: em0:
Collision/Carrier extension errors
				= 0
				Feb 22 13:33:52 inet-gw kernel: em0: RX
overruns = 0
				Feb 22 13:33:52 inet-gw kernel: em0:
watchdog timeouts = 6
				Feb 22 13:33:52 inet-gw kernel: em0: RX
MSIX IRQ = 0 TX MSIX IRQ = 0
				LINK MSIX IRQ = 0
				Feb 22 13:33:52 inet-gw kernel: em0: XON
Rcvd = 0
				Feb 22 13:33:52 inet-gw kernel: em0: XON
Xmtd = 0
				Feb 22 13:33:52 inet-gw kernel: em0:
XOFF Rcvd = 0
				Feb 22 13:33:52 inet-gw kernel: em0:
XOFF Xmtd = 0
				Feb 22 13:33:52 inet-gw kernel: em0:
Good Packets Rcvd = 424303810
				Feb 22 13:33:52 inet-gw kernel: em0:
Good Packets Xmtd = 576529136
				Feb 22 13:33:52 inet-gw kernel: em0: TSO
Contexts Xmtd = 0
				Feb 22 13:33:52 inet-gw kernel: em0: TSO
Contexts Failed = 0
				Feb 22 13:34:12 inet-gw kernel: em2:
Excessive collisions = 0
				Feb 22 13:34:12 inet-gw kernel: em2:
Sequence errors = 0
				Feb 22 13:34:12 inet-gw kernel: em2:
Defer count = 20
				Feb 22 13:34:12 inet-gw kernel: em2:
Missed Packets = 68059
				Feb 22 13:34:12 inet-gw kernel: em2:
Receive No Buffers = 275612
				Feb 22 13:34:12 inet-gw kernel: em2:
Receive Length Errors = 0
				Feb 22 13:34:12 inet-gw kernel: em2:
Receive errors = 0
				Feb 22 13:34:12 inet-gw kernel: em2: Crc
errors = 0
				Feb 22 13:34:12 inet-gw kernel: em2:
Alignment errors = 0
				Feb 22 13:34:12 inet-gw kernel: em2:
Collision/Carrier extension errors
				= 0
				Feb 22 13:34:12 inet-gw kernel: em2: RX
overruns = 17
				Feb 22 13:34:12 inet-gw kernel: em2:
watchdog timeouts = 38
				Feb 22 13:34:12 inet-gw kernel: em2: RX
MSIX IRQ = 0 TX MSIX IRQ = 0
				LINK MSIX IRQ = 0
				Feb 22 13:34:12 inet-gw kernel: em2: XON
Rcvd = 21
				Feb 22 13:34:12 inet-gw kernel: em2: XON
Xmtd = 8344
				Feb 22 13:34:12 inet-gw kernel: em2:
XOFF Rcvd = 30
				Feb 22 13:34:12 inet-gw kernel: em2:
XOFF Xmtd = 9159
				Feb 22 13:34:12 inet-gw kernel: em2:
Good Packets Rcvd = 713607509
				Feb 22 13:34:12 inet-gw kernel: em2:
Good Packets Xmtd = 569694020
				Feb 22 13:34:12 inet-gw kernel: em2: TSO
Contexts Xmtd = 0
				Feb 22 13:34:12 inet-gw kernel: em2: TSO
Contexts Failed = 0
				Feb 22 13:35:10 inet-gw kernel: em2:
Excessive collisions = 0
				Feb 22 13:35:10 inet-gw kernel: em2:
Sequence errors = 0
				Feb 22 13:35:10 inet-gw kernel: em2:
Defer count = 20
				Feb 22 13:35:10 inet-gw kernel: em2:
Missed Packets = 68059
				Feb 22 13:35:10 inet-gw kernel: em2:
Receive No Buffers = 275612
				Feb 22 13:35:10 inet-gw kernel: em2:
Receive Length Errors = 0
				Feb 22 13:35:10 inet-gw kernel: em2:
Receive errors = 0
				Feb 22 13:35:10 inet-gw kernel: em2: Crc
errors = 0
				Feb 22 13:35:10 inet-gw kernel: em2:
Alignment errors = 0
				Feb 22 13:35:10 inet-gw kernel: em2:
Collision/Carrier extension errors
				= 0
				Feb 22 13:35:10 inet-gw kernel: em2: RX
overruns = 17
				Feb 22 13:35:10 inet-gw kernel: em2:
watchdog timeouts = 38
				Feb 22 13:35:10 inet-gw kernel: em2: RX
MSIX IRQ = 0 TX MSIX IRQ = 0
				LINK MSIX IRQ = 0
				Feb 22 13:35:10 inet-gw kernel: em2: XON
Rcvd = 21
				Feb 22 13:35:10 inet-gw kernel: em2: XON
Xmtd = 8344
				Feb 22 13:35:10 inet-gw kernel: em2:
XOFF Rcvd = 30
				Feb 22 13:35:10 inet-gw kernel: em2:
XOFF Xmtd = 9159
				Feb 22 13:35:10 inet-gw kernel: em2:
Good Packets Rcvd = 715555016
				Feb 22 13:35:10 inet-gw kernel: em2:
Good Packets Xmtd = 571157561
				Feb 22 13:35:10 inet-gw kernel: em2: TSO
Contexts Xmtd = 0
				Feb 22 13:35:10 inet-gw kernel: em2: TSO
Contexts Failed = 0
				Feb 22 13:39:12 inet-gw kernel: em2:
Excessive collisions = 0
				Feb 22 13:39:12 inet-gw kernel: em2:
Sequence errors = 0
				Feb 22 13:39:12 inet-gw kernel: em2:
Defer count = 20
				Feb 22 13:39:12 inet-gw kernel: em2:
Missed Packets = 68059
				Feb 22 13:39:12 inet-gw kernel: em2:
Receive No Buffers = 275612
				Feb 22 13:39:12 inet-gw kernel: em2:
Receive Length Errors = 0
				Feb 22 13:39:12 inet-gw kernel: em2:
Receive errors = 0
				Feb 22 13:39:12 inet-gw kernel: em2: Crc
errors = 0
				Feb 22 13:39:12 inet-gw kernel: em2:
Alignment errors = 0
				Feb 22 13:39:12 inet-gw kernel: em2:
Collision/Carrier extension errors
				= 0
				Feb 22 13:39:12 inet-gw kernel: em2: RX
overruns = 17
				Feb 22 13:39:12 inet-gw kernel: em2:
watchdog timeouts = 38
				Feb 22 13:39:12 inet-gw kernel: em2: RX
MSIX IRQ = 0 TX MSIX IRQ = 0
				LINK MSIX IRQ = 0
				Feb 22 13:39:12 inet-gw kernel: em2: XON
Rcvd = 21
				Feb 22 13:39:12 inet-gw kernel: em2: XON
Xmtd = 8344
				Feb 22 13:39:12 inet-gw kernel: em2:
XOFF Rcvd = 30
				Feb 22 13:39:12 inet-gw kernel: em2:
XOFF Xmtd = 9159
				Feb 22 13:39:12 inet-gw kernel: em2:
Good Packets Rcvd = 723521981
				Feb 22 13:39:12 inet-gw kernel: em2:
Good Packets Xmtd = 577211431
				Feb 22 13:39:12 inet-gw kernel: em2: TSO
Contexts Xmtd = 0
				Feb 22 13:39:12 inet-gw kernel: em2: TSO
Contexts Failed = 0

				Can this be the problem? "Receive No
Buffers = 275612"

				---- Kirk
				Kirk Davis
				Senior Network Analyst, ITS
				Edmonton Public Schools
				One Kingsway Ave.
				Edmonton, Alberta, Canada
				T5H 4G9

				phone: 1-780-429-8308