LOR on em in HEAD ( was Re: em driver regression
John Baldwin
jhb at freebsd.org
Mon Apr 12 15:12:29 UTC 2010
On Friday 09 April 2010 3:09:24 pm Jack Vogel wrote:
> Someone else also pointed this out. I'm dubious about its claim.
> This happens because there is an RX lock taken in rxeof, its held
> thru the call into the stack, it then encounters another lock there
> and hence this complaint. I've had the RX hold as it is for a long
> while and would rather not have to give it up, can someone look
> at it and advise?
I've seen it happen with igb. I suspect it is a transitive lock order. That
is, you probably never have the UDP lock acquired before an em/igb RX lock.
However, if you have an em/igb adapter TX lock held when you acquire an em/igb
RX lock in one place, and in if_start() you acquire the TX lock while the UDP
lock is held, that can trigger the LOR. Specifically, those two paths would
give you these two orders:
TX -> RX
UDP -> TX
which implies the order
UDP -> RX
(lock order relationsips are transitive, just like a > b and b > c implies
a > c).
However, I haven't been able to track down what the raw orders are that might
lead to this transitive order. Attilio added some sysctls to dump all the raw
lock orders in one of the debug.witness sysctls. You can also try hardcoding
the 'RX -> UDP' order using WITNESS_DEFINEORDER() before any of the em/igb
RX/TX locks are acquired to see what different LOR is triggered. If that LOR
looks valid then you can keep hardcoding valid orders until you find the
invalid one.
> On Fri, Apr 9, 2010 at 12:00 PM, Mike Tancsa <mike at sentex.net> wrote:
>
> > While testing an i5 box with HEAD checked out from this morning, bringing
> > up the second NIC generated this LOR on the console
> >
> > em1: link state changed to UP
> > lock order reversal:
> > 1st 0xc5dc7c10 em1:rx(1) (em1:rx(1)) @
> > /usr/HEAD/src/sys/modules/em/../../dev/e1000/if_em.c:4089
> > 2nd 0xc0f7e88c udp (udp) @ /usr/HEAD/src/sys/netinet/udp_usrreq.c:454
> > KDB: stack backtrace:
> > db_trace_self_wrapper(c0cb4d33,c5b72a70,c08e4d65,c08d50db,c0cb7d58,...) at
> > db_trace_self_wrapper+0x26
> > kdb_backtrace(c08d50db,c0cb7d58,c5d31a98,c5d2cb60,c5b72acc,...) at
> > kdb_backtrace+0x29
> > _witness_debugger(c0cb7d58,c0f7e88c,c0c9cf0b,c5d2cb60,c0cd04ca,...) at
> > _witness_debugger+0x25
> > witness_checkorder(c0f7e88c,1,c0cd04ca,1c6,0,...) at
> > witness_checkorder+0x839
> > _rw_rlock(c0f7e88c,c0cd04ca,1c6,c5d33088,c5e8be24,...) at _rw_rlock+0x9c
> > udp_input(c67faa00,14,0,c5e8bd80,c0dfa860,...) at udp_input+0x246
> > ip_input(c67faa00,c5f2f380,c5b72be8,c07463b6,c0dfa860,...) at
> > ip_input+0x606
> > netisr_dispatch_src(1,0,c67faa00,c5b72c20,c0954bc1,...) at
> > netisr_dispatch_src+0xcd
> > netisr_dispatch(1,c67faa00,c6018c00,c6018c00,c6852800,...) at
> > netisr_dispatch+0x20
> > ether_demux(c6018c00,c67faa00,3,0,3,...) at ether_demux+0x1a1
> > ether_input(c6018c00,c67faa00,c11a0e17,ff9,64,...) at ether_input+0x365
> > em_rxeof(c5e8bd80,109,c6016180,0,c5b72cc8,...) at em_rxeof+0x13c
> > em_msix_rx(c5dc7c00,c5b72cc8,c088eb14,c0e133c0,c60342b8,...) at
> > em_msix_rx+0x25
> > intr_event_execute_handlers(c5d807f8,c6034280,c0cacd7e,533,c60342f0,...)
at
> > intr_event_execute_handlers+0x125
> > ithread_loop(c603b4a0,c5b72d38,c0cacaed,343,c5d807f8,...) at
> > ithread_loop+0x9f
> > fork_exit(c0877800,c603b4a0,c5b72d38) at fork_exit+0xb8
> > fork_trampoline() at fork_trampoline+0x8
> > --- trap 0, eip = 0, esp = 0xc5b72d70, ebp = 0 ---
> >
> > 0(i5b)# uname -a
> > FreeBSD i5b.sentex.ca 9.0-CURRENT FreeBSD 9.0-CURRENT #2: Fri Apr 9
> > 11:56:25 EDT 2010
mdtancsa at ich10.sentex.ca:/usr/HEAD/obj/usr/HEAD/src/sys/GENERIC
> > i386
> > 0(i5b)#
> >
> > em0 at pci0:0:25:0: class=0x020000 card=0x34ec8086 chip=0x10ef8086
> > rev=0x05 hdr=0x00
> > vendor = 'Intel Corporation'
> > class = network
> > subclass = ethernet
> > cap 01[c8] = powerspec 2 supports D0 D3 current D0
> > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
> > cap 13[e0] = PCI Advanced Features: FLR TP
> > em1 at pci0:3:0:0: class=0x020000 card=0x34ec8086 chip=0x10d38086 rev=0x00
> > hdr=0x00
> > vendor = 'Intel Corporation'
> > device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
> > class = network
> > subclass = ethernet
> > cap 01[c8] = powerspec 2 supports D0 D3 current D0
> > cap 05[d0] = MSI supports 1 message, 64 bit
> > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
> > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled
> >
> >
> > 0(i5b)# vmstat -i
> > interrupt total rate
> > irq4: uart0 6156 3
> > irq8: rtc 224879 127
> > irq21: ehci0 2662 1
> > irq23: ehci1 3674 2
> > cpu0: timer 1754210 998
> > irq256: em0 10778 6
> > irq257: em1 331 0
> > irq258: em1 4 0
> > irq260: em1 4 0
> > irq261: em1 8 0
> > irq262: ahci0 69 0
> > cpu3: timer 1753938 998
> > cpu2: timer 1753932 998
> > cpu1: timer 1753886 998
> > Total 7264531 4134
> > 0(i5b)#
> >
> > CPU: Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz (2666.65-MHz 686-
class
> > CPU)
> > Origin = "GenuineIntel" Id = 0x106e5 Family = 6 Model = 1e Stepping =
> > 5
> >
> >
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> >
> >
Features2=0x98e3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT>
> > AMD Features=0x28100000<NX,RDTSCP,LM>
> > AMD Features2=0x1<LAHF>
> > TSC: P-state invariant
> > real memory = 4294967296 (4096 MB)
> > avail memory = 2577711104 (2458 MB)
> > ACPI APIC Table: <INTEL S3420GPC>
> > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
> >
> >
> >
> >
> > --------------------------------------------------------------------
> > Mike Tancsa, tel +1 519 651 3400
> > Sentex Communications, mike at sentex.net
> > Providing Internet since 1994 www.sentex.net
> > Cambridge, Ontario Canada www.sentex.net/mike
> >
> >
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"
>
--
John Baldwin
More information about the freebsd-current
mailing list