FreeBSD 7-STABLE mbuf corruption

[Arnaud Lacombe]

Hi,

On Tue, Sep 13, 2011 at 2:36 PM, Arnaud Lacombe <lacombar at gmail.com> wrote:
> Hi,
>
> On Wed, Sep 7, 2011 at 7:57 PM, Jack Vogel <jfvogel at gmail.com> wrote:
>> I have seen this, but I don't have any hot ideas right off the top of my
>> head yet :(
>>
> I've been running for 19h now the following patches:
>  - backport of kmacy@'s buf_ring(9) API, from FreeBSD 8 (from [0], see
> attachment for full diff)
>  - conversion of igb(4), from CURRENT, to use buf_ring(9) on FreeBSD
> 7.1 (see attachment)
>  - all the original patches I already sent
>
> It did not crash, yet. The only downside is that after 3h30 and ~4h,
> igb(4) queues' handler started spinning infinitely, breaking network
> connectivity.
>
> I would be tempted to say that the infinite loop issue is an igb(4)
> (separate from the original crashes),
Here's a backtrace of the 3 "igb1 que" spinning on CPU0, CPU1 and
CPU3, when the condition occurs:

Tracing command igb1 que pid 44 tid 100043 td 0xc6a79460
cpustop_handler(9e0008,28,28,c0838783,1220,...) at cpustop_handler+0x32
Xcpustop() at Xcpustop+0x2d
--- interrupt, eip = 0xc062f518, esp = 0xf4589bbc, ebp = 0xf4589bdc ---
_mtx_unlock_flags(c69e7e9c,0,c0838783,1220,c6a3f8d4,...) at
_mtx_unlock_flags+0x98
igb_rxeof(f4589ca4,c062f502,c6a73b9c,4,c0850e5e,...) at igb_rxeof+0x543
igb_handle_que(c6a3f8b4,1,c0856713,54,c6a73b84,...) at igb_handle_que+0x76
taskqueue_run(c6a73b80,c6a73b9c,0,c08492c5,0,...) at taskqueue_run+0xa1
taskqueue_thread_loop(c6a3f8e4,f4589d38,c084e8d3,31c,c6a7f570,...) at
taskqueue_thread_loop+0x6a
fork_exit(c06718f0,c6a3f8e4,f4589d38) at fork_exit+0xa6
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xf4589d70, ebp = 0 ---

Tracing command igb1 que pid 42 tid 100041 td 0xc6a798c0
cpustop_handler(8,28,f4580028,0,0,...) at cpustop_handler+0x32
Xcpustop() at Xcpustop+0x2d
--- interrupt, eip = 0xc07f6d5b, esp = 0xf4583c40, ebp = 0xf4583c44 ---
spinlock_exit(c08e58b4,0,c0856087,108,f4583c7c,...) at spinlock_exit+0x2b
sleepq_release(c6a73c00,0,ffffffff,0,c6a73c00,...) at sleepq_release+0x37
wakeup_one(c6a73c00,0,c0856713,4b,c6a73c1c,...) at wakeup_one+0x41
taskqueue_enqueue(c6a73c00,c6a3f898,c0856713,54,c6a73c04,...) at
taskqueue_enqueue+0x169
taskqueue_run(c6a73c00,c6a73c1c,0,c08492c5,0,...) at taskqueue_run+0xa1
taskqueue_thread_loop(c6a3f8a8,f4583d38,c084e8d3,31c,c6a7fae0,...) at
taskqueue_thread_loop+0x6a
fork_exit(c06718f0,c6a3f8a8,f4583d38) at fork_exit+0xa6
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xf4583d70, ebp = 0 ---

Tracing command igb1 que pid 40 tid 100039 td 0xc6a79d20
cpustop_handler(8,f4570028,28,0,c6a3f85c,...) at cpustop_handler+0x32
Xcpustop() at Xcpustop+0x2d
--- interrupt, eip = 0xc07f6d5b, esp = 0xf457dc44, ebp = 0xf457dc48 ---
spinlock_exit(c08e5f44,0,c0856087,108,f457dc88,...) at spinlock_exit+0x2b
sleepq_release(c6a3f85c,c6a73c80,f457dc88,c066f7a7,c08e5f44,...) at
sleepq_release+0x37
sleepq_broadcast(c6a3f85c,0,ffffffff,0,c6a3f85c,...) at sleepq_broadcast+0xd7
wakeup(c6a3f85c,0,c0856713,4b,c6a73c84,...) at wakeup+0x32
taskqueue_run(c6a73c80,c6a73c9c,0,c08492c5,0,...) at taskqueue_run+0xdb
taskqueue_thread_loop(c6a3f86c,f457dd38,c084e8d3,31c,c6a2a000,...) at
taskqueue_thread_loop+0x6a
fork_exit(c06718f0,c6a3f86c,f457dd38) at fork_exit+0xa6
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xf457dd70, ebp = 0 ---

 - Arnaud