kern/138266: kernel panic when udp benchmark test used as regular user

Fri Aug 28 10:00:09 UTC 2009

>Number:         138266
>Category:       kern
>Synopsis:       kernel panic when udp benchmark test used as regular user
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Aug 28 10:00:08 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator:     Viktor CISTICZ
>Release:        8.0-BETA3
>Organization:
codelab.cz
>Environment:
FreeBSD twin1 8.0-BETA3 FreeBSD 8.0-BETA3 #2: Fri Aug 28 08:58:50 CEST 2009     viktor at twin1:/usr/obj/usr/src/sys/GEN_NO_DBG  amd64

>Description:
Basically I have 2 machines running FreeBSD 8.0 and do netio UDP test. The client machine dies after short time. TCP test doesn't provoke it.

After a minute, the client machine ends up in kernel panic. Also
posibility of non working ethernet interfaces may happen.

GEN_NO_DBG is GENERIC kernel without debugging options, GENERIC kernel acts the same way.

ifconfig igb0 down & ifconfig igb0 up may fix it for a short time

This was displayed while testing:

8.0-BETA2 shell

twin1# GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12

8.0-BETA2 /var/log/messages

Aug 17 21:13:32 twin1 kernel: igb0: link state changed to DOWN
Aug 17 21:13:33 twin1 kernel: igb0: link state changed to UP
Aug 17 21:21:58 twin1 kernel: GET BUF: dmamap load failure - 12
Aug 17 21:22:02 twin1 last message repeated 8 times
Aug 17 21:22:13 twin1 kernel: interrupt storm detected on "irq260:";
throttling interrupt source
Aug 17 21:22:28 twin1 kernel: GET BUF: dmamap load failure - 12
Aug 17 21:22:59 twin1 last message repeated 37 times
Aug 17 21:24:24 twin1 last message repeated 134 times
Aug 17 21:24:24 twin1 login: ROOT LOGIN (root) ON ttyv0
Aug 17 21:24:26 twin1 kernel: GET BUF: dmamap load failure - 12
Aug 17 21:24:57 twin1 last message repeated 21 times
Aug 17 21:25:39 twin1 last message repeated 40 times
Aug 17 21:25:39 twin1 kernel:
Aug 17 21:25:39 twin1 kernel: GET BUF: dmamap load failure - 12
Aug 17 21:25:39 twin1 last message repeated 4 times
Aug 17 21:25:39 twin1 kernel:
Aug 17 21:25:39 twin1 kernel:
Aug 17 21:25:39 twin1 kernel: Fatal trap 9: general protection fault
while in kernel mode
Aug 17 21:25:39 twin1 kernel: cpuid = 2; apic id = 12
Aug 17 21:25:39 twin1 kernel: instruction pointer       =
0x20:0xffffffff805d6755
Aug 17 21:25:39 twin1 kernel: stack pointer             =
0x28:0xffffff80af029a30
Aug 17 21:25:39 twin1 kernel: frame pointer             =
0x28:0xffffff80af029a50
Aug 17 21:25:39 twin1 kernel: code segment              = base 0x0,
limit 0xfffff, type 0x1b
Aug 17 21:25:39 twin1 kernel: = DPL 0, pres 1, long 1, def32 0, gran 1

I've upgraded the machine to 8.0-BETA3 and redo the test. It is failing in the same way.

8.0-BETA3 /var/log/messages (parts of the file)

Aug 25 15:26:07 twin1 kernel: The Regents of the University of
California. All rights reserved.
Aug 25 15:26:07 twin1 kernel: FreeBSD is a registered trademark of The
FreeBSD Foundation.
Aug 25 15:26:07 twin1 kernel: FreeBSD 8.0-BETA3 #0: Sat Aug 22 02:00:45
UTC 2009
Aug 25 15:26:07 twin1 kernel:
root at mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC

Aug 25 15:38:17 twin1 kernel: Memory modified after free 0xffffff000d36a300(256) val=a8 @ 0xffffff000d36a318
Aug 25 15:38:17 twin1 kernel: Memory modified after free 0xffffff000d36a100(256) val=a8 @ 0xffffff000d36a118
Aug 25 15:38:17 twin1 kernel: Memory modified after free 0xffffff000d369e00(256) val=a8 @ 0xffffff000d369e18
Aug 25 15:38:17 twin1 kernel: Memory modified after free 0xffffff000d39a900(256) val=a8 @ 0xffffff000d39a918
Aug 25 15:38:17 twin1 kernel: Memory modified after free 0xffffff000d39ab00(256) val=a8 @ 0xffffff000d39ab18
Aug 25 15:38:17 twin1 kernel: Memory modified after free 0xffffff000d369c00(256) val=a8 @ 0xffffff000d369c18
Aug 25 15:38:17 twin1 kernel: Memory modified after free 0xffffff000d39ad00(256) val=a8 @ 0xffffff000d39ad18
Aug 25 15:38:17 twin1 kernel: Memory modified after free 0xffffff000d39b000(256) val=a8 @ 0xffffff000d39b018
Aug 25 15:38:18 twin1 kernel: Memory modified after free 0xffffff000d3c2e00(256) val=a8 @ 0xffffff000d3c2e18
Aug 25 15:38:18 twin1 kernel: Memory modified after free 0xffffff000d3c4600(256) val=a8 @ 0xffffff000d3c4618
Aug 25 15:38:18 twin1 kernel: Memory modified after free 0xffffff000d3c3100(256) val=a8 @ 0xffffff000d3c3118GET BUF: dmamap load
failure - 12
Aug 25 15:38:18 twin1 kernel: Memory modified after free 0xffffff000d369e00(256) val=a8 @ 0xffffff000d369e18
Aug 25 15:38:18 twin1 kernel:
Aug 25 15:38:18 twin1 kernel: Memory modified after free 0xffffff000d3c3300(256) val=a8 @ 0xffffff000d3c3318
Aug 25 15:38:18 twin1 kernel: Memory modified after free 0xffffff000d3c4400(256) val=a8 @ 0xffffff000d3c4418

Before the network traffic was cutoff, I've got netstat -m message:

67688/1957/69645 mbufs in use (current/cache/total)
24804/796/25600/25600 mbuf clusters in use (current/cache/total/max)
24290/542 mbuf+clusters out of packet secondary zone in use (current/cache)
12787/13/12800/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
117678K/2133K/119811K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

The transcription of kernel panic message captured screen(might be with some mispelling):

GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12

Fatal trap 12: page fault while in kernel mode
cpuid = 5; apic id = 15
fault virtual address   = 0x0
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff805db435
stack pointer           = 0x28:0xffffff80afc33a30
frame pointer           = 0x28:0xffffff80afc33a50
code segment            = base 0x0, limit 0xfffff, type 0x1b
                      = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (irq260: igb1)
trap number             = 12
panic: page fault
cpuid = 5
Uptime: 10m59s
Physical memory: 6121MB
Dumping 1532 MB:GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12
GET BUF: dmamap load failure - 12

>How-To-Repeat:
start netio server : server2$ netio -s
run netio udp test : server1$ netio -u address_to_server2 
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted: