threads/103127: Kernel panic while using thread features in Squid 2.6

[Anders Nordby]

Using libthr instead of pthread didn't help at all.

It might be a kqueue issue after all. On recommendation by Pawel Worach,
I set these sysctls:

debug.trace_on_panic=1
debug.debugger_on_panic=0

Then I could get more debugging info:

# dmesg -M /var/crash/vmcore.0 -N /usr/obj/usr/src/sys/CACHE/kernel.debug
(..)
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 06
fault virtual address   = 0x0
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0x0
stack pointer           = 0x28:0xeb886b98
frame pointer           = 0x28:0xeb886bcc
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 545 (squid)
trap number             = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
kdb_backtrace(100,c9599480,28,eb886b58,c) at kdb_backtrace+0x29
panic(c087f113,c08ce651,0,fffff,c959d49b) at panic+0x114
trap_fatal(eb886b58,0,c9599480,c8f8a708,c) at trap_fatal+0x2ce
trap_pfault(eb886b58,0,0) at trap_pfault+0x1d7
trap(8,c9590028,ce020028,c9feb0b4,eb886c30) at trap+0x2fd
calltrap() at calltrap+0x5
--- trap 0xc, eip = 0, esp = 0xeb886b98, ebp = 0xeb886bcc ---
(null)(c9560800,eb886c1c,c9599480,1,0) at 0
kern_kevent(c9599480,3,37,80,eb886cc8) at kern_kevent+0xc9
kevent(c9599480,eb886d04,6,4bacc,202) at kevent+0x55
syscall(3b,3b,bfbf003b,82180b1,0) at syscall+0x2bf
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (363, FreeBSD ELF32, kevent), eip = 0xb71e24af, esp =
0xbfbfe9cc, ebp = 0xbfbfea18 ---
Uptime: 21h18m39s
Dumping 3839 MB (2 chunks)
  chunk 0: 1MB (159 pages) ... ok
  chunk 1: 3839MB (982778 pages) 3824 3808 3792 3776 3760 3744 3728 3712
3696 3680 3664 3648 3632 3616 3600 3584 3568 3552 3536 3520 3504 3488
3472 3456 3440 3424 3408 3392 3376 3360 3344 3328 3312 3296 3280 3264
3248 3232 3216 3200 3184 3168 3152 3136 3120 3104 3088 3072 3056 3040
3024 3008 2992 2976 2960 2944 2928 2912 2896 2880 2864 2848 2832 2816
2800 2784 2768 2752 2736 2720 2704 2688 2672 2656 2640 2624 2608 2592
2576 2560 2544 2528 2512 2496 2480 2464 2448 2432 2416 2400 2384 2368
2352 2336 2320 2304 2288 2272 2256 2240 2224 2208 2192 2176 2160 2144
2128 2112 2096 2080 2064 2048 2032 2016 2000 1984 1968 1952 1936 1920
1904 1888 1872 1856 1840 1824 1808 1792 1776 1760 1744 1728 1712 1696
1680 1664 1648 1632 1616 1600 1584 1568 1552 1536 1520 1504 1488 1472
1456 1440 1424 1408 1392 1376 1360 1344 1328 1312 1296 1280 1264 1248
1232 1216 1200 1184 1168 1152 1136 1120 1104 1088 1072 1056 1040 1024
1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736
720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448
432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160
144 128 112 96 80 64 48 32 16

Checking those kevent's, I get:

(kgdb) l *kevent+0x55
0xc06542fd is in kevent (/usr/src/sys/kern/kern_event.c:571).
566                             return (error);
567                     tsp = &ts;
568             } else
569                     tsp = NULL;
570
571             return (kern_kevent(td, uap->fd, uap->nchanges,
uap->nevents,
572                 &k_ops, tsp));
573     }
574
575     /*
(kgdb) l *kern_kevent+0xc9
0xc0654439 is in kern_kevent (/usr/src/sys/kern/kern_event.c:637).
632                             goto done;
633                     changes = keva;
634                     for (i = 0; i < n; i++) {
635                             kevp = &changes[i];
636                             kevp->flags &= ~EV_SYSFLAGS;
637                             error = kqueue_register(kq, kevp, td,
1);
638                             if (error) {
639                                     if (nevents != 0) {
640                                             kevp->flags = EV_ERROR;
641                                             kevp->data = error;
(kgdb) 

This is in FreeBSD 6.1-RELEASE-p5. One physical processor, but two logical
(HyperThreading enabled).

Regards,
Anders.