Random panics (page fault). ktls_decrypt might be involved?

From: Ruben van Staveren <ruben_at_verweg.com>
Date: Fri, 21 Jul 2023 13:40:05 UTC
Hi list,

I’m getting some page fault panics recently, but if I look in the crashinfo dumps there seems always be a ktls_decrypt present. I’m using kernel TLS with dnsdist.

Just checking if this is the right assumption before I’m going to file a pr

Best Regards,
    Ruben


Unread portion of the kernel message buffer:
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 2 (thr_9)
trap number             = 12
panic: page fault
cpuid = 4
time = 1689945150
KDB: stack backtrace:
#0 0xffffffff80c53d95 at kdb_backtrace+0x65
#1 0xffffffff80c06711 at vpanic+0x151
#2 0xffffffff80c065b3 at panic+0x43
#3 0xffffffff810b1fa7 at trap_fatal+0x387
#4 0xffffffff810b1fff at trap_pfault+0x4f
#5 0xffffffff81088e48 at calltrap+0x8
#6 0xffffffff80bc2f9e at fork_exit+0x7e
#7 0xffffffff81089ebe at fork_trampoline+0xe
Uptime: 6h29m8s
Dumping 18598 out of 130332 MB: (CTRL-C to abort) ..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

….

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:396
#2  0xffffffff80c062da in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:484
#3  0xffffffff80c0677e in vpanic (fmt=<optimized out>, 
    ap=ap@entry=0xfffffe017d99cc50) at /usr/src/sys/kern/kern_shutdown.c:923
#4  0xffffffff80c065b3 in panic (fmt=<unavailable>)
    at /usr/src/sys/kern/kern_shutdown.c:847
#5  0xffffffff810b1fa7 in trap_fatal (frame=0xfffffe017d99cd40, eva=81)
    at /usr/src/sys/amd64/amd64/trap.c:942
#6  0xffffffff810b1fff in trap_pfault (frame=0xfffffe017d99cd40, 
    usermode=false, signo=<optimized out>, ucode=<optimized out>)
    at /usr/src/sys/amd64/amd64/trap.c:761
#7  <signal handler called>
#8  0xffffffff80c97b88 in ktls_decrypt (so=0xfffff801e8313000)
    at /usr/src/sys/kern/uipc_ktls.c:1913
#9  ktls_work_thread (ctx=ctx@entry=0xfffff801037ebc80)
    at /usr/src/sys/kern/uipc_ktls.c:2407
#10 0xffffffff80bc2f9e in fork_exit (
    callout=0xffffffff80c97950 <ktls_work_thread>, arg=0xfffff801037ebc80, 
    frame=0xfffffe017d99cf40) at /usr/src/sys/kern/kern_fork.c:1093
#11 <signal handler called>
#12 0x171e5d4cc9e9e0a3 in ?? ()
Backtrace stopped: Cannot access memory at address 0x430c6a4638d29dfb
(kgdb) 

------------------------------------------------------------------------

….
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ     FAILSLEEP XDOMAIN
…
ktls_session:           192,      0,    1177,     497,    2352,   0,   0,   0
…