svn commit: r289279 - in head/sys: kern vm

[Konstantin Belousov]

On Thu, Oct 29, 2015 at 09:25:54PM +0100, Tijl Coosemans wrote:
> On Wed, 14 Oct 2015 02:10:07 +0000 (UTC) Jeff Roberson <jeff at FreeBSD.org> wrote:
> > Author: jeff
> > Date: Wed Oct 14 02:10:07 2015
> > New Revision: 289279
> > URL: https://svnweb.freebsd.org/changeset/base/289279
> > 
> > Log:
> >   Parallelize the buffer cache and rewrite getnewbuf().  This results in a
> >   8x performance improvement in a micro benchmark on a 4 socket machine.
> >   
> >    - Get buffer headers from a per-cpu uma cache that sits in from of the
> >      free queue.
> >    - Use a per-cpu quantum cache in vmem to eliminate contention for kva.
> >    - Use multiple clean queues according to buffer cache size to eliminate
> >      clean queue lock contention.
> >    - Introduce a bufspace daemon that attempts to prevent getnewbuf() callers
> >      from blocking or doing direct recycling.
> >    - Close some bufspace allocation races that could lead to endless
> >      recycling.
> >    - Further the transition to a more modern style of small functions grouped
> >      by prefix in order to improve growing complexity.
> 
> I have an i386 system that locks up easily after this commit.  Booting
> into single user and running make installkernel triggers it consistently.
> I haven't been able to reproduce it on amd64.  Examining threads with
> DDB shows that they are all at sched_switch (a few at fork_trampoline).
> The only lock being held is Giant (by the interrupt handler for
> ctrl+alt+esc I think).  So it doesn't look like a dead lock.  It's more
> a sleeping beauty situation.  All threads in the castle are sleeping and
> there's no prince to wake them up.
> 
> (kgdb) info thread
>   Id   Target Id         Frame
> 
> These are from make installkernel:
> 
>   72   Thread 100071 (PID=107: install) sched_switch (td=0xc667d000, 
>     newtd=0xc6407000, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   71   Thread 100070 (PID=81: make) sched_switch (td=0xc667d340, 
>     newtd=0xc667d000, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   70   Thread 100067 (PID=30: make) sched_switch (td=0xc667e000, 
>     newtd=0xc667d340, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   69   Thread 100066 (PID=25: make) sched_switch (td=0xc667e340, 
>     newtd=0xc667e000, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
> 
> Single user shell:
> 
>   68   Thread 100065 (PID=17: sh) sched_switch (td=0xc6406000, 
>     newtd=0xc667e340, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
> 
> Kernel threads:
> 
>   67   Thread 100063 (PID=16: vnlru) sched_switch (td=0xc6406680, 
>     newtd=0xc6407340, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   66   Thread 100062 (PID=9: syncer) sched_switch (td=0xc64069c0, 
>     newtd=0xc667d000, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   65   Thread 100061 (PID=8: bufspacedaemon) sched_switch (td=0xc6407000, 
>     newtd=0xc62dc000, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   64   Thread 100060 (PID=7: bufdaemon) sched_switch (td=0xc6407340, 
>     newtd=0xc6408000, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   63   Thread 100068 (PID=7: bufdaemon//var worker) sched_switch (
>     td=0xc667d9c0, newtd=0xc6407000, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   62   Thread 100069 (PID=7: bufdaemon//usr worker) sched_switch (
>     td=0xc667d680, newtd=0xc667d000, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   61   Thread 100059 (PID=6: pagezero) sched_switch (td=0xc6407680, 
>     newtd=0xc55ba680, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   60   Thread 100058 (PID=5: vmdaemon) sched_switch (td=0xc64079c0, 
>     newtd=0xc6407340, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   59   Thread 100057 (PID=4: pagedaemon) sched_switch (td=0xc6408000, 
>     newtd=0xc6407000, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   58   Thread 100064 (PID=4: pagedaemon/uma) sched_switch (td=0xc6406340, 
>     newtd=0xc55b9340, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
>   57   Thread 100050 (PID=15: acpi_cooling0) sched_switch (td=0xc62dc340, 
>     newtd=0xc6407000, flags=<optimized out>)
>     at /usr/src/sys/kern/sched_ule.c:1969
> ....
> 
> Anything else you need to debug this?

Start with gathering the information listed in
https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html