svn commit: r289279 - in head/sys: kern vm

[Tijl Coosemans]

On Wed, 14 Oct 2015 02:10:07 +0000 (UTC) Jeff Roberson <jeff at FreeBSD.org> wrote:
> Author: jeff
> Date: Wed Oct 14 02:10:07 2015
> New Revision: 289279
> URL: https://svnweb.freebsd.org/changeset/base/289279
> 
> Log:
>   Parallelize the buffer cache and rewrite getnewbuf().  This results in a
>   8x performance improvement in a micro benchmark on a 4 socket machine.
>   
>    - Get buffer headers from a per-cpu uma cache that sits in from of the
>      free queue.
>    - Use a per-cpu quantum cache in vmem to eliminate contention for kva.
>    - Use multiple clean queues according to buffer cache size to eliminate
>      clean queue lock contention.
>    - Introduce a bufspace daemon that attempts to prevent getnewbuf() callers
>      from blocking or doing direct recycling.
>    - Close some bufspace allocation races that could lead to endless
>      recycling.
>    - Further the transition to a more modern style of small functions grouped
>      by prefix in order to improve growing complexity.

I have an i386 system that locks up easily after this commit.  Booting
into single user and running make installkernel triggers it consistently.
I haven't been able to reproduce it on amd64.  Examining threads with
DDB shows that they are all at sched_switch (a few at fork_trampoline).
The only lock being held is Giant (by the interrupt handler for
ctrl+alt+esc I think).  So it doesn't look like a dead lock.  It's more
a sleeping beauty situation.  All threads in the castle are sleeping and
there's no prince to wake them up.

(kgdb) info thread
  Id   Target Id         Frame

These are from make installkernel:

  72   Thread 100071 (PID=107: install) sched_switch (td=0xc667d000, 
    newtd=0xc6407000, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  71   Thread 100070 (PID=81: make) sched_switch (td=0xc667d340, 
    newtd=0xc667d000, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  70   Thread 100067 (PID=30: make) sched_switch (td=0xc667e000, 
    newtd=0xc667d340, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  69   Thread 100066 (PID=25: make) sched_switch (td=0xc667e340, 
    newtd=0xc667e000, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969

Single user shell:

  68   Thread 100065 (PID=17: sh) sched_switch (td=0xc6406000, 
    newtd=0xc667e340, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969

Kernel threads:

  67   Thread 100063 (PID=16: vnlru) sched_switch (td=0xc6406680, 
    newtd=0xc6407340, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  66   Thread 100062 (PID=9: syncer) sched_switch (td=0xc64069c0, 
    newtd=0xc667d000, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  65   Thread 100061 (PID=8: bufspacedaemon) sched_switch (td=0xc6407000, 
    newtd=0xc62dc000, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  64   Thread 100060 (PID=7: bufdaemon) sched_switch (td=0xc6407340, 
    newtd=0xc6408000, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  63   Thread 100068 (PID=7: bufdaemon//var worker) sched_switch (
    td=0xc667d9c0, newtd=0xc6407000, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  62   Thread 100069 (PID=7: bufdaemon//usr worker) sched_switch (
    td=0xc667d680, newtd=0xc667d000, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  61   Thread 100059 (PID=6: pagezero) sched_switch (td=0xc6407680, 
    newtd=0xc55ba680, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  60   Thread 100058 (PID=5: vmdaemon) sched_switch (td=0xc64079c0, 
    newtd=0xc6407340, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  59   Thread 100057 (PID=4: pagedaemon) sched_switch (td=0xc6408000, 
    newtd=0xc6407000, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  58   Thread 100064 (PID=4: pagedaemon/uma) sched_switch (td=0xc6406340, 
    newtd=0xc55b9340, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
  57   Thread 100050 (PID=15: acpi_cooling0) sched_switch (td=0xc62dc340, 
    newtd=0xc6407000, flags=<optimized out>)
    at /usr/src/sys/kern/sched_ule.c:1969
....

Anything else you need to debug this?