Help debugging stable/10

[Shane Ambler]

Since upgrading to 10.1 (RC2) I have had trouble getting uptimes greater
than 1 day. I have little experience debugging the OS so could use some
help.

# uname -a
FreeBSD leader.local 10.1-STABLE FreeBSD 10.1-STABLE #0 r275364: Tue Dec
2 08:13:06 ACDT 2014 
root at leader.local:/usr/obj/usr/src-stable/sys/GENERIC amd64

This is on an ASUS P8H61-M LE/USB3 corei5 8GB with 3x 2TB Seagate drives
in raidz.
Full backtraces and dmesg at http://shaneware.biz/freebsddebugdata/

The thing that breaks which forces me to reset the machine is that I am
unable to start new processes. Existing processes continue to work I
just can't start new ones. Some simple commands do work but top ps
procstat usbconfig all fail to start. I have been able use script to
get some backtraces from kgdb before restarting.

Back in RC2 and 3 I had it freeze up leaving me to reset, since RC4 and
now on stable I have been able to close up most processes before resetting.

I don't know how to re-create this issue on command, but I just had 4
days uptime which is the best I've had in the last month.

How do I find what is preventing new processes starting?

What can I do to get more data when things go south?

I noticed an odd thing in the backtrace today -

Thread 16 (Thread 101376):
#0  sched_switch (td=0xfffff8011c1e7000, newtd=<value optimized out>, 
flags=<value optimized out>)
     at /usr/src-stable/sys/kern/sched_ule.c:1945
#1  0x0000000000000020 in ?? ()
#2  0xfffffe021e5a8200 in ?? ()
#3  0xffffffff80913166 in malloc (size=<value optimized out>, mtp=<value 
optimized out>, flags=<value optimized out>)
     at /usr/src-stable/sys/kern/kern_malloc.c:352
#4  0xec81485354415541 in ?? ()
#5  0x4dce894d00000268 in ?? ()
#6  0xd58949cc8949c789 in ?? ()
#7  0x9358253c8148fb89 in ?? ()
#8  0x850f000000008164 in ?? ()

<snip for email>

#16890 0x067ee8f68944df89 in ?? ()
#16891 0x8618251c8948fdd7 in ?? ()
#16892 0x415b10c4834882c1 in ?? ()
#16893 0x0000000000c35d5e in ?? ()
#16894 0x0000000000000000 in ?? ()

Yes there are nearly 17,000 lines in that backtrace, no idea what it is
though.

-- 
FreeBSD - the place to B...Software Developing

Shane Ambler