misc/131290: How to completely freeze FreeBSD 7.1 under a non-privileged user

Fri Feb 6 03:33:22 PST 2009

On Fri, 06 Feb 2009 12:28:09 +0200 Mikolaj Golub wrote:

> On Thu, 05 Feb 2009 21:23:16 +0200 Mikolaj Golub wrote:
>
>> On Thu, 5 Feb 2009 07:51:31 -0800 Sergey S wrote:
>>
>>  SS> Hello.
>>
>>  >> I can't hang the system running many cycles of loop.sh...
>>
>>  SS> :(
>>
>>  SS> I've no idea, what can be done in order to elucidate the problem...
>>
>> When I went home I left the machine with loop.sh run and now it is not
>> pingable, so it looks like it hangs :-) Tomorrow, I will know this for sure.
>>
>> I noticed from your top output that your system was idle when you did your
>> test while mine was rather loaded. That is why, I suppose, it required more
>> time for me to hang it.
>
> Yes, it hanged. I added to the GENERIC kernel config the followin options:
>
> options     INVARIANTS
> options     INVARIANT_SUPPORT
> options     WITNESS
> options     DEBUG_LOCKS
> options     DEBUG_VFS_LOCKS
> options     DIAGNOSTIC
>
> options KDB
> options DDB
>
> and run script on idle system. The system froze on the second cycle run. It
> did not panic and I was not able to enter the debugger, so no crush dump is
> available.
>
> Then I changed the scheduler to 4BSD. It hangs with 4BSD too but it looks like
> it requires more cycles. When it froze first time with 4BSD I managed to enter
> debugger pressing Ctr-ALt-ESC. But I made silly thing, I decided to check if
> the system would resume to normal state if I just return from the debugger. It
> did not and I was not able to enter ddb again. After this I hanged the system
> several times to try to enter ddb but without success :-(

But I can make my system panic in another way. When I did tests, before
running loop.sh I stopped some of my services to make my system idle. One time
when I stopped ejabberd (it is written on erlang if somebody does not know)
the system panicked. It is reproducible. Next time it required for me several
times to do

/usr/local/etc/rc.d/ejabberd start
/usr/local/etc/rc.d/ejabberd stop

cycles to make the system panic. I don't know if this is related to this
thread initial problem, but below is some gdb info for this crash.

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: Bad link elm 0xc4f0c1f0 next->prev != elm
cpuid = 0
KDB: enter: panic
exclusive sleep mutex sellck r = 0 (0xc0cc7204) locked @ /usr/src/sys/kern/sys_generic.c:1127
exclusive sleep mutex pipe mutex r = 0 (0xc4f0c2fc) locked @ /usr/src/sys/kern/sys_pipe.c:1132
panic: from debugger
cpuid = 0
Uptime: 38m7s
Physical memory: 1003 MB
Dumping 116 MB: 101 85 69 53 37 21 5

Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from /boot/kernel/pf.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pf.ko
Reading symbols from /boot/kernel/smbfs.ko...Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/smbfs.ko
Reading symbols from /boot/kernel/libiconv.ko...Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/libiconv.ko
Reading symbols from /boot/kernel/libmchain.ko...Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/libmchain.ko
Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linux.ko
Reading symbols from /boot/kernel/logo_saver.ko...Reading symbols from /boot/kernel/logo_saver.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/logo_saver.ko
Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /boot/kernel/ng_socket.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_socket.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/if_bridge.ko...Reading symbols from /boot/kernel/if_bridge.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_bridge.ko
Reading symbols from /boot/kernel/bridgestp.ko...Reading symbols from /boot/kernel/bridgestp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/bridgestp.ko
#0  doadump () at pcpu.h:196
196	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) backtrace
#0  doadump () at pcpu.h:196
#1  0xc079a07e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc079a352 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc0493a07 in db_panic (addr=Could not find the frame base for "db_panic".
) at /usr/src/sys/ddb/db_command.c:446
#4  0xc049440c in db_command (last_cmdp=0xc0c48114, cmd_table=0x0, dopager=1)
    at /usr/src/sys/ddb/db_command.c:413
#5  0xc049451a in db_command_loop () at /usr/src/sys/ddb/db_command.c:466
#6  0xc0495d0d in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_main.c:228
#7  0xc07c3866 in kdb_trap (type=3, code=0, tf=0xe69ceac8)
    at /usr/src/sys/kern/subr_kdb.c:524
#8  0xc0a9fb5b in trap (frame=0xe69ceac8) at /usr/src/sys/i386/i386/trap.c:688
#9  0xc0a8541b in calltrap () at /usr/src/sys/i386/i386/exception.s:159
#10 0xc07c39ea in kdb_enter_why (why=0xc0b42038 "panic", 
    msg=0xc0b42038 "panic") at cpufunc.h:60
#11 0xc079a336 in panic (fmt=0xc0af1d18 "Bad link elm %p next->prev != elm")
    at /usr/src/sys/kern/kern_shutdown.c:557
#12 0xc07d4b3b in doselwakeup (sip=0xc4f0c1f0, pri=88)
    at /usr/src/sys/kern/sys_generic.c:1138
#13 0xc07d4c1e in selwakeuppri (sip=0xc4f0c1f0, pri=88)
    at /usr/src/sys/kern/sys_generic.c:1114
#14 0xc07d9c8e in pipe_write (fp=0xc42b95f0, uio=0xe69cec60, 
    active_cred=0xc4aa8800, flags=0, td=0xc4bc8aa0)
    at /usr/src/sys/kern/sys_pipe.c:528
---Type <return> to continue, or q <return> to quit---#15 0xc07d6095 in dofilewrite (td=0xc4bc8aa0, fd=4, fp=0xc42b95f0, 
    auio=0xe69cec60, offset=-1, flags=0) at file.h:256
#16 0xc07d6318 in kern_writev (td=0xc4bc8aa0, fd=4, auio=0xe69cec60)
    at /usr/src/sys/kern/sys_generic.c:401
#17 0xc07d638f in write (td=0xc4bc8aa0, uap=0xe69cecfc)
    at /usr/src/sys/kern/sys_generic.c:317
#18 0xc0a9f2d3 in syscall (frame=0xe69ced38)
    at /usr/src/sys/i386/i386/trap.c:1090
#19 0xc0a85480 in Xint0x80_syscall ()
    at /usr/src/sys/i386/i386/exception.s:255
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) list *0xc07d4b3b
0xc07d4b3b is in doselwakeup (/usr/src/sys/kern/sys_generic.c:1138).
1133		}
1134		if (td == NULL) {
1135			mtx_unlock(&sellock);
1136			return;
1137		}
1138		TAILQ_REMOVE(&td->td_selq, sip, si_thrlist);
1139		sip->si_thread = NULL;
1140		thread_lock(td);
1141		td->td_flags &= ~TDF_SELECT;
1142		thread_unlock(td);
(kgdb) 

For next panic I configured ddb to produce textdump. Obtained textdump.tar is
attached.

Please, let me know if I can provide any other useful info.

-- 
Mikolaj Golub