Recurring problem: processes block accessing UFS file system

Greg Rivers gcr+freebsd-stable at tharned.org
Wed Nov 23 03:48:10 GMT 2005


On Mon, 21 Nov 2005, Kris Kennaway wrote:

> It may not be the same problem.  You should also try to obtain a trace 
> when snapshots are not implicated.
>

Agreed.  I'll do so at the first opportunity.


> 'show lockedvnods' is very important for diagnosing filesystem 
> deadlocks, and 'alltrace' is the easiest way to obtain stack traces for 
> the resulting processes that are holding locks.  You'll want to 'set 
> $lines=0' and capture the output from the serial console to a file.
>

Thanks Kris, these are exactly the clues I needed.  Since the deadlock 
during a snapshot is fairly easy to reproduce, I did so and collected this 
information below.  "alltrace" didn't work as I expected (didn't produce a 
trace), so I traced each pid associated with a locked vnode separately.

-- 
Greg


# sync;sync; date; uname -a
Tue Nov 22 20:31:07 CST 2005
FreeBSD mx17.sac.fedex.com 6.0-STABLE FreeBSD 6.0-STABLE #0: Mon Nov 21 23:46:18 CST 2005     root at mx17.sac.fedex.com:/usr/obj/usr/src/sys/MX  i386
# KDB: enter: Line break on console
[thread pid 12 tid 100004 ]
Stopped at      kdb_enter+0x30: leave 
db> set $lines=0
db> ps
   pid   proc     uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
99774 ced88000    0   673   673 0000100 new [INACTIVE] sendmail
99773 cecc7624  100   673   673 0000100 [SLPQ connec 0xced3c302][SLP] sendmail
99772 cecc7830  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
99771 cecc7a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99770 cecc7c48  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
99769 ced8820c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99768 ced88418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99767 ced44a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99766 ced88624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99765 ceb5aa3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99764 ceb5a830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99763 ced45c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99762 cec30624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99761 c8ad820c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99760 ce685000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99759 cecc8000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99758 cecc820c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99757 cecc8418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99756 c8941624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99755 cecc8830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99750 ced44c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99743 ced45000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99742 ced4520c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99741 ced45418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99740 ced45624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99739 ced45830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99738 ced45a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99737 cec3020c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99736 cec30418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99733 cecc9624 11521 99729 99726 0004000 [SLPQ piperd 0xc8e98660][SLP] awk
99731 cecc9830 11521 99729 99726 0004100 [SLPQ ufs 0xc8991d18][SLP] sendmail
99730 c896a830 11521 99728 99728 0004000 [SLPQ nanslp 0xc06596ac][SLP] perl5.8.7
99729 cec30830 11521 99726 99726 0004000 [SLPQ piperd 0xc8e47cc0][SLP] ksh
99728 cecc9a3c 11521 99725 99728 0004000 [SLPQ wait 0xcecc9a3c][SLP] sh
99726 cec30a3c 11521 99723 99726 0004000 [SLPQ wait 0xcec30a3c][SLP] sh
99725 c890920c    0   689   689 0000000 [SLPQ piperd 0xcaae0660][SLP] cron
99723 ce68d000    0   689   689 0000000 [SLPQ piperd 0xc89604c8][SLP] cron
99722 cec30c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99721 cec31000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99720 cecc9c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99719 ced1a000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99718 ced1a20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99717 ced1a418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99716 ced1a624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99715 c8e91624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99714 ced1a830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99713 ced1aa3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99712 ced1ac48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99711 cec3120c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99710 ced1e000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99709 ced1e20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99708 ced1e418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99707 ced1e624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99706 ced1e830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99705 ceb92c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99704 cebb9000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99703 cebb920c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99702 cec31418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99701 cec31624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99700 cec31830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99699 cec31a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99698 cec31c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99697 cecc9000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99696 cecc920c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99695 cecc9418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99694 cec37418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99693 cec37624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99692 cec37830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99691 cec37a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99690 cec37c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99689 cec69000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99688 cec6920c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99687 cec69418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99686 cec69624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99685 cec69830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99684 cec69a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99683 cec69c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99682 cec6e000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99681 cec6e20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99680 cec6e418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99679 cec6e624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99678 cec6e830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99677 cec6ea3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99676 cec34000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99675 cec6ec48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99674 cec6f000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99673 cec6f20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99672 cec6f418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99671 cec6f624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99670 cec6f830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99669 cec6fa3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99668 cec6fc48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99667 cec70000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99666 cec7020c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99665 cec70418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99664 cec70624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99663 cec70830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99662 cec70a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99661 cebbb20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99660 cebbb418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99659 cebbb624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99657 cebbb830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99656 cebbba3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99655 cebbbc48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99654 cec3420c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99653 cebdd000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99652 cebdd20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99651 cebdd418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99650 cec34418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99649 cebdd624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99648 cebdd830  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
99647 cebdda3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99646 cebddc48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99645 cebdf000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99644 cebdf20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99643 cebdf418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99642 cebdf624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99641 cebdf830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99640 cebdfa3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99639 cebdfc48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99638 cebe0000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99637 cebe020c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99636 cebe0418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99635 cec34624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99634 cec34830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99633 cebb9418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99632 cec34a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99631 cebb9624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99630 cebb9830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99629 cebb9a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99628 cebb9c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99627 cebba000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99626 cec34c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99625 cec35000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99624 cec3520c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99623 cec35418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99622 cec35624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99621 cec35830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99620 cec35a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99619 cec35c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99618 cec37000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99617 cebe0624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99616 cebe0830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99615 cebba20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99614 cec3720c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99613 cebe0a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99612 cebe0c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99611 cebe1000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99610 cebe120c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99609 ce68a000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99608 cebe1418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99607 cebe1624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99606 ceaca624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99605 cebe1830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99604 ceb5a000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99603 ceb5a20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99602 ceb5a418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99601 ceaca830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99600 ceb5a624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99597 ceb5ac48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99596 ceb5e000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99595 ceb5e20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99594 ceb5e418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99593 ceb5e624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99592 ceb5e830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99591 ceb5ea3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99590 ceb5ec48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99587 ceb5f000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99586 ceb5f20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99585 ceacaa3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99584 ceacac48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99583 ceb5f418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99582 ceb40000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99581 ceb5f624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99580 ceb5f830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99579 ceb5fa3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99578 cebba418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99577 ceb5fc48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99576 cebba624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99575 ceb60000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99574 ceb6020c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99573 ceb60418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99572 ceb4020c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99571 cebba830 11521 99568 99566 0004000 [SLPQ piperd 0xc8e48198][SLP] awk
99570 cebbaa3c 11521 99568 99566 0004100 [SLPQ ufs 0xc8991d18][SLP] sendmail
99569 ceb60624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99568 cebbac48 11521 99566 99566 0004000 [SLPQ piperd 0xc8960660][SLP] ksh
99566 cebbb000 11521 99564 99566 0004000 [SLPQ wait 0xcebbb000][SLP] sh
99564 ceb40418    0   689   689 0000000 [SLPQ piperd 0xce752330][SLP] cron
99562 ceb40830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99561 ceb40a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99560 ceb40c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99559 ceb43000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99558 ceb4320c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99557 ceb43418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99556 ceb43624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99555 ceb60830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99554 ceb43830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99553 ceb61830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99552 ceb61a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99551 ceb61c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99550 ceb8e000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99549 ceb60a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99548 ceb8e20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99547 ce68520c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99546 ceb43a3c   25   677   677 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99545 ceb43c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99544 ceb8e418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99543 ceb8e624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99542 ceb8e830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99541 ceb8ea3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99540 ceb8ec48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99539 ceb44000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99538 ceb92000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99537 ceb9220c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99536 ceb60c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99535 ceb92418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99534 ceb92624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99532 ceb92830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99531 ceb92a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99530 ceb4420c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99529 ceb44418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99528 ceb44624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99527 ceb44830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99526 ceb44a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99525 ceb61000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99524 ceb6120c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99523 ceb61418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99522 ceb61624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99521 c901b624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99520 ce681418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99519 ccb1b000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99518 ceb44c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99517 ceb46000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99516 ceb4620c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99515 c8ad020c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99514 c8969830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99513 ce681a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99512 ceb46418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99511 c8e90a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99510 ceb46624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99509 cc3f9a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99508 ceb46830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99507 cea68830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99506 ceb46a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99505 ceb46c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99504 cea8fc48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99503 ceac2000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99502 ceac220c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99501 ceac2418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99500 ceac2624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99499 ceac2830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99498 c901dc48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99497 c9022418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99496 c8e91000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99495 c901da3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99494 ce689000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99493 c894720c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99492 ceac2a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99491 ceac2c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99490 ceac4000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99489 ceac420c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99488 ceac4418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99487 ceac4624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99486 cea68a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99485 cea68c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99484 ceac4830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99483 ceac4a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99482 ceac4c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99481 ceac7000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99480 ceac720c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99479 ceac7418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99478 ceac7624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99477 cea8e000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99476 ceac7830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99475 ceac7a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99474 ceac7c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99473 ceac8000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99472 ceac820c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99471 ceac8418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99470 ceac8624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99469 ceac8830 11521 99467 99466 0004000 [SLPQ piperd 0xc895f660][SLP] awk
99468 ceac8a3c 11521 99467 99466 0004100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99467 ce68d20c 11521 99466 99466 0004000 [SLPQ piperd 0xc8971cc0][SLP] ksh
99466 ce689c48 11521 99465 99466 0004000 [SLPQ wait 0xce689c48][SLP] sh
99465 ceac8c48    0   689   689 0000000 [SLPQ piperd 0xc8cdf198][SLP] cron
99464 ceaca000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99463 ceaca20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99462 cea8e20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99461 cea8e418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99460 cea8e624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99459 ceaca418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99458 ce682830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99457 cea8e830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99456 cea8ea3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99455 cea8ec48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99454 cea61624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99453 cea61000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99452 cea6120c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99449 cea61830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99448 cea61a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99447 cea8f000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99446 cea61c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99445 cea62000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99444 cea6220c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99443 cea62418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99442 cea8f20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99441 cea8f418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99440 cea8f624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99439 cea62624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99438 cea8f830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99437 cea8fa3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99436 cea62830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99435 ce68da3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99434 cea62a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99433 cea62c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99432 ce68dc48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99431 ce9c7000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99430 ce9c720c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99429 cea65000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99428 ce9c7418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99427 ce9c7624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99426 cea6520c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99425 ce9c7830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99424 cea65418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99423 cea65624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99422 cc3f4a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99421 ce68920c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99420 cea65830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99419 cea65a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99418 cea65c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99417 cc3f4000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99416 cea66000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99415 cea6620c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99414 ce9c7a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99413 cea66418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99412 cea66624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99411 cea66830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99410 cea66a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99409 cea66c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99408 cea68000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99407 cea6820c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99406 ce68aa3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99405 cea68418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99404 cea68624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99403 c8e9220c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99402 cc3f9c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99401 ce682c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99400 ccb1ba3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99399 c8ad6c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99398 c894120c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99397 ce685830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99396 ce682a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99395 c86ed830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99394 ccb17418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99393 ce9c7c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99392 ce685a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99391 ccb20c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99390 ce9cb000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99389 ce68ac48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99388 c9a59a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99387 c9a59418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99386 c901b418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99384 ce9cb20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99383 ce9cb418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99382 ce9cb624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99381 c8ad8a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99380 ce9cb830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99379 ce9cba3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99378 c894620c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99377 ce688c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99376 ccb2220c  100   673   673 0000100 [SLPQ ufs 0xc8d30498][SLP] sendmail
99375 ccb20a3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99374 c901bc48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99373 c8ad620c  100   673   673 0000100 [SLPQ ufs 0xc8dc1af8][SLP] sendmail
99372 c8ad0418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99371 c8e92418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99370 c8ad0830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99369 c8e59c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99368 ccb16418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99367 c9022c48  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
99366 ce681830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99365 ce68420c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99364 ccb21a3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99363 c8e8ca3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99362 c8e58c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99361 ce682624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99360 c9022a3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99359 c896a000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99358 ce67b20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99356 ce9cbc48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99355 ce67bc48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99354 ce688000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99353 c9a5a20c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99352 c901d000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99351 c8947000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99350 cc3fa20c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99349 ce684c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99348 ce68120c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99347 c9021830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99345 ccb21624 11008  4703 99345 0004002 [SLPQ ttyin 0xcc8b3810][SLP] ksh
99344 cc3f8624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99343 c8904830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99342 c9a5a624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99341 c901ba3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99340 ccb16830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99339 c890820c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99338 c8969418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99337 c8e8c624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99334 c901b20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99333 ccb22624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99332 c8e8c000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99331 ce689a3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99330 cc3f8c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99329 ccb22000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99328 c9a5220c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99327 c8e8c20c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99326 ccb1620c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99325 c8e59830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99324 ccb20624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99323 c902220c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99322 ce68a830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99321 ce684a3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99320 ccb16000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99319 c8ad0000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99318 c896a20c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99317 c8e90418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99316 c9021624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99315 c8969000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99308 c8e8cc48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99307 ce67ba3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99306 c8946418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99305 c8e9020c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99304 ce67b830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99303 c9a59000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99302 c9021000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99301 ce67b000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99299 c896aa3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99297 ce68220c 11521 99294 99290 0004000 [SLPQ piperd 0xc895f198][SLP] awk
99296 ce689830 11521 99294 99290 0004100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99294 ccb16a3c 11521 99290 99290 0004000 [SLPQ piperd 0xce423330][SLP] ksh
99290 c901d830 11521 99286 99290 0004000 [SLPQ wait 0xc901d830][SLP] sh
99286 ce685624    0   689   689 0000000 [SLPQ piperd 0xc8960198][SLP] cron
99285 c9a5920c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99284 c86e9000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99283 c8941830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99282 c8947624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99281 ce68a20c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99280 ce67b418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99279 c9a52000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99278 c86e920c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99277 cc3f4418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99276 c896920c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99275 c890420c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99274 c8e59418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99273 c9a52624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99272 c9021418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99271 ccb22c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99270 ccb1b418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99269 cc3f420c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99268 ce682000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99267 ccb1b624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99266 ce689624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99265 c8908c48  100   673   673 0000100 [SLPQ ufs 0xc8d30278][SLP] sendmail
99263 c9a58000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99262 ce68d418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99261 ce685c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99260 c8908000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99259 ccb1b830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99258 c8947c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99257 c8946c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99256 c9a5a418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99255 ccb20830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99254 c8e9120c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99253 c8e92c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99252 ccb22418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99251 ce681000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99250 c9a52a3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99249 c8965418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99248 ce684000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99247 c86ed624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99246 ce681c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99245 ce688830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99244 c8947418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99243 ce68a624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99242 ccb22830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99241 c8ad8830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99240 ccb20000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99239 c901d20c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99238 c86edc48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99237 c9021c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99236 c8e58624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99235 c8e90c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99234 c9a58418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99233 ccb17a3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99232 c8e52830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99231 c8e90830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99230 c8904c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99229 c9a52830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99227 c8e8c418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99226 c9022624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99225 c8e92a3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99224 ce68a418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99223 c8e90624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99222 ce684830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99221 ccb17000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99220 c8e5920c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99219 c8947a3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99218 ccb16c48  100     1 99218 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99215 c8e59000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99214 c9a59830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99213 ce684418  100     1 99213 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99211 c9a58a3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99210 c9a5a000  100     1 99210 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99206 c8904000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99204 ccb1b20c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99200 ce684624  100     1 99200 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
99183 cc3fa418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99182 c901d624  100     1 99182 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99176 c8e92624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99165 c9a52418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99117 c9a5a830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99099 c901d418  100     1 99099 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99086 c896520c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
99078 cc3fac48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
98956 c8e52624  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
98921 ce68d830  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
98854 ce68820c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
98665 c9a52c48    0 98282 98665 0004002 [SLPQ suspfs 0xc86d706c][SLP] ls
98639 c8e91418 11008 98637 97958 0004102 [SLPQ ufs 0xc8991d18][SLP] mksnap_ffs
98637 c8e5820c 11008 97958 97958 0004002 [SLPQ wait 0xc8e5820c][SLP] sh
98282 c8e58418    0 98261 98282 0004002 [SLPQ pause 0xc8e5844c][SLP] ksh
98261 c8965624 11008 97022 98261 0004102 [SLPQ wait 0xc8965624][SLP] su
97958 c8941000 11008  4705 97958 0004002 [SLPQ wait 0xc8941000][SLP] dump
97439 ce68d624  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
97310 c8ad0c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
97158 c896a418  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
97022 c8e91a3c 11008  4703 97022 0004002 [SLPQ pause 0xc8e91a70][SLP] ksh
97004 cc3f8830  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
96699 ccb21418  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
96109 ccb21830  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
95970 c8904418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
95957 cc3f4830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
95932 c8ad6418  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
95907 cc3faa3c  100   673   673 0000100 [SLPQ sbwait 0xc8f5479c][SLP] sendmail
94322 cc3f9418  100   673   673 0000100 [SLPQ sbwait 0xcb4800a8][SLP] sendmail
93567 c9a58c48  100   673   673 0000100 [SLPQ pause 0xc9a58c7c][SLP] sendmail
93563 cc3f9000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
91307 c8e58a3c  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
89069 c9022000  100   673   673 0000100 [SLPQ sbwait 0xce675d2c][SLP] sendmail
87575 ccb17624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
85547 ccb21000  100   673   673 0000100 [SLPQ sbwait 0xce7cb0a8][SLP] sendmail
85524 c8946624  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
85431 c8908624  100   673   673 0000100 [SLPQ sbwait 0xcc809a64][SLP] sendmail
85005 c8909624  100   673   673 0000100 [SLPQ sbwait 0xcc809638][SLP] sendmail
83829 c8e5220c  100   673   673 0000100 [SLPQ sbwait 0xce67420c][SLP] sendmail
83436 c8e59624  100   673   673 0000100 [SLPQ sbwait 0xc94e70a8][SLP] sendmail
83282 cc3f8a3c  100   673   673 0000100 [SLPQ sbwait 0xcc40c370][SLP] sendmail
81449 ccb2020c  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
76255 c8ad8000  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
74272 ce688a3c  100   673   673 0000100 [SLPQ sbwait 0xccbb10a8][SLP] sendmail
73313 ce67b624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
71860 c9a5ac48  100   673   673 0000100 [SLPQ sbwait 0xce644e90][SLP] sendmail
69304 c9a58624  100   673   673 0000100 [SLPQ sbwait 0xce64ce90][SLP] sendmail
61830 ccb1bc48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
59550 cc3f8418  100   673   673 0000100 [SLPQ sbwait 0xce6e54d4][SLP] sendmail
56973 cc3f4c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
55659 cc3fa000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
54198 ce688418  100   673   673 0000100 [SLPQ sbwait 0xc8f540a8][SLP] sendmail
53424 c896ac48  100   673   673 0000100 [SLPQ sbwait 0xccbb1638][SLP] sendmail
50232 c8908a3c  100   673   673 0000100 [SLPQ sbwait 0xce64c900][SLP] sendmail
46440 ccb1720c  100   673   673 0000100 [SLPQ pause 0xccb17240][SLP] sendmail
45756 ccb22a3c  100   673   673 0000100 [SLPQ sbwait 0xce7abbc8][SLP] sendmail
44967 c8965a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
42202 cc3f4624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
41899 c9a59c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
35265 ccb21c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
22008 c9a5aa3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
21146 c8909830  100   673   673 0000100 [SLPQ pause 0xc8909864][SLP] sendmail
  4993 cc3f8000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
81603 cc3fa624  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
70166 cc3f820c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
64399 c86eda3c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
63092 c8ad6000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
62406 c8965000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
60658 c8946830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
59668 c8e58000  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
58412 c9a58830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
57591 c8965830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
55626 ccb16624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
54177 ccb17830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
49657 ccb17c48  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
35742 ccb2120c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
20979 cc3f9830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
  6030 c8909418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
91041 ce681624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
76748 ce689418  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
62737 c8e91830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
52019 c901b830  100   673   673 0000100 [SLPQ select 0xc0660244][SLP] sendmail
41885 c8e90000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
33339 c8965c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
25301 cc3fa830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
15180 c8946000  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
  5670 c8e52000  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
93621 c8e52c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
76951 cc3f9624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
68197 c896a624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
61840 c9022830  100 61839 61839 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
61839 c8941c48  100     1 61839 0000100 [SLPQ wait 0xc8941c48][SLP] sendmail
59681 c8ad0624  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
51464 c9a5820c  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
40069 c8e8c830  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
30519 c8909c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
22495 c8e92830  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
13893 c8909a3c  100   673   673 0000100 [SLPQ ufs 0xc8acf388][SLP] sendmail
  4772 c9a59624    0  4753  4772 0025002 stop[SUSP] ksh
  4753 c8e92000 11008  4705  4753 0025102 stop[SUSP] su
  4705 c902120c 11008  4703  4705 0004002 [SLPQ pause 0xc9021240][SLP] ksh
  4703 c8908830 11008  4700  4703 0000100 [SLPQ select 0xc0660244][SLP] screen
  4700 c9021a3c 11008  4609  4700 0004102 [SLPQ pause 0xc9021a70][SLP] screen
  4609 c8ad6a3c 11008  4600  4609 0004002 [SLPQ pause 0xc8ad6a70][SLP] ksh
  4600 c8904624 11008  4588  4588 0000100 [SLPQ select 0xc0660244][SLP] sshd
  4588 c901b000    0   668  4588 0004100 [SLPQ sbwait 0xc94ef638][SLP] sshd
  1860 c8e52418    0  1857  1860 0004002 [SLPQ ttyin 0xc8714810][SLP] ksh
  1857 c8e58830    0     1  1857 0004102 [SLPQ wait 0xc8e58830][SLP] login
  1856 c8941a3c    0     1  1856 0004002 [SLPQ ttyin 0xc870a010][SLP] getty
  1855 c8e59a3c    0     1  1855 0004002 [SLPQ ttyin 0xc8714010][SLP] getty
  1854 c8969a3c    0     1  1854 0004002 [SLPQ ttyin 0xc8716810][SLP] getty
  1853 c8e52a3c    0     1  1853 0004002 [SLPQ ttyin 0xc8714410][SLP] getty
  1844 c8e91c48    0     1  1844 0000000 [SLPQ select 0xc0660244][SLP] inetd
   689 c8ad8624    0     1   689 0000000 [SLPQ nanslp 0xc06596ac][SLP] cron
   677 c8ad8418   25     1   677 0000100 [SLPQ pause 0xc8ad844c][SLP] sendmail
   676 c8969c48  100   673   673 0000100 [SLPQ suspfs 0xc86d706c][SLP] sendmail
   673 c8941418    0     1   673 0000100 [CPU 0] sendmail
   668 c86ed418    0     1   668 0000100 [SLPQ select 0xc0660244][SLP] sshd
   650 c8ad0a3c    0     1   650 0000000 [SLPQ select 0xc0660244][SLP] ntpd
   558 c8969624   53     1   558 0000100 [SLPQ select 0xc0660244][SLP] named
   493 c8908418    0     1   493 0000000 [SLPQ select 0xc0660244][SLP] syslogd
   460 c8947830    0     1   460 0000000 [SLPQ select 0xc0660244][SLP] devd
   223 c8ad6830    0     0     0 0000204 [SLPQ mdwait 0xc8857000][SLP] md1
   214 c8904a3c    0     0     0 0000204 [SLPQ mdwait 0xc892e000][SLP] md0
   155 c8909000    0     0     0 0000204 [SLPQ m:w2 0xc88b2300][SLP] g_mirror gm1
   154 c86e9418    0     0     0 0000204 [SLPQ m:w2 0xc88eda00][SLP] g_mirror gm0
   153 c86e9624    0     0     0 0000204 [SLPQ m:w2 0xc8683200][SLP] g_mirror gm2
   152 c86e9830    0     0     0 0000204 [SLPQ - 0xe8c56cfc][SLP] schedcpu
   151 c86e9a3c    0     0     0 0000204 [SLPQ vlruwt 0xc86e9a3c][SLP] vnlru
   150 c86e9c48    0     0     0 0000204 [SLPQ syncer 0xc065941c][SLP] syncer
   149 c86ec000    0     0     0 0000204 [SLPQ psleep 0xc066078c][SLP] bufdaemon
   148 c86ec20c    0     0     0 0000204 [SLPQ pollid 0xc0658a60][SLP] idlepoll
   147 c86ec418    0     0     0 000020c [SLPQ pgzero 0xc0663f24][SLP] pagezero
   146 c86ec624    0     0     0 0000204 [SLPQ psleep 0xc0663a74][SLP] vmdaemon
   145 c86ec830    0     0     0 0000204 [SLPQ psleep 0xc0663a30][SLP] pagedaemon
   144 c86eca3c    0     0     0 0000204 [IWAIT] swi0: sio
   143 c86ecc48    0     0     0 0000204 [SLPQ cooling 0xc86f58d4][SLP] acpi_cooling0
   142 c86ed000    0     0     0 0000204 [SLPQ tzpoll 0xc07924f4][SLP] acpi_thermal
   141 c86ed20c    0     0     0 0000204 [SLPQ idle 0xc86abc74][SLP] ciss_notify0
     9 c8610a3c    0     0     0 0000204 [SLPQ - 0xc8634500][SLP] kqueue taskq
   140 c8610c48    0     0     0 0000204 [IWAIT] swi5:+
     8 c8623000    0     0     0 0000204 [SLPQ - 0xc8634680][SLP] acpi_task2
     7 c862320c    0     0     0 0000204 [SLPQ - 0xc8634680][SLP] acpi_task1
     6 c8623418    0     0     0 0000204 [SLPQ - 0xc8634680][SLP] acpi_task0
     5 c8623624    0     0     0 0000204 [SLPQ - 0xc8634700][SLP] thread taskq
   139 c8623830    0     0     0 0000204 [IWAIT] swi6:+
   138 c8623a3c    0     0     0 0000204 [IWAIT] swi6: task queue
   137 c8623c48    0     0     0 0000204 [IWAIT] swi2: cambio
   136 c8627000    0     0     0 0000204 [SLPQ - 0xc0655de0][SLP] yarrow
     4 c862720c    0     0     0 0000204 [SLPQ - 0xc0656748][SLP] g_down
     3 c8627418    0     0     0 0000204 [SLPQ - 0xc0656744][SLP] g_up
     2 c8627624    0     0     0 0000204 [SLPQ - 0xc065673c][SLP] g_event
   135 c8627830    0     0     0 0000204 [IWAIT] swi3: vm
   134 c8627a3c    0     0     0 000020c [IWAIT] swi4: clock sio
   133 c8627c48    0     0     0 0000204 [IWAIT] swi1: net
   132 c85f9624    0     0     0 0000204 [IWAIT] irq119:
   131 c85f9830    0     0     0 0000204 [IWAIT] irq118:
   130 c85f9a3c    0     0     0 0000204 [IWAIT] irq117:
   129 c85f9c48    0     0     0 0000204 [IWAIT] irq116:
   128 c860c000    0     0     0 0000204 [IWAIT] irq115:
   127 c860c20c    0     0     0 0000204 [IWAIT] irq114:
   126 c860c418    0     0     0 0000204 [IWAIT] irq113:
   125 c860c624    0     0     0 0000204 [IWAIT] irq112:
   124 c860c830    0     0     0 0000204 [IWAIT] irq111:
   123 c860ca3c    0     0     0 0000204 [IWAIT] irq110:
   122 c860cc48    0     0     0 0000204 [IWAIT] irq109:
   121 c8610000    0     0     0 0000204 [IWAIT] irq108:
   120 c861020c    0     0     0 0000204 [IWAIT] irq107:
   119 c8610418    0     0     0 0000204 [IWAIT] irq106:
   118 c8610624    0     0     0 0000204 [IWAIT] irq105:
   117 c8610830    0     0     0 0000204 [IWAIT] irq104:
   116 c85e520c    0     0     0 0000204 [IWAIT] irq103:
   115 c85e5418    0     0     0 0000204 [IWAIT] irq102:
   114 c85e5624    0     0     0 0000204 [IWAIT] irq101:
   113 c85e5830    0     0     0 0000204 [IWAIT] irq100:
   112 c85e5a3c    0     0     0 0000204 [IWAIT] irq99:
   111 c85e5c48    0     0     0 0000204 [IWAIT] irq98: em1
   110 c85f8000    0     0     0 0000204 [IWAIT] irq97: em0
   109 c85f820c    0     0     0 0000204 [IWAIT] irq96:
   108 c85f8418    0     0     0 0000204 [IWAIT] irq95:
   107 c85f8624    0     0     0 0000204 [IWAIT] irq94:
   106 c85f8830    0     0     0 0000204 [IWAIT] irq93:
   105 c85f8a3c    0     0     0 0000204 [IWAIT] irq92:
   104 c85f8c48    0     0     0 0000204 [IWAIT] irq91:
   103 c85f9000    0     0     0 0000204 [IWAIT] irq90:
   102 c85f920c    0     0     0 0000204 [IWAIT] irq89:
   101 c85f9418    0     0     0 0000204 [IWAIT] irq88:
   100 c85e1000    0     0     0 0000204 [IWAIT] irq87:
    99 c85e120c    0     0     0 0000204 [IWAIT] irq86:
    98 c85e1418    0     0     0 0000204 [IWAIT] irq85:
    97 c85e1624    0     0     0 0000204 [IWAIT] irq84:
    96 c85e1830    0     0     0 0000204 [IWAIT] irq83:
    95 c85e1a3c    0     0     0 0000204 [IWAIT] irq82:
    94 c85e1c48    0     0     0 0000204 [IWAIT] irq81:
    93 c85e4000    0     0     0 0000204 [IWAIT] irq80:
    92 c85e420c    0     0     0 0000204 [IWAIT] irq79:
    91 c85e4418    0     0     0 0000204 [IWAIT] irq78:
    90 c85e4624    0     0     0 0000204 [IWAIT] irq77:
    89 c85e4830    0     0     0 0000204 [IWAIT] irq76:
    88 c85e4a3c    0     0     0 0000204 [IWAIT] irq75:
    87 c85e4c48    0     0     0 0000204 [IWAIT] irq74:
    86 c85e5000    0     0     0 0000204 [IWAIT] irq73:
    85 c85cd000    0     0     0 0000204 [IWAIT] irq72:
    84 c85cd20c    0     0     0 0000204 [IWAIT] irq71:
    83 c85cd418    0     0     0 0000204 [IWAIT] irq70:
    82 c85cd624    0     0     0 0000204 [IWAIT] irq69:
    81 c85cd830    0     0     0 0000204 [IWAIT] irq68:
    80 c85cda3c    0     0     0 0000204 [IWAIT] irq67:
    79 c85cdc48    0     0     0 0000204 [IWAIT] irq66:
    78 c85d0000    0     0     0 0000204 [IWAIT] irq65:
    77 c85d020c    0     0     0 0000204 [IWAIT] irq64:
    76 c85d0418    0     0     0 0000204 [IWAIT] irq63:
    75 c85d0624    0     0     0 0000204 [IWAIT] irq62:
    74 c85d0830    0     0     0 0000204 [IWAIT] irq61:
    73 c85d0a3c    0     0     0 0000204 [IWAIT] irq60:
    72 c85d0c48    0     0     0 0000204 [IWAIT] irq59:
    71 c85a220c    0     0     0 0000204 [IWAIT] irq58:
    70 c85a2418    0     0     0 0000204 [IWAIT] irq57:
    69 c85a2624    0     0     0 0000204 [IWAIT] irq56:
    68 c85a2830    0     0     0 0000204 [IWAIT] irq55:
    67 c85a2a3c    0     0     0 0000204 [IWAIT] irq54:
    66 c85a2c48    0     0     0 0000204 [IWAIT] irq53:
    65 c85be000    0     0     0 0000204 [IWAIT] irq52:
    64 c85be20c    0     0     0 0000204 [IWAIT] irq51: ciss0
    63 c85be418    0     0     0 0000204 [IWAIT] irq50:
    62 c85be624    0     0     0 0000204 [IWAIT] irq49:
    61 c85be830    0     0     0 0000204 [IWAIT] irq48:
    60 c85bea3c    0     0     0 0000204 [IWAIT] irq47:
    59 c85bec48    0     0     0 0000204 [IWAIT] irq46:
    58 c8593624    0     0     0 0000204 [IWAIT] irq45:
    57 c8593830    0     0     0 0000204 [IWAIT] irq44:
    56 c8593a3c    0     0     0 0000204 [IWAIT] irq43:
    55 c8593c48    0     0     0 0000204 [IWAIT] irq42:
    54 c85a1000    0     0     0 0000204 [IWAIT] irq41:
    53 c85a120c    0     0     0 0000204 [IWAIT] irq40:
    52 c85a1418    0     0     0 0000204 [IWAIT] irq39:
    51 c85a1624    0     0     0 0000204 [IWAIT] irq38:
    50 c85a1830    0     0     0 0000204 [IWAIT] irq37:
    49 c85a1a3c    0     0     0 0000204 [IWAIT] irq36:
    48 c85a1c48    0     0     0 0000204 [IWAIT] irq35:
    47 c85a2000    0     0     0 0000204 [IWAIT] irq34:
    46 c8586c48    0     0     0 0000204 [IWAIT] irq33:
    45 c8592000    0     0     0 0000204 [IWAIT] irq32:
    44 c859220c    0     0     0 0000204 [IWAIT] irq31:
    43 c8592418    0     0     0 0000204 [IWAIT] irq30:
    42 c8592624    0     0     0 0000204 [IWAIT] irq29:
    41 c8592830    0     0     0 0000204 [IWAIT] irq28:
    40 c8592a3c    0     0     0 0000204 [IWAIT] irq27:
    39 c8592c48    0     0     0 0000204 [IWAIT] irq26: bge1
    38 c8593000    0     0     0 0000204 [IWAIT] irq25: bge0
    37 c859320c    0     0     0 0000204 [IWAIT] irq24:
    36 c8593418    0     0     0 0000204 [IWAIT] irq23:
    35 c8578624    0     0     0 0000204 [IWAIT] irq22:
    34 c8578830    0     0     0 0000204 [IWAIT] irq21:
    33 c8578a3c    0     0     0 0000204 [IWAIT] irq20:
    32 c8578c48    0     0     0 0000204 [IWAIT] irq19:
    31 c8586000    0     0     0 0000204 [IWAIT] irq18:
    30 c858620c    0     0     0 0000204 [IWAIT] irq17:
    29 c8586418    0     0     0 0000204 [IWAIT] irq16:
    28 c8586624    0     0     0 0000204 [IWAIT] irq15: ata1
    27 c8586830    0     0     0 0000204 [IWAIT] irq14: ata0
    26 c8586a3c    0     0     0 0000204 [IWAIT] irq13:
    25 c852620c    0     0     0 0000204 [IWAIT] irq12:
    24 c8526418    0     0     0 0000204 [IWAIT] irq11:
    23 c8526624    0     0     0 0000204 [IWAIT] irq10:
    22 c8526830    0     0     0 0000204 [IWAIT] irq9: acpi0
    21 c8526a3c    0     0     0 0000204 [IWAIT] irq8:
    20 c8526c48    0     0     0 0000204 [IWAIT] irq7:
    19 c8578000    0     0     0 0000204 [IWAIT] irq6:
    18 c857820c    0     0     0 0000204 [IWAIT] irq5:
    17 c8578418    0     0     0 0000204 [IWAIT] irq4: sio0
    16 c8521000    0     0     0 0000204 [IWAIT] irq3: sio1
    15 c852120c    0     0     0 0000204 [IWAIT] irq0:
    14 c8521418    0     0     0 0000204 [IWAIT] irq1: atkbd0
    13 c8521624    0     0     0 000020c [Can run] idle: cpu0
    12 c8521830    0     0     0 000020c [CPU 1] idle: cpu1
    11 c8521a3c    0     0     0 000020c [CPU 2] idle: cpu2
    10 c8521c48    0     0     0 000020c [CPU 3] idle: cpu3
     1 c8526000    0     0     1 0004200 [SLPQ wait 0xc8526000][SLP] init
     0 c0656840    0     0     0 0000200 [IWAIT] swapper
db> show lockedvnods
Locked vnodes

0xc8991cc0: tag ufs, type VDIR
     usecount 4, writecount 0, refcount 6 mountedhere 0
     flags (VV_ROOT)
     v_object 0xc922f7bc ref 0 pages 1
      lock type ufs: EXCL (count 1) by thread 0xceac9900 (pid 99468) with 3 pending
 	ino 2, on dev stripe/gs0

0xc8acf330: tag ufs, type VDIR
     usecount 972, writecount 0, refcount 974 mountedhere 0
     flags ()
     v_object 0xc8d476b4 ref 0 pages 1
      lock type ufs: EXCL (count 1) by thread 0xccb1d180 (pid 99376) with 384 pending
 	ino 2221504, on dev stripe/gs0

0xc8d30440: tag ufs, type VDIR
     usecount 2, writecount 0, refcount 4 mountedhere 0
     flags ()
     v_object 0xca292b58 ref 0 pages 1
      lock type ufs: EXCL (count 1) by thread 0xc8ad1a80 (pid 99373) with 1 pending
 	ino 2221506, on dev stripe/gs0

0xc8dc1aa0: tag ufs, type VDIR
     usecount 1, writecount 0, refcount 4 mountedhere 0
     flags ()
     v_object 0xca127ad4 ref 0 pages 40
      lock type ufs: EXCL (count 1) by thread 0xce67cd80 (pid 99361) with 1 pending
 	ino 2221507, on dev stripe/gs0

0xc8d38990: tag ufs, type VDIR
     usecount 1, writecount 0, refcount 3 mountedhere 0
     flags ()
     v_object 0xc96709cc ref 0 pages 1
      lock type ufs: EXCL (count 1) by thread 0xc86ebd80 (pid 99265)
 	ino 2221514, on dev stripe/gs0

0xc8d30220: tag ufs, type VDIR
     usecount 1, writecount 0, refcount 4 mountedhere 0
     flags ()
     v_object 0xcce0c318 ref 0 pages 38
      lock type ufs: EXCL (count 1) by thread 0xce680000 (pid 99355) with 1 pending
 	ino 2221515, on dev stripe/gs0

0xc8d11dd0: tag ufs, type VDIR
     usecount 0, writecount 0, refcount 3 mountedhere 0
     flags ()
     v_object 0xcb51ad68 ref 0 pages 40
      lock type ufs: EXCL (count 1) by thread 0xce67d900 (pid 91041)
 	ino 2221527, on dev stripe/gs0

0xc8dd7110: tag ufs, type VDIR
     usecount 0, writecount 0, refcount 3 mountedhere 0
     flags ()
     v_object 0xc8e3f420 ref 0 pages 39
      lock type ufs: EXCL (count 1) by thread 0xccb1e180 (pid 99375)
 	ino 2221545, on dev stripe/gs0

0xcd2ae990: tag ufs, type VREG
     usecount 1, writecount 0, refcount 6493 mountedhere 0
     flags ()
      lock type ufs: EXCL (count 1) by thread 0xc8e8e300 (pid 98639)
 	ino 4, on dev stripe/gs0
db> alltrace

Tracing command sendmail pid 99774 tid 100861 td 0xced87000
db> alltrace 99468 99376 99373 99361 99265 99355 91041 99375 98639

Tracing command sendmail pid 99774 tid 100861 td 0xced87000
db> trace 99468
Tracing pid 99468 tid 100572 td 0xceac9900
sched_switch(ceac9900,0,1,c6dcade6,9d4675e8) at sched_switch+0x158
mi_switch(1,0,c04d7b33,c8acf388,ebc8f930) at mi_switch+0x1d5
sleepq_switch(c8acf388,ebc8f964,c04bb9ce,c8acf388,50) at sleepq_switch+0x16f
sleepq_wait(c8acf388,50,c06186b3,0,ebc8f978) at sleepq_wait+0x11
msleep(c8acf388,c06576d4,50,c06186b3,0) at msleep+0x3d7
acquire(ebc8f9cc,40,60000,c05104f8,c8991d18) at acquire+0x89
lockmgr(c8acf388,2002,c8acf3ac,ceac9900,ebc8f9f4) at lockmgr+0x45f
vop_stdlock(ebc8fa4c,c05f9f5b,c0646cc0,ebc8fa4c,ebc8fa04) at vop_stdlock+0x2f
VOP_LOCK_APV(c0647200,ebc8fa4c,ebc8fa1c,c05f9f43,ebc8fa4c) at VOP_LOCK_APV+0x44
ffs_lock(ebc8fa4c,0,2002,c8acf330,ebc8fa68) at ffs_lock+0x19
VOP_LOCK_APV(c0646cc0,ebc8fa4c,c0653620,c8991cc0,2002) at VOP_LOCK_APV+0x44
vn_lock(c8acf330,2002,ceac9900,c8991cc0,4) at vn_lock+0x132
vget(c8acf330,2002,ceac9900,40,cc557900) at vget+0xee
cache_lookup(c8991cc0,ebc8fc74,ebc8fc88,ceac9900,cc557900) at cache_lookup+0x501
vfs_cache_lookup(ebc8fba8,ebc8fb50,c8991cc0,0,ebc8fbc4) at vfs_cache_lookup+0xa1
VOP_LOOKUP_APV(c0646cc0,ebc8fba8,ebc8fb94,ceac9900,ce6436f4) at VOP_LOOKUP_APV+0x44
lookup(ebc8fc60,c8910c00,400,ebc8fc7c,0) at lookup+0x4a8
namei(ebc8fc60,ebc8fc68,c04b4bb1,ebc8fc78,c05a4108) at namei+0x442
kern_chdir(ceac9900,bfbfcc00,0,ebc8fd30,c05ef1c1) at kern_chdir+0x42
chdir(ceac9900,ebc8fd04,4,ceac9900,8146000) at chdir+0x22
syscall(3b,3b,3b,8113ca0,0) at syscall+0x324
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (12, FreeBSD ELF32, chdir), eip = 0x88297ba3, esp = 0xbfbfbb3c, ebp = 0xbfbfd018 ---
db> trace 99376
Tracing pid 99376 tid 100414 td 0xccb1d180
sched_switch(ccb1d180,0,1,997d638c,601ad0fb) at sched_switch+0x158
mi_switch(1,0,c04d7b33,c8d30498,eba4d878) at mi_switch+0x1d5
sleepq_switch(c8d30498,eba4d8ac,c04bb9ce,c8d30498,50) at sleepq_switch+0x16f
sleepq_wait(c8d30498,50,c06186b3,0,0) at sleepq_wait+0x11
msleep(c8d30498,c065852c,50,c06186b3,0) at msleep+0x3d7
acquire(eba4d914,40,60000,c0597494,1000) at acquire+0x89
lockmgr(c8d30498,2002,c8d304bc,ccb1d180,eba4d93c) at lockmgr+0x45f
vop_stdlock(eba4d994,0,c0646cc0,eba4d994,eba4d94c) at vop_stdlock+0x2f
VOP_LOCK_APV(c0647200,eba4d994,eba4d964,c05f9f43,eba4d994) at VOP_LOCK_APV+0x44
ffs_lock(eba4d994,0,2002,c8d30440,eba4d9b0) at ffs_lock+0x19
VOP_LOCK_APV(c0646cc0,eba4d994,eba4dc1c,eba4dc08,3fff) at VOP_LOCK_APV+0x44
vn_lock(c8d30440,2002,ccb1d180,eba4d9f4,c04a4957) at vn_lock+0x132
vget(c8d30440,2002,ccb1d180,40,ce72d380) at vget+0xee
cache_lookup(c8acf330,eba4dc08,eba4dc1c,ccb1d180,ce72d380) at cache_lookup+0x501
vfs_cache_lookup(eba4daf0,eba4da98,c8acf330,ce6eda00,eba4db0c) at vfs_cache_lookup+0xa1
VOP_LOOKUP_APV(c0646cc0,eba4daf0,ccb1d180,c88a5250,0) at VOP_LOOKUP_APV+0x44
lookup(eba4dbf4,c8910000,400,eba4dc10,1) at lookup+0x4a8
namei(eba4dbf4,100,1,2ec,c926f730) at namei+0x442
kern_stat(ccb1d180,80eec20,0,eba4dc68,eba4dc94) at kern_stat+0x3d
stat(ccb1d180,eba4dd04,8,ccb1d180,80eb000) at stat+0x2f
syscall(3b,bfbf003b,bfbf003b,2000,4000) at syscall+0x324
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0x88305907, esp = 0xbfbfa52c, ebp = 0xbfbfa5c8 ---
db> trace 99373
Tracing pid 99373 tid 100232 td 0xc8ad1a80
sched_switch(c8ad1a80,0,1,d6bbcf42,9f2c12b3) at sched_switch+0x158
mi_switch(1,0,c04d7b33,c8dc1af8,eb722878) at mi_switch+0x1d5
sleepq_switch(c8dc1af8,eb7228ac,c04bb9ce,c8dc1af8,50) at sleepq_switch+0x16f
sleepq_wait(c8dc1af8,50,c06186b3,0,c88f9000) at sleepq_wait+0x11
msleep(c8dc1af8,c0658574,50,c06186b3,0) at msleep+0x3d7
acquire(eb722914,40,60000,c0597494,0) at acquire+0x89
lockmgr(c8dc1af8,2002,c8dc1b1c,c8ad1a80,eb72293c) at lockmgr+0x45f
vop_stdlock(eb722994,0,c0646cc0,eb722994,eb72294c) at vop_stdlock+0x2f
VOP_LOCK_APV(c0647200,eb722994,eb722964,c05f9f43,eb722994) at VOP_LOCK_APV+0x44
ffs_lock(eb722994,0,2002,c8dc1aa0,eb7229b0) at ffs_lock+0x19
VOP_LOCK_APV(c0646cc0,eb722994,ce9f2c78,23,0) at VOP_LOCK_APV+0x44
vn_lock(c8dc1aa0,2002,c8ad1a80,eb7229f4,c04a4957) at vn_lock+0x132
vget(c8dc1aa0,2002,c8ad1a80,40,ce7b9c00) at vget+0xee
cache_lookup(c8d30440,eb722c08,eb722c1c,c8ad1a80,ce7b9c00) at cache_lookup+0x501
vfs_cache_lookup(eb722af0,eb722a98,c8d30440,0,eb722b0c) at vfs_cache_lookup+0xa1
VOP_LOOKUP_APV(c0646cc0,eb722af0,c8ad1a80,c88a5250,0) at VOP_LOOKUP_APV+0x44
lookup(eb722bf4,c8e1b000,400,eb722c10,1) at lookup+0x4a8
namei(eb722bf4,eb722b9c,c05a1789,ce9f2bfc,0) at namei+0x442
kern_stat(c8ad1a80,8141520,0,eb722c68,0) at kern_stat+0x3d
stat(c8ad1a80,eb722d04,8,c8ad1a80,8154000) at stat+0x2f
syscall(3b,bfbf003b,bfbf003b,0,814f3c0) at syscall+0x324
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0x88305907, esp = 0xbfbfa6dc, ebp = 0xbfbfa778 ---
db> trace 99361
Tracing pid 99361 tid 100437 td 0xce67cd80
sched_switch(ce67cd80,0,1,3d4e53a2,95c40548) at sched_switch+0x158
mi_switch(1,0,c04d7b33,c86d706c,ebb00908) at mi_switch+0x1d5
sleepq_switch(c86d706c,ebb0093c,c04bb9ce,c86d706c,9f) at sleepq_switch+0x16f
sleepq_wait(c86d706c,9f,c061a026,0,c0647200) at sleepq_wait+0x11
msleep(c86d706c,c86d7044,29f,c061a026,0,c0653720,c8dc1aa0,ebb00978,ce67cd80) at msleep+0x3d7
vn_write_suspend_wait(c8dc1aa0,c86d7000,1,0,ca07f817) at vn_write_suspend_wait+0x181
ufs_inactive(ebb009c8,ebb009dc,c051b330,c0646cc0,ebb009c8) at ufs_inactive+0x1b4
VOP_INACTIVE_APV(c0646cc0,ebb009c8,a03,ebb009d0,c051054f) at VOP_INACTIVE_APV+0x3a
vinactive(c8dc1aa0,ce67cd80,ffffffdf,ebb00a24,c0513592) at vinactive+0x82
vput(c8dc1aa0,ffffffdf,2,ebb00a50,0) at vput+0x187
vn_open_cred(ebb00bc0,ebb00cc0,180,c9430580,5) at vn_open_cred+0xfb
vn_open(ebb00bc0,ebb00cc0,180,5,4b5fcfad) at vn_open+0x33
kern_open(ce67cd80,81414e0,0,a03,180) at kern_open+0xca
open(ce67cd80,ebb00d04,c,ce67cd80,8155000) at open+0x36
syscall(3b,3b,3b,0,a02) at syscall+0x324
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0x88305967, esp = 0xbfbfa40c, ebp = 0xbfbfa498 ---
db> trace 99265
Tracing pid 99265 tid 100174 td 0xc86ebd80
sched_switch(c86ebd80,0,1,86db4e3a,c0bf63cf) at sched_switch+0x158
mi_switch(1,0,c04d7b33,c8d30278,e93ee878) at mi_switch+0x1d5
sleepq_switch(c8d30278,e93ee8ac,c04bb9ce,c8d30278,50) at sleepq_switch+0x16f
sleepq_wait(c8d30278,50,c06186b3,0,0) at sleepq_wait+0x11
msleep(c8d30278,c0658724,50,c06186b3,0) at msleep+0x3d7
acquire(e93ee914,40,60000,c0597494,1000) at acquire+0x89
lockmgr(c8d30278,2002,c8d3029c,c86ebd80,e93ee93c) at lockmgr+0x45f
vop_stdlock(e93ee994,0,c0646cc0,e93ee994,e93ee94c) at vop_stdlock+0x2f
VOP_LOCK_APV(c0647200,e93ee994,e93ee964,c05f9f43,e93ee994) at VOP_LOCK_APV+0x44
ffs_lock(e93ee994,0,2002,c8d30220,e93ee9b0) at ffs_lock+0x19
VOP_LOCK_APV(c0646cc0,e93ee994,e93eec1c,e93eec08,3fff) at VOP_LOCK_APV+0x44
vn_lock(c8d30220,2002,c86ebd80,e93ee9f4,c04a4957) at vn_lock+0x132
vget(c8d30220,2002,c86ebd80,40,ce6d6d80) at vget+0xee
cache_lookup(c8d38990,e93eec08,e93eec1c,c86ebd80,ce6d6d80) at cache_lookup+0x501
vfs_cache_lookup(e93eeaf0,e93eea98,c8d38990,0,e93eeb0c) at vfs_cache_lookup+0xa1
VOP_LOOKUP_APV(c0646cc0,e93eeaf0,c86ebd80,c065b0a0,40) at VOP_LOOKUP_APV+0x44
lookup(e93eebf4,ce89b800,400,e93eec10,1) at lookup+0x4a8
namei(e93eebf4,100,e93eeb9c,5b4,c9236730) at namei+0x442
kern_stat(c86ebd80,80eec20,0,e93eec68,e93eec94) at kern_stat+0x3d
stat(c86ebd80,e93eed04,8,c86ebd80,80eb000) at stat+0x2f
syscall(3b,bfbf003b,bfbf003b,2000,4000) at syscall+0x324
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0x88305907, esp = 0xbfbfa52c, ebp = 0xbfbfa5c8 ---
db> trace 99355
Tracing pid 99355 tid 100426 td 0xce680000
sched_switch(ce680000,0,1,a8b2a004,794d29ea) at sched_switch+0x158
mi_switch(1,0,c04d7b33,c86d706c,ebb21908) at mi_switch+0x1d5
sleepq_switch(c86d706c,ebb2193c,c04bb9ce,c86d706c,9f) at sleepq_switch+0x16f
sleepq_wait(c86d706c,9f,c061a026,0,c0647200) at sleepq_wait+0x11
msleep(c86d706c,c86d7044,29f,c061a026,0,c0653720,c8d30220,ebb21978,ce680000) at msleep+0x3d7
vn_write_suspend_wait(c8d30220,c86d7000,1,0,c8e81817) at vn_write_suspend_wait+0x181
ufs_inactive(ebb219c8,ebb219dc,c051b330,c0646cc0,ebb219c8) at ufs_inactive+0x1b4
VOP_INACTIVE_APV(c0646cc0,ebb219c8,a03,ebb219d0,c051054f) at VOP_INACTIVE_APV+0x3a
vinactive(c8d30220,ce680000,ffffffdf,ebb21a24,c0513592) at vinactive+0x82
vput(c8d30220,ffffffdf,2,ebb21a50,0) at vput+0x187
vn_open_cred(ebb21bc0,ebb21cc0,180,c9296280,5) at vn_open_cred+0xfb
vn_open(ebb21bc0,ebb21cc0,180,5,97303610) at vn_open+0x33
kern_open(ce680000,8141500,0,a03,180) at kern_open+0xca
open(ce680000,ebb21d04,c,ce680000,8151000) at open+0x36
syscall(bfbf003b,bfbf003b,bfbf003b,0,a02) at syscall+0x324
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0x88305967, esp = 0xbfbfa5ac, ebp = 0xbfbfa638 ---
db> trace 91041
Tracing pid 91041 tid 100430 td 0xce67d900
sched_switch(ce67d900,0,1,5100ec54,ba61202) at sched_switch+0x158
mi_switch(1,0,c04d7b33,c86d706c,ebb15908) at mi_switch+0x1d5
sleepq_switch(c86d706c,ebb1593c,c04bb9ce,c86d706c,9f) at sleepq_switch+0x16f
sleepq_wait(c86d706c,9f,c061a026,0,c0647200) at sleepq_wait+0x11
msleep(c86d706c,c86d7044,29f,c061a026,0,c0653720,c8d11dd0,ebb15978,ce67d900) at msleep+0x3d7
vn_write_suspend_wait(c8d11dd0,c86d7000,1,0,c8ced017) at vn_write_suspend_wait+0x181
ufs_inactive(ebb159c8,ebb159dc,c051b330,c0646cc0,ebb159c8) at ufs_inactive+0x1b4
VOP_INACTIVE_APV(c0646cc0,ebb159c8,a22,ebb159d0,c051054f) at VOP_INACTIVE_APV+0x3a
vinactive(c8d11dd0,ce67d900,ffffffdf,ebb15a24,c0513592) at vinactive+0x82
vput(c8d11dd0,ffffffdf,2,0,0) at vput+0x187
vn_open_cred(ebb15bc0,ebb15cc0,180,c8fc5d80,8) at vn_open_cred+0xfb
vn_open(ebb15bc0,ebb15cc0,180,8,c0653120) at vn_open+0x33
kern_open(ce67d900,bfbfa400,0,a22,180) at kern_open+0xca
open(ce67d900,ebb15d04,c,ce67d900,ce67d900) at open+0x36
syscall(80e003b,3b,bfbf003b,8108d00,0) at syscall+0x324
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0x88305967, esp = 0xbfbf976c, ebp = 0xbfbfb118 ---
db> trace 99375
Tracing pid 99375 tid 100404 td 0xccb1e180
sched_switch(ccb1e180,0,1,2efe85fa,5fbad7c4) at sched_switch+0x158
mi_switch(1,0,c04d7b33,c86d706c,eba6b908) at mi_switch+0x1d5
sleepq_switch(c86d706c,eba6b93c,c04bb9ce,c86d706c,9f) at sleepq_switch+0x16f
sleepq_wait(c86d706c,9f,c061a026,0,c0647200) at sleepq_wait+0x11
msleep(c86d706c,c86d7044,29f,c061a026,0,c0653720,c8dd7110,eba6b978,ccb1e180) at msleep+0x3d7
vn_write_suspend_wait(c8dd7110,c86d7000,1,0,c8916417) at vn_write_suspend_wait+0x181
ufs_inactive(eba6b9c8,eba6b9dc,c051b330,c0646cc0,eba6b9c8) at ufs_inactive+0x1b4
VOP_INACTIVE_APV(c0646cc0,eba6b9c8,a03,eba6b9d0,c051054f) at VOP_INACTIVE_APV+0x3a
vinactive(c8dd7110,ccb1e180,ffffffdf,eba6ba24,c0513592) at vinactive+0x82
vput(c8dd7110,ffffffdf,2,eba6ba50,0) at vput+0x187
vn_open_cred(eba6bbc0,eba6bcc0,180,ccb73b80,5) at vn_open_cred+0xfb
vn_open(eba6bbc0,eba6bcc0,180,5,188c500) at vn_open+0x33
kern_open(ccb1e180,8141560,0,a03,180) at kern_open+0xca
open(ccb1e180,eba6bd04,c,ccb1e180,8152000) at open+0x36
syscall(3b,bfbf003b,bfbf003b,0,a02) at syscall+0x324
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0x88305967, esp = 0xbfbfa5ac, ebp = 0xbfbfa638 ---
db> trace 98639
Tracing pid 98639 tid 100282 td 0xc8e8e300
sched_switch(c8e8e300,0,1,cc167362,733b6597) at sched_switch+0x158
mi_switch(1,0,c04d7b33,c8991d18,eb80c544) at mi_switch+0x1d5
sleepq_switch(c8991d18,eb80c578,c04bb9ce,c8991d18,50) at sleepq_switch+0x16f
sleepq_wait(c8991d18,50,c06186b3,0,c065ae80) at sleepq_wait+0x11
msleep(c8991d18,c065888c,50,c06186b3,0) at msleep+0x3d7
acquire(eb80c5e0,40,60000,c8e8e300,0) at acquire+0x89
lockmgr(c8991d18,2002,c8991d3c,c8e8e300,eb80c608) at lockmgr+0x45f
vop_stdlock(eb80c660,0,c0646cc0,eb80c660,eb80c618) at vop_stdlock+0x2f
VOP_LOCK_APV(c0647200,eb80c660,eb80c630,c05f9f43,eb80c660) at VOP_LOCK_APV+0x44
ffs_lock(eb80c660,0,2002,c8991cc0,eb80c67c) at ffs_lock+0x19
VOP_LOCK_APV(c0646cc0,eb80c660,c06402e0,eb80c7d0,0) at VOP_LOCK_APV+0x44
vn_lock(c8991cc0,2002,c8e8e300,4000,c851f300) at vn_lock+0x132
ffs_snapshot(c86d7000,cc978e00,eb80c9a4,6c,eb80c964) at ffs_snapshot+0x152b
ffs_mount(c86d7000,c8e8e300,0,c8e8e300,c9f71bb0) at ffs_mount+0x9af
vfs_domount(c8e8e300,c88a8760,c88a8870,11211000,c9a77510) at vfs_domount+0x728
vfs_donmount(c8e8e300,11211000,eb80cbec,c9314580,e) at vfs_donmount+0x12e
kernel_mount(c8737b80,11211000,eb80cc30,6c,bfbfe8c8) at kernel_mount+0x46
ffs_cmount(c8737b80,bfbfe110,11211000,c8e8e300,0) at ffs_cmount+0x85
mount(c8e8e300,eb80cd04,10,c8e8e300,8052000) at mount+0x21b
syscall(3b,3b,3b,8814819c,bfbfe0b0) at syscall+0x324
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (21, FreeBSD ELF32, mount), eip = 0x880b8b03, esp = 0xbfbfe07c, ebp = 0xbfbfe778 ---
db> continue

# shutdown -r now
Shutdown NOW!
shutdown: [pid 99927]

*** FINAL System shutdown message from root at mx17.sac.fedex.com *** 
System going down IMMEDIATELY


Nov 22 20:31:48 mx17 shutdown: reboot by root: 
# 
System shutdown time has arrived
...


Below are the names of the locked inodes.  Note that mqueue/.hoststat is 
the mount point for a memory filesystem created via mdmfs(8).

# cd /var/spool
# find . \( -inum 2 -o -inum 2221504 -o -inum 2221506 -o -inum 2221507 -o -inum 2221514 -o -inum 2221515 -o -inum 2221527 -o -inum 2221545 -o -inum 4 \) -print | xargs ls -id
       2 .                       2221507 ./mqueue/q00/df
       4 ./.snap/dump_snapshot   2221514 ./mqueue/q03
2221504 ./mqueue                2221515 ./mqueue/q03/df
       2 ./mqueue/.hoststat      2221527 ./mqueue/q07/qf
2221506 ./mqueue/q00            2221545 ./mqueue/q09/df


More information about the freebsd-stable mailing list