Jail is pausing . . Ideas please. . ?

Fri Apr 6 22:00:51 UTC 2007

Listers,

Currently I am having some strange issues with regard to a jail
pausing, hoping someone here might have some ideas. .
Here is my Usenet post. . :

I am running FreeBSD 6.1-STABLE (SMP),  and the the system seems to be
pausing.  System details:

I have da0, da1, da2, da3, each 500GB, I'm also using GEOM_CONCAT to
concatenation, amd64 SMP kernel, and 16GB of ram. Running 7 jails,
primarily running apache, samba, postfix, pgsql.

On what appears to be random occasions (usually several times in 5m)
the system seems to pause.  For instance, vipw takes >40s to respond,
or the smbd which clients use for their mailbox.pst ignores requests
from outlook to act on the file.  Then moments later it is happy
again, and begins working normally.  I have been monitoring top while
it happens and it appears like it is doing very little. . ie. .

last pid: 75014;  load averages:  0.00,  0.02,  
0.07                       up 203+07:20:57 15:24:53
246 processes: 1 running, 244 sleeping, 1 stopped
CPU states:  0.0% user,  0.0% nice,  0.2% system,  0.1% interrupt,  
99.7% idle
Mem: 967M Active, 13G Inact, 320M Wired, 782M Cache, 214M Buf, 569M Free
Swap: 4096M Total, 2504K Used, 4093M Free

Tried running nice -20 vipw and it still took some time for it to
run.  Could it be a file locking issue?. .

Any thoughts or ideas on further troubleshooting would really be
appreciated

----------
Since that post it actually appears to only be happening in one jail
called drzoe.  The host system seems to be working properly during
these slow downs

Other things I've considered:
1)  Is there an upper limit to the number of connections a NIC can
support?  Am I exceeding it? NiC & Switches aren't showing any packet
loss.
2)  Am I running out of IO, to and from the disks?  Tried looking at
iostat, but I'm exactly sure what a problem would look like.  Seems
like this wouldn't be jail specific

Give it seems to be limited to this jail it seems unlikely to be  
hardware. .

from rc.conf

jail_enable="YES"
jail_list="droutward drinward database drzoe development drimage drmail"
# Disaster recovery setup for drzoe
jail_drzoe_rootdir="/usr/home/drzoe-jail"
jail_drzoe_hostname="drzoe.mtadistributors.com"
jail_drzoe_ip="10.0.0.115"
jail_drzoe_exec_start="/bin/sh /etc/rc"
jail_drzoe_exec_stop="/bin/sh /etc/rc.shutdown"
jail_drzoe_devfs_enable="YES"

[root at drzoe /]#pkg_info
autoconf-2.59_2     Automatically configure source code on many Un*x
platforms
bash-3.1.10_1       The GNU Project's Bourne Again SHell
bsdpan-Filesys-Virtual-0.05 Filesys::Virtual - Perl extension to
provide a framework fo
bsdpan-Filesys-Virtual-Plain-0.08 Filesys::Virtual::Plain - A Plain
virtual filesystem
bsdpan-Net-DAV-Server-1.28 Net::DAV::Server - Provide a DAV Server
cups-base-1.1.23.0_8 The Common UNIX Printing System: headers, libs,
& daemons
cvsup-without-gui-16.1h_2 General network file distribution system
optimized for CVS
elinks-0.11.1       Elinks - links text WWW browser with enhancements
gettext-0.14.5_1    GNU gettext package
gmake-3.81_1        GNU version of 'make' utility
gnutls-1.2.9        GNU Transport Layer Security library
help2man-1.36.4_1   Automatically generating simple manual pages from
program o
identify-0.7        Client side ident protocol daemon wrapper
jbigkit-1.6         Lossless compression for bi-level images such as
scanned pa
jpeg-6b_3           IJG's jpeg compression utilities
libgcrypt-1.2.2     "General purpose crypto library based on code
used in GnuPG
libgpg-error-1.1    Common error values for all GnuPG components
libiconv-1.9.2_1    A character set conversion library
m4-1.4.8_1          GNU m4
netpbm-10.26.41     A toolkit for conversion of images between
different format
p5-Authen-PAM-0.14  A Perl interface to the PAM library
p5-Net-SSLeay-1.30_1 Perl5 interface to SSL
p5-gettext-1.05_1   Message handling functions
pcre-6.6_1          Perl Compatible Regular Expressions library
perl-5.8.7_2        Practical Extraction and Report Language
pkgconfig-0.20      A utility to retrieve information about installed
libraries
png-1.2.8_3         Library for manipulating PNG images
popt-1.7            A getopt(3) like library with a number of
enhancements, fro
portaudit-0.5.10    Checks installed ports against a list of security
vulnerabi
postgresql-client-8.1.4 PostgreSQL database (client)
proftpd-1.3.1.r2_3  Highly configurable ftp daemon
rsync-2.6.7_1       A network file distribution/synchronization utility
samba-2.2.12_2      A free SMB and CIFS client and server for UNIX
tiff-3.8.0          Tools and library routines for working with TIFF
images
unison-2.13.16_1    A user-level file synchronization tool
usermin-1.220_1     Web-based interface for performing some user tasks
vim-lite-7.0.66     Vi "workalike", with many additional features
(Lite package
webmin-1.290        Web-based interface for system administration for
Unix

[root at drzoe /]# portaudit
Affected package: gnutls-1.2.9
Type of problem: gnutls -- RSA Signature Forgery Vulnerability.
Reference: <http://www.FreeBSD.org/ports/portaudit/
64bf6234-520d-11db-8f1a-000a48049292.html>

Affected package: samba-2.2.12_2
Type of problem: samba -- integer overflow vulnerability.
Reference: <http://www.FreeBSD.org/ports/portaudit/3b3676be-52e1-11d9-
a9e7-0001020eed82.html>

2 problem(s) in your installed packages found.

Any ideas you may have on troubleshooting or better yet what is  
causing it  would be
really appreciated.

Troy Kocher
MTA Distributors
tkocher(at)mtadistributors(dot)com

_________________________________________________
Scanned by IBM Email Security Management Services 
powered by MessageLabs.
_________________________________________________