Jail is pausing . . Ideas please. . ?

Sat Apr 7 13:16:29 UTC 2007

Troy Kocher <tkocher at mtadistributors.com> wrote:
>
> Listers,
> 
> Currently I am having some strange issues with regard to a jail
> pausing, hoping someone here might have some ideas. .
> Here is my Usenet post. . :
> 
> I am running FreeBSD 6.1-STABLE (SMP),  and the the system seems to be
> pausing.  System details:
> 
> I have da0, da1, da2, da3, each 500GB, I'm also using GEOM_CONCAT to
> concatenation, amd64 SMP kernel, and 16GB of ram. Running 7 jails,
> primarily running apache, samba, postfix, pgsql.
> 
> On what appears to be random occasions (usually several times in 5m)
> the system seems to pause.  For instance, vipw takes >40s to respond,
> or the smbd which clients use for their mailbox.pst ignores requests
> from outlook to act on the file.  Then moments later it is happy
> again, and begins working normally.  I have been monitoring top while
> it happens and it appears like it is doing very little. . ie. .
> 
> last pid: 75014;  load averages:  0.00,  0.02,  
> 0.07                       up 203+07:20:57 15:24:53
> 246 processes: 1 running, 244 sleeping, 1 stopped
> CPU states:  0.0% user,  0.0% nice,  0.2% system,  0.1% interrupt,  
> 99.7% idle
> Mem: 967M Active, 13G Inact, 320M Wired, 782M Cache, 214M Buf, 569M Free
> Swap: 4096M Total, 2504K Used, 4093M Free
> 
> Tried running nice -20 vipw and it still took some time for it to
> run.  Could it be a file locking issue?. .
> 
> Any thoughts or ideas on further troubleshooting would really be
> appreciated
> 
> ----------
> Since that post it actually appears to only be happening in one jail
> called drzoe.  The host system seems to be working properly during
> these slow downs
> 
> Other things I've considered:
> 1)  Is there an upper limit to the number of connections a NIC can
> support?  Am I exceeding it? NiC & Switches aren't showing any packet
> loss.

There's a limit to everything.  What does "sockstat -4 | wc" give you?
I seriously doubt you're hitting any limit there, but it's possible.

> 2)  Am I running out of IO, to and from the disks?  Tried looking at
> iostat, but I'm exactly sure what a problem would look like.  Seems
> like this wouldn't be jail specific

I prefer using systat to watch this behaviour as it happens.  The
vmstat screen is particularly useful.  See the man page.

> Give it seems to be limited to this jail it seems unlikely to be  
> hardware. .

Based on your problems with vipw, it sounds like you have a lot of
processes contending for write access to the password file.  The
next time you see the problem, execute "fstat /etc/master.passwd"
and see how many processes are accessing it and what they are.
(Don't get jailbrained.  Execute fstat /etc/master.passwd from
within the jail, or execute fstat /path/to/jail/etc/master.passwd
from the host :)

HTH

> 
> from rc.conf
> 
> jail_enable="YES"
> jail_list="droutward drinward database drzoe development drimage drmail"
> # Disaster recovery setup for drzoe
> jail_drzoe_rootdir="/usr/home/drzoe-jail"
> jail_drzoe_hostname="drzoe.mtadistributors.com"
> jail_drzoe_ip="10.0.0.115"
> jail_drzoe_exec_start="/bin/sh /etc/rc"
> jail_drzoe_exec_stop="/bin/sh /etc/rc.shutdown"
> jail_drzoe_devfs_enable="YES"
> 
> [root at drzoe /]#pkg_info
> autoconf-2.59_2     Automatically configure source code on many Un*x
> platforms
> bash-3.1.10_1       The GNU Project's Bourne Again SHell
> bsdpan-Filesys-Virtual-0.05 Filesys::Virtual - Perl extension to
> provide a framework fo
> bsdpan-Filesys-Virtual-Plain-0.08 Filesys::Virtual::Plain - A Plain
> virtual filesystem
> bsdpan-Net-DAV-Server-1.28 Net::DAV::Server - Provide a DAV Server
> cups-base-1.1.23.0_8 The Common UNIX Printing System: headers, libs,
> & daemons
> cvsup-without-gui-16.1h_2 General network file distribution system
> optimized for CVS
> elinks-0.11.1       Elinks - links text WWW browser with enhancements
> gettext-0.14.5_1    GNU gettext package
> gmake-3.81_1        GNU version of 'make' utility
> gnutls-1.2.9        GNU Transport Layer Security library
> help2man-1.36.4_1   Automatically generating simple manual pages from
> program o
> identify-0.7        Client side ident protocol daemon wrapper
> jbigkit-1.6         Lossless compression for bi-level images such as
> scanned pa
> jpeg-6b_3           IJG's jpeg compression utilities
> libgcrypt-1.2.2     "General purpose crypto library based on code
> used in GnuPG
> libgpg-error-1.1    Common error values for all GnuPG components
> libiconv-1.9.2_1    A character set conversion library
> m4-1.4.8_1          GNU m4
> netpbm-10.26.41     A toolkit for conversion of images between
> different format
> p5-Authen-PAM-0.14  A Perl interface to the PAM library
> p5-Net-SSLeay-1.30_1 Perl5 interface to SSL
> p5-gettext-1.05_1   Message handling functions
> pcre-6.6_1          Perl Compatible Regular Expressions library
> perl-5.8.7_2        Practical Extraction and Report Language
> pkgconfig-0.20      A utility to retrieve information about installed
> libraries
> png-1.2.8_3         Library for manipulating PNG images
> popt-1.7            A getopt(3) like library with a number of
> enhancements, fro
> portaudit-0.5.10    Checks installed ports against a list of security
> vulnerabi
> postgresql-client-8.1.4 PostgreSQL database (client)
> proftpd-1.3.1.r2_3  Highly configurable ftp daemon
> rsync-2.6.7_1       A network file distribution/synchronization utility
> samba-2.2.12_2      A free SMB and CIFS client and server for UNIX
> tiff-3.8.0          Tools and library routines for working with TIFF
> images
> unison-2.13.16_1    A user-level file synchronization tool
> usermin-1.220_1     Web-based interface for performing some user tasks
> vim-lite-7.0.66     Vi "workalike", with many additional features
> (Lite package
> webmin-1.290        Web-based interface for system administration for
> Unix
> 
> [root at drzoe /]# portaudit
> Affected package: gnutls-1.2.9
> Type of problem: gnutls -- RSA Signature Forgery Vulnerability.
> Reference: <http://www.FreeBSD.org/ports/portaudit/
> 64bf6234-520d-11db-8f1a-000a48049292.html>
> 
> Affected package: samba-2.2.12_2
> Type of problem: samba -- integer overflow vulnerability.
> Reference: <http://www.FreeBSD.org/ports/portaudit/3b3676be-52e1-11d9-
> a9e7-0001020eed82.html>
> 
> 2 problem(s) in your installed packages found.
> 
> Any ideas you may have on troubleshooting or better yet what is  
> causing it  would be
> really appreciated.
> 
> Troy Kocher
> MTA Distributors
> tkocher(at)mtadistributors(dot)com
> 
> 
> 
> _________________________________________________
> Scanned by IBM Email Security Management Services 
> powered by MessageLabs.
> _________________________________________________
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"

-- 
Bill Moran
http://www.potentialtech.com