Jails not quite stable..

[dap99 at i-55.com]

I am working on a project to move various services running directly under
FreeBSD 4.8-REL to run under jails on the same servers. Setting up the jails
is no problem at all (I can follow manpages), and bringing the jails up
using either '/bin/sh /etc/rc' or '/usr/local/sbin/jailer' works as well.
Basically, I can get a jail up and running with the desired service.

However, I've found that jails are producing some problems during my
testing.

1. On several occassions I have been unable to kill a process in a jail,
even with a 'kill -9' from inside or outside the jail as root.

europa# ps aux|grep J
root  90423  0.0  0.4  1268  920  p0- DJ   10:44PM   0:00.01 /bin/csh
europa# kill 90423
europa# ps aux | grep J
root  90423  0.0  0.4  1268  920  p0- DJ   10:44PM   0:00.01 /bin/csh
europa# kill -9 90423
europa# ps aux | grep J
root  90423  0.0  0.4  1268  920  p0- DJ   10:44PM   0:00.01 /bin/csh

(If I reboot this machine it will probably hang. See below.)

2. On one occasion (and I haven't attempted to replicate this), I tried a
server reboot (with 'reboot') after a jailed process would not die, and the
server promptly went offline to never return. I had to have someone hard
reset the server. Unfortunately, I am not sure what was on the screen as I
wasn't around. I can say though that the server was pingable but not
reachable otherwise.

3. If a process is hung, such as /bin/csh, then odd things happen when
accessing the location of the jail:

# cd /dsk/jails/
# ll
total 51684
drwxr-xr-x  13 root  wheel       512 Jan 10 22:17 mail1
-rw-------   1 root  wheel  52896075 Jan 10 20:47 skel.tgz
# cd mail1europa
# ll
(ls just hangs at this point)

I have to kill my ssh session using ~. to get out of this. Note that
/dsk/jails/mail1 is not mounted via NFS. It's on the actual local disk.

These problems are reproducible across machines running both FreeBSD 4.8-REL
and FreeBSD 4.9-STABLE:

# uname -v
FreeBSD 4.9-STABLE #0:

I WANT to use jails to host most of our services, if for no other reason
than the increase manageability, but there just seems to be a stability
issue here. I realize I will get a lot of "It works for me", but again,
these problems are reproducible, so I'm sure someone else has seen it. :)

Thoughts on this?