interesting device full issue on jail host machine

[Chad Leigh -- Shire.Net LLC]

pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full
pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full
pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full
pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full
pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full
pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full
pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full
pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full
pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full
pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full
pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full
pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full
pid 80941 (tcsh), uid 5051 inumber 166876 on /local/jails/jail1:  
filesystem full

Ok, I had a billion (several complete system logs worth in one  
morning) of the above message in my system log of a machine used to  
host jails.  This is on a 5.3-R machine. The file system mentioned, / 
local/jail/jail1  is a /dev/md -- md(4) type file system used to  
constrain the user in that specific jail to a certain amount of space  
and to make it easy to dump that filesystem for backups.

I don't know what the user of that jail was doing or when it started  
happening.  Is the "inumber 166876" an inode inside the filesystem?   
Or what does it refer to?

I don't know why it continuously stuck that message thousands of  
times in the system log but I killed the process with that PID and  
all was well.

My interesting issue is that on ssh login to OTHER JAILS on other / 
dev/md type filesystems, that had nothing to do with that jail or  
filesystem, login was prevented as the login process would print a  
similar message  to the one above in the login window over and over  
and over and would never complete login.

/local/jails/jail1: write failed, filesystem is full

/local/jails/jail1: write failed, filesystem is full

/local/jails/jail1: write failed, filesystem is full

/local/jails/jail1: write failed, filesystem is full

/local/jails/jail1: write failed, filesystem is full

Only when I killed the offending process did it allow the login to  
continue.  However, this seemed to be hit or miss.  Sometimes an ssh  
login to a jail (not the offending jail) would work and sometimes it  
would do the above.  Why would this issue with this one jail file  
system affect jails that had nothing to do with the full filesystem  
and what  happens at login that would dump these console type  
messages to the ssh login window?

Thanks
Chad



---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net