6.1 quota issues

[Charles Sprickman]

Hello all,

I'm in the process of rolling out a new shell server and for numerous 
reasons have decided 6.x is the best fit (jail improvements, SMP 
improvements, 3Ware driver, pf).  The shell server is within a jail, and 
the uids there are unique so that quotas remain sane.  There are about 
5000 active accounts using about 40GB of a 210GB partition.  The 
quota.user file is about 4GB.

I just started work on getting quotas setup for everyone after rsyncing 
all the homedirs from the old server over.  At first, all seemed well, 
then I ran into a few issues on subsequent rsyncs.  I had people with 
large (1GB+) homedirs and quotas in the 1GB-4GB range and as rsync was 
chowning the files to the users it was throwing errors about "quota 
exceeded".  Here's a brief example that illustrates what I was seeing:

ot at beta[/home/staff/micro/tmp]# quota micro
Disk quotas for user micro (uid 5315):
Filesystem   usage   quota   limit   grace   files   quota   limit grace
      /       1630026 3000000 3100000         13393       0       0
root at beta[/home/staff/micro/tmp]# chown micro index.html
chown: index.html: Disc quota exceeded
root at beta[/home/staff/micro/tmp]#

I know in the past when I've seen inconsistencies indicating that I needed 
a manual run of quotacheck, they would show up in the output of the quota 
command; ie: the "quota" command would show the user had more usage than 
"du" would indicate.  The above example is a bit odd - "quota" shows that 
he's well within his limits, but the kernel thinks otherwise.

Thinking it would be a good idea to stop the jails, turn off quotas, 
umount the partition, fsck it, mount it and then run quotacheck, I found 
more problems.  My first run of quotacheck ran for a few minutes, reported 
many inconsistencies and then sat there for quite some time before 
spitting this out:

quotacheck: /jails/quota.user: seek failed: Invalid argument

Trying again, it reported the same inconsistencies then sat there for more 
than an hour taking up all the available CPU on the box until I killed it. 
The mtime on quota.user had not changed during the run.

Running it yet again now gives me this:

/jails:          fixed: inodes 27 -> 0  blocks 156 -> 0
quotacheck: /jails/quota.user: seek failed: Invalid argument
THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY:
         /dev/twed0s1g (/jails)

For now I can live without quotas, but if there's anything I can test from 
-stable that might address this I'd like to try it.  I'd say this thing is 
still a good month from going live since we have lots of dependancy mess 
on the old box to clean up before cutting over.

Any ideas what's going on here?  Is this related to the large number of 
users and the size of the partition?  I've seen some of the discussions 
about snapshots + quotas, but that seems like an entirely different issue. 
For the time being I've killed "background_fsck" and "check_quotas" in 
rc.conf, and I'll avoid dumping that fs with the snapshot flag.

What other information can I provide to help better define where this bug 
lives?

Thanks,

Charles