6.1 quota issues

[Charles Sprickman]

Replying to myself and top-posting...

A very helpful person contacted me off-list and pointed me to this PR that 
dates back to 2.2:

http://www.freebsd.org/cgi/query-pr.cgi?pr=2325

I'm going to submit a followup to that so that whomever claims it can see 
that it persists into at least 6.1.

In short, the machine I was rsyncing from had some very high UIDs and 
these seem to trip up the quota code.  The big hint was the 4GB+ 
quota.user file.  I'm still doing some more testing, but so far it looks 
very much like this bug was the root of all my problems.

Thanks,

Charles

On Fri, 7 Jul 2006, Charles Sprickman wrote:

> Hello all,
>
> I'm in the process of rolling out a new shell server and for numerous reasons 
> have decided 6.x is the best fit (jail improvements, SMP improvements, 3Ware 
> driver, pf).  The shell server is within a jail, and the uids there are 
> unique so that quotas remain sane.  There are about 5000 active accounts 
> using about 40GB of a 210GB partition.  The quota.user file is about 4GB.
>
> I just started work on getting quotas setup for everyone after rsyncing all 
> the homedirs from the old server over.  At first, all seemed well, then I ran 
> into a few issues on subsequent rsyncs.  I had people with large (1GB+) 
> homedirs and quotas in the 1GB-4GB range and as rsync was chowning the files 
> to the users it was throwing errors about "quota exceeded".  Here's a brief 
> example that illustrates what I was seeing:
>
> ot at beta[/home/staff/micro/tmp]# quota micro
> Disk quotas for user micro (uid 5315):
> Filesystem   usage   quota   limit   grace   files   quota   limit grace
>     /       1630026 3000000 3100000         13393       0       0
> root at beta[/home/staff/micro/tmp]# chown micro index.html
> chown: index.html: Disc quota exceeded
> root at beta[/home/staff/micro/tmp]#
>
> I know in the past when I've seen inconsistencies indicating that I needed a 
> manual run of quotacheck, they would show up in the output of the quota 
> command; ie: the "quota" command would show the user had more usage than "du" 
> would indicate.  The above example is a bit odd - "quota" shows that he's 
> well within his limits, but the kernel thinks otherwise.
>
> Thinking it would be a good idea to stop the jails, turn off quotas, umount 
> the partition, fsck it, mount it and then run quotacheck, I found more 
> problems.  My first run of quotacheck ran for a few minutes, reported many 
> inconsistencies and then sat there for quite some time before spitting this 
> out:
>
> quotacheck: /jails/quota.user: seek failed: Invalid argument
>
> Trying again, it reported the same inconsistencies then sat there for more 
> than an hour taking up all the available CPU on the box until I killed it. 
> The mtime on quota.user had not changed during the run.
>
> Running it yet again now gives me this:
>
> /jails:          fixed: inodes 27 -> 0  blocks 156 -> 0
> quotacheck: /jails/quota.user: seek failed: Invalid argument
> THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY:
>        /dev/twed0s1g (/jails)
>
> For now I can live without quotas, but if there's anything I can test from 
> -stable that might address this I'd like to try it.  I'd say this thing is 
> still a good month from going live since we have lots of dependancy mess on 
> the old box to clean up before cutting over.
>
> Any ideas what's going on here?  Is this related to the large number of users 
> and the size of the partition?  I've seen some of the discussions about 
> snapshots + quotas, but that seems like an entirely different issue. For the 
> time being I've killed "background_fsck" and "check_quotas" in rc.conf, and 
> I'll avoid dumping that fs with the snapshot flag.
>
> What other information can I provide to help better define where this bug 
> lives?
>
> Thanks,
>
> Charles
>