[9.3 PRE] filesystem full even if there are free descriptors and disk space

Sat May 17 12:45:32 UTC 2014

Damian Danielecki wrote:
> I am receiving errors like this on heavily loaded FreeBSD
> 9.3-PRERELEASE nfs server:
> May 17 05:01:02 nfsd kernel: pid 4173 (nfsd), uid 0 inumber 185391682
> on /exports: filesystem full
> 
Do you have -maproot=root on the line for /exports in /etc/exports?

If not, it might be running into a quota for "nobody", which is what
"uid 0" is mapped to by default.

rick

> Filesystem is newly created and server newly installed with custom
> minimalist kernel&world. I am experienced user.
> There are many free inodes and much disk space:
> 
> # df -i
> Filesystem          1K-blocks       Used      Avail Capacity  iused
>  ifree %iused  Mounted on
> /dev/mirror/gm0s1a  473048844    3449532  431755408     1%   306673
> 60848397    1%   /
> devfs                       1          1          0   100%        0
>      0  100%   /dev
> /dev/da0p1         3783567612 1772595572 1708286632    51% 20964037
> 468035769    4%   /exports
> 
> # df -h
> Filesystem            Size    Used   Avail Capacity  Mounted on
> /dev/mirror/gm0s1a    451G    3.3G    411G     1%    /
> devfs                 1.0k    1.0k      0B   100%    /dev
> /dev/da0p1            3.5T    1.7T    1.6T    51%    /exports
> 
> # uname -a
> FreeBSD nfsd.xxx.pl 9.3-PRERELEASE FreeBSD 9.3-PRERELEASE #5: Fri May
> 16 15:41:36 CEST 2014
> root at nfsd.xxx.pl:/usr/obj/usr/src/sys/FREEBSD9  amd64
> 
> Filesystem is clean and not fragmented.
> For sure fsck has been done on unmounted filesystem.
> I see these errors also after fsck.
> 
> # fsck -t ufs -y /dev/da0p1
> ** /dev/da0p1
> ** Last Mounted on /exports
> ** Phase 1 - Check Blocks and Sizes
> ** Phase 2 - Check Pathnames
> ** Phase 3 - Check Connectivity
> ** Phase 4 - Check Reference Counts
> ** Phase 5 - Check Cyl groups
> 20964037 files, 443148893 used, 502743010 free (396842 frags,
> 62793271
> blocks, 0.0% fragmentation)
> 
> These are my filesystem params.
> 
> # tunefs -p /exports
> tunefs: POSIX.1e ACLs: (-a)                                disabled
> tunefs: NFSv4 ACLs: (-N)                                   disabled
> tunefs: MAC multilabel: (-l)                               disabled
> tunefs: soft updates: (-n)                                 enabled
> tunefs: soft update journaling: (-j)                       disabled
> tunefs: gjournal: (-J)                                     disabled
> tunefs: trim: (-t)                                         disabled
> tunefs: maximum blocks per file in a cylinder group: (-e)  4096
> tunefs: average file size: (-f)                            16384
> tunefs: average number of files in a directory: (-s)       64
> tunefs: minimum percentage of free space: (-m)             8%
> tunefs: space to hold for metadata blocks: (-k)            6408
> tunefs: optimization preference: (-o)                      time
> tunefs: volume label: (-L)
> 
> My mouting flags are:
> /dev/da0p1 on /exports (ufs, NFS exported, local, noatime, nosuid,
> with quotas, soft-updates)
> 
> I found file releated to sample inode, it's users's backup.
> 
>   File: "/exports/(...)/xxx.tar.gz"
>   Size: 53706752     FileType: Regular File
>   Mode: (0640/-rw-r-----)         Uid: ( 6225/  (6225))  Gid: ( 1676/
>    (1676))
> Device: 0,78   Inode: 185391682    Links: 1
> Access: Sat May 17 05:00:57 2014
> Modify: Sat May 17 05:01:02 2014
> Change: Sat May 17 05:01:48 2014
> 
> I am unable to decompress it, it's actually damaged:
> # cp xxx.tar.gz /otherfilesystem/
> # cd /otherfilesystem && gzip -d xxx.tar.gz
> gzip: xxx.tar.gz: unexpected end of file
> gzip: xxx.tar.gz: uncompress failed
> 
> I've tried to rerun this user's backup and I am able to reproduce
> problem just now. The same backup file of the same user now has
> different inode but problem is the same:
> pid 4173 (nfsd), uid 0 inumber 185391652 on /exports: filesystem full
> 
> # stat -x xxx.tar.gz
>   File: "xxx.tar.gz"
>   Size: 55902208     FileType: Regular File
>   Mode: (0640/-rw-r-----)         Uid: ( 6225/  (6225))  Gid: ( 1676/
>    (1676))
> Device: 0,78   Inode: 185391652    Links: 1
> Access: Sat May 17 12:51:47 2014
> Modify: Sat May 17 12:51:51 2014
> Change: Sat May 17 12:52:52 2014
> 
> Of course I am able to create any new big file from /dev/random,
> there
> is free space. I've easily created single 30GB file via nfs.
> I've also tried to count md5 sum of newly generated 1GB file many
> times via nfs and it's still the same. I checked this to be sure nfs
> transmission is valid and da0 device is working properly.
> 
> I've added some primitive debug to kernel sources but this is
> production env so I will be able to reboot server only at night. Just
> now it's impossible.
> 
> /usr/src/sys/ufs # grep -R 'filesystem full' *
> ffs/ffs_balloc.c:                               ffs_fserr(fs,
> ip->i_number, "filesystem full line 320");
> ffs/ffs_balloc.c:                               ffs_fserr(fs,
> ip->i_number, "filesystem full line 397");
> ffs/ffs_balloc.c:                               ffs_fserr(fs,
> ip->i_number, "filesystem full line 882");
> ffs/ffs_balloc.c:                               ffs_fserr(fs,
> ip->i_number, "filesystem full line 960");
> ffs/ffs_alloc.c:                ffs_fserr(fs, ip->i_number,
> "filesystem full line 227");
> ffs/ffs_alloc.c:                ffs_fserr(fs, ip->i_number,
> "filesystem full line 438");
> 
> Any help will be appreciated! Should be corrected before 9.3-RELEASE.
> 
> DD
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
> "freebsd-stable-unsubscribe at freebsd.org"
> 