[9.3 PRE] filesystem full even if there are free descriptors and disk space

[Damian Danielecki]

I am receiving errors like this on heavily loaded FreeBSD
9.3-PRERELEASE nfs server:
May 17 05:01:02 nfsd kernel: pid 4173 (nfsd), uid 0 inumber 185391682
on /exports: filesystem full

Filesystem is newly created and server newly installed with custom
minimalist kernel&world. I am experienced user.
There are many free inodes and much disk space:

# df -i
Filesystem          1K-blocks       Used      Avail Capacity  iused
 ifree %iused  Mounted on
/dev/mirror/gm0s1a  473048844    3449532  431755408     1%   306673
60848397    1%   /
devfs                       1          1          0   100%        0
     0  100%   /dev
/dev/da0p1         3783567612 1772595572 1708286632    51% 20964037
468035769    4%   /exports

# df -h
Filesystem            Size    Used   Avail Capacity  Mounted on
/dev/mirror/gm0s1a    451G    3.3G    411G     1%    /
devfs                 1.0k    1.0k      0B   100%    /dev
/dev/da0p1            3.5T    1.7T    1.6T    51%    /exports

# uname -a
FreeBSD nfsd.xxx.pl 9.3-PRERELEASE FreeBSD 9.3-PRERELEASE #5: Fri May
16 15:41:36 CEST 2014
root at nfsd.xxx.pl:/usr/obj/usr/src/sys/FREEBSD9  amd64

Filesystem is clean and not fragmented.
For sure fsck has been done on unmounted filesystem.
I see these errors also after fsck.

# fsck -t ufs -y /dev/da0p1
** /dev/da0p1
** Last Mounted on /exports
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
20964037 files, 443148893 used, 502743010 free (396842 frags, 62793271
blocks, 0.0% fragmentation)

These are my filesystem params.

# tunefs -p /exports
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: soft update journaling: (-j)                       disabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  4096
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: space to hold for metadata blocks: (-k)            6408
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)

My mouting flags are:
/dev/da0p1 on /exports (ufs, NFS exported, local, noatime, nosuid,
with quotas, soft-updates)

I found file releated to sample inode, it's users's backup.

  File: "/exports/(...)/xxx.tar.gz"
  Size: 53706752     FileType: Regular File
  Mode: (0640/-rw-r-----)         Uid: ( 6225/  (6225))  Gid: ( 1676/  (1676))
Device: 0,78   Inode: 185391682    Links: 1
Access: Sat May 17 05:00:57 2014
Modify: Sat May 17 05:01:02 2014
Change: Sat May 17 05:01:48 2014

I am unable to decompress it, it's actually damaged:
# cp xxx.tar.gz /otherfilesystem/
# cd /otherfilesystem && gzip -d xxx.tar.gz
gzip: xxx.tar.gz: unexpected end of file
gzip: xxx.tar.gz: uncompress failed

I've tried to rerun this user's backup and I am able to reproduce
problem just now. The same backup file of the same user now has
different inode but problem is the same:
pid 4173 (nfsd), uid 0 inumber 185391652 on /exports: filesystem full

# stat -x xxx.tar.gz
  File: "xxx.tar.gz"
  Size: 55902208     FileType: Regular File
  Mode: (0640/-rw-r-----)         Uid: ( 6225/  (6225))  Gid: ( 1676/  (1676))
Device: 0,78   Inode: 185391652    Links: 1
Access: Sat May 17 12:51:47 2014
Modify: Sat May 17 12:51:51 2014
Change: Sat May 17 12:52:52 2014

Of course I am able to create any new big file from /dev/random, there
is free space. I've easily created single 30GB file via nfs.
I've also tried to count md5 sum of newly generated 1GB file many
times via nfs and it's still the same. I checked this to be sure nfs
transmission is valid and da0 device is working properly.

I've added some primitive debug to kernel sources but this is
production env so I will be able to reboot server only at night. Just
now it's impossible.

/usr/src/sys/ufs # grep -R 'filesystem full' *
ffs/ffs_balloc.c:                               ffs_fserr(fs,
ip->i_number, "filesystem full line 320");
ffs/ffs_balloc.c:                               ffs_fserr(fs,
ip->i_number, "filesystem full line 397");
ffs/ffs_balloc.c:                               ffs_fserr(fs,
ip->i_number, "filesystem full line 882");
ffs/ffs_balloc.c:                               ffs_fserr(fs,
ip->i_number, "filesystem full line 960");
ffs/ffs_alloc.c:                ffs_fserr(fs, ip->i_number,
"filesystem full line 227");
ffs/ffs_alloc.c:                ffs_fserr(fs, ip->i_number,
"filesystem full line 438");

Any help will be appreciated! Should be corrected before 9.3-RELEASE.

DD