Improving ZFS performance for large directories

[Ronald Klop]

On Wed, 30 Jan 2013 00:20:15 +0100, Kevin Day <toasty at dragondata.com>  
wrote:

>
> I'm trying to improve performance when using ZFS in large (>60000 files)  
> directories. A common activity is to use "getdirentries" to enumerate  
> all the files in the directory, then "lstat" on each one to get  
> information about it. Doing an "ls -l" in a large directory like this  
> can take 10-30 seconds to complete. Trying to figure out why, I did:
>
> ktrace ls -l /path/to/large/directory
> kdump -R |sort -rn |more

Does ls -lf /pat/to/large/directory make a difference. It makes ls not to  
sort the directory so it can use a more efficient way of traversing the  
directory.

Ronald.


>
> to see what sys calls were taking the most time, I ended up with:
>
>  69247 ls       0.190729 STRU  struct stat {dev=846475008, ino=46220085,  
> mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295,  
> atime=1333196714, stime=1201004393, ctime=1333196714.547566024,  
> birthtime=1333196714.547566024, size=30784, blksize=31232, blocks=62,  
> flags=0x0 }
>  69247 ls       0.180121 STRU  struct stat {dev=846475008, ino=46233417,  
> mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295,  
> atime=1333197088, stime=1209814737, ctime=1333197088.913571042,  
> birthtime=1333197088.913571042, size=3162220, blksize=131072,  
> blocks=6409, flags=0x0 }
>  69247 ls       0.152370 RET   getdirentries 4088/0xff8
>  69247 ls       0.139939 CALL  stat(0x800d8f598,0x7fffffffcca0)
>  69247 ls       0.130411 RET   __acl_get_link 0
>  69247 ls       0.121602 RET   __acl_get_link 0
>  69247 ls       0.105799 RET   getdirentries 4064/0xfe0
>  69247 ls       0.105069 RET   getdirentries 4068/0xfe4
>  69247 ls       0.096862 RET   getdirentries 4028/0xfbc
>  69247 ls       0.085012 RET   getdirentries 4088/0xff8
>  69247 ls       0.082722 STRU  struct stat {dev=846475008, ino=72941319,  
> mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295,  
> atime=1348686155, stime=1348347621, ctime=1348686155.768875422,  
> birthtime=1348686155.768875422, size=6686225, blksize=131072,  
> blocks=13325, flags=0x0 }
>  69247 ls       0.070318 STRU  struct stat {dev=846475008, ino=46211679,  
> mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295,  
> atime=1333196475, stime=1240230314, ctime=1333196475.038567672,  
> birthtime=1333196475.038567672, size=829895, blksize=131072,  
> blocks=1797, flags=0x0 }
>  69247 ls       0.068060 RET   getdirentries 4048/0xfd0
>  69247 ls       0.065118 RET   getdirentries 4088/0xff8
>  69247 ls       0.062536 RET   getdirentries 4096/0x1000
>  69247 ls       0.061118 RET   getdirentries 4020/0xfb4
>  69247 ls       0.055038 STRU  struct stat {dev=846475008, ino=46220358,  
> mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295,  
> atime=1333196720, stime=1274282669, ctime=1333196720.972567345,  
> birthtime=1333196720.972567345, size=382344, blksize=131072, blocks=773,  
> flags=0x0 }
>  69247 ls       0.054948 STRU  struct stat {dev=846475008, ino=75025952,  
> mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295,  
> atime=1351071350, stime=1349726805, ctime=1351071350.800873870,  
> birthtime=1351071350.800873870, size=2575559, blksize=131072,  
> blocks=5127, flags=0x0 }
>  69247 ls       0.054828 STRU  struct stat {dev=846475008, ino=65021883,  
> mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295,  
> atime=1335730367, stime=1332843230, ctime=1335730367.541567371,  
> birthtime=1335730367.541567371, size=226347, blksize=131072, blocks=517,  
> flags=0x0 }
>  69247 ls       0.053743 STRU  struct stat {dev=846475008, ino=46222016,  
> mode=-rw-r--r-- , nlink=1, uid=0, gid=0, rdev=4294967295,  
> atime=1333196765, stime=1257110706, ctime=1333196765.206574132,  
> birthtime=1333196765.206574132, size=62112, blksize=62464, blocks=123,  
> flags=0x0 }
>  69247 ls       0.052015 RET   getdirentries 4060/0xfdc
>  69247 ls       0.051388 RET   getdirentries 4068/0xfe4
>  69247 ls       0.049875 RET   getdirentries 4088/0xff8
>  69247 ls       0.049156 RET   getdirentries 4032/0xfc0
>  69247 ls       0.048609 RET   getdirentries 4040/0xfc8
>  69247 ls       0.048279 RET   getdirentries 4032/0xfc0
>  69247 ls       0.048062 RET   getdirentries 4064/0xfe0
>  69247 ls       0.047577 RET   getdirentries 4076/0xfec
> (snip)
>
> the STRU are returns from calling lstat().
>
> It looks like both getdirentries and lstat are taking quite a while to  
> return. The shortest return for any lstat() call is 0.000004 seconds,  
> the maximum is 0.190729 and the average is around 0.0004. Just from  
> lstat() alone, that makes "ls" take over 20 seconds.
>
> I'm prepared to try an L2arc cache device (with  
> secondarycache=metadata), but I'm having trouble determining how big of  
> a device I'd need. We've got >30M inodes now on this filesystem,  
> including some files with extremely long names. Is there some way to  
> determine the amount of metadata on a ZFS filesystem?
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"