ZFS vnode lock deadlock in zfs_fhtovp was: High Kernel Load with nfsv4
Loïc Blot
loic.blot at unix-experience.fr
Mon Dec 22 08:57:51 UTC 2014
Hi,
To clarify because of our exchanges. Here are the current sysctl options for server:
vfs.nfsd.enable_nobodycheck=0
vfs.nfsd.enable_nogroupcheck=0
vfs.nfsd.maxthreads=200
vfs.nfsd.tcphighwater=10000
vfs.nfsd.tcpcachetimeo=300
vfs.nfsd.server_min_nfsvers=4
kern.maxvnodes=10000000
kern.ipc.maxsockbuf=4194304
net.inet.tcp.sendbuf_max=4194304
net.inet.tcp.recvbuf_max=4194304
vfs.lookup_shared=0
Regards,
Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr
22 décembre 2014 09:42 "Loïc Blot" <loic.blot at unix-experience.fr> a écrit:
Hi Rick,
my 5 jails runs this weekend and now i have some stats on this monday.
Hopefully deadlock was fixed, yeah, but everything isn't good :(
On NFSv4 server (FreeBSD 10.1) system uses 35% CPU
As i can see this is because of nfsd:
918 root 96 20 0 12352K 3372K rpcsvc 6 51.4H 273.68% nfsd: server (nfsd)
If i look at dmesg i see:
nfsd server cache flooded, try increasing vfs.nfsd.tcphighwater
vfs.nfsd.tcphighwater was set to 10000, i increase it to 15000
Here is 'nfsstat -s' output:
Server Info:
Getattr Setattr Lookup Readlink Read Write Create Remove
12600652 1812 2501097 156 1386423 1983729 123 162067
Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
36762 9 0 0 0 3147 0 623524
Mknod Fsstat Fsinfo PathConf Commit
0 0 0 0 328117
Server Ret-Failed
0
Server Faults
0
Server Cache Stats:
Inprog Idem Non-idem Misses
0 0 0 12635512
Server Write Gathering:
WriteOps WriteRPC Opsaved
1983729 1983729 0
And here is 'procstat -kk' for nfsd (server)
918 100528 nfsd nfsd: master mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_timedwait_sig+0x10 _cv_timedwait_sig_sbt+0x18b svc_run_internal+0x4a1 svc_run+0x1de nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c amd64_syscall+0x351 Xfast_syscall+0xfb
918 100568 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100569 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100570 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100571 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100572 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100573 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100574 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100575 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100576 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100577 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100578 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100579 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100580 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100581 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100582 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100583 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100584 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100585 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100586 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100587 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100588 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100589 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100590 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100591 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100592 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100593 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100594 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100595 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100596 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100597 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100598 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100599 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100600 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100601 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100602 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100603 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100604 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100605 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100606 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100607 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100608 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100609 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100610 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100611 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100612 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100613 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100614 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100615 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100616 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100617 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100618 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100619 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100620 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100621 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100622 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100623 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100624 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100625 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100626 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100627 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100628 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100629 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100630 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100631 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100632 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100633 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100634 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100635 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100636 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100637 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100638 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100639 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100640 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100641 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100642 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100643 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100644 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100645 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100646 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100647 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100648 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100649 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100650 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100651 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100652 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100653 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100654 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100655 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100656 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100657 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100658 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100659 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100660 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100661 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100662 nfsd nfsd: service mi_switch+0xe1 sleepq_catch_signals+0xab sleepq_wait_sig+0xf _cv_wait_sig+0x16a svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
---
Now if we look at client (FreeBSD 9.3)
We see system was very busy and do many and many interrupts
CPU: 0.0% user, 0.0% nice, 37.8% system, 51.2% interrupt, 11.0% idle
A look at process list shows that there are many sendmail process in state nfstry
nfstry 18 32:27 0.88% sendmail: Queue runner at 00:30:00 for /var/spool/clientm
Here is 'nfsstat -c' output:
Client Info:
Rpc Counts:
Getattr Setattr Lookup Readlink Read Write Create Remove
1051347 1724 2494481 118 903902 1901285 162676 161899
Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
36744 2 0 114 40 3131 0 544136
Mknod Fsstat Fsinfo PathConf Commit
9 0 0 0 245821
Rpc Info:
TimedOut Invalid X Replies Retries Requests
0 0 0 0 8356557
Cache Info:
Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses
108754455 491475 54229224 2437229 46814561 821723 5132123 1871871
BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses
144035 118 53736 2753 27813 1 57238839 544205
If you need more things, tell me, i let the PoC in this state.
Thanks
Regards,
Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr
21 décembre 2014 01:33 "Rick Macklem" <rmacklem at uoguelph.ca> a écrit:
Loic Blot wrote:
> Hi Rick,
> ok, i don't need locallocks, i haven't understand option was for that
> usage, i removed it.
> I do more tests on monday.
> Thanks for the deadlock fix, for other people :)
Good. Please let us know if running with vfs.nfsd.enable_locallocks=0
gets rid of the deadlocks? (I think it fixes the one you saw.)
On the performance side, you might also want to try different values of
readahead, if the Linux client has such a mount option. (With the
NFSv4-ZFS sequential vs random I/O heuristic, I have no idea what the
optimal readahead value would be.)
Good luck with it and please let us know how it goes, rick
ps: I now have a patch to fix the deadlock when vfs.nfsd.enable_locallocks=1
is set. I'll post it for anyone who is interested after I put it
through some testing.
--
Best regards,
Loïc BLOT,
UNIX systems, security and network engineer
http://www.unix-experience.fr
Le jeudi 18 décembre 2014 à 19:46 -0500, Rick Macklem a écrit :
Loic Blot wrote:
> Hi rick,
> i tried to start a LXC container on Debian Squeeze from my
> freebsd
> ZFS+NFSv4 server and i also have a deadlock on nfsd
> (vfs.lookup_shared=0). Deadlock procs each time i launch a
> squeeze
> container, it seems (3 tries, 3 fails).
Well, I`ll take a look at this `procstat -kk`, but the only thing
I`ve seen posted w.r.t. avoiding deadlocks in ZFS is to not use
nullfs. (I have no idea if you are using any nullfs mounts, but
if so, try getting rid of them.)
Here`s a high level post about the ZFS and vnode locking problem,
but there is no patch available, as far as I know.
http://docs.FreeBSD.org/cgi/mid.cgi?54739F41.8030407
rick
921 - D 0:00.02 nfsd: server (nfsd)
Here is the procstat -kk
PID TID COMM TDNAME KSTACK
921 100538 nfsd nfsd: master mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
svc_run_internal+0xc77 svc_run+0x1de nfsrvd_nfsd+0x1ca
nfssvc_nfsd+0x107 sys_nfssvc+0x9c
921 100572 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100573 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100574 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100575 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100576 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100577 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100578 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100579 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100580 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100581 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100582 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100583 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100584 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100585 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100586 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100587 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100588 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100589 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100590 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100591 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100592 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100593 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100594 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100595 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100596 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100597 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100598 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100599 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100600 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100601 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100602 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100603 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100604 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100605 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100606 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100607 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100608 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100609 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100610 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100611 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100612 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100613 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100614 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100615 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100616 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
921 100617 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100618 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
921 100619 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100620 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100621 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100622 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100623 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100624 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100625 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100626 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100627 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100628 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100629 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100630 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100631 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100632 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100633 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100634 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100635 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100636 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100637 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100638 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100639 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100640 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100641 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100642 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100643 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100644 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100645 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100646 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100647 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100648 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100649 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100650 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100651 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100652 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100653 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100654 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100655 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100656 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100657 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100658 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100659 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100660 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100661 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100662 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100663 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100664 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100665 nfsd nfsd: service mi_switch+0xe1
sleepq_catch_signals+0xab sleepq_wait_sig+0xf
_cv_wait_sig+0x16a
svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
921 100666 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
nfsrvd_dorpc+0xc76
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
Regards,
Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr
15 décembre 2014 15:18 "Rick Macklem" <rmacklem at uoguelph.ca> a
écrit:
Loic Blot wrote:
> For more informations, here is procstat -kk on nfsd, if you
> need
> more
> hot datas, tell me.
>
> Regards, PID TID COMM TDNAME KSTACK
> 918 100529 nfsd nfsd: master mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77 svc_run+0x1de
> nfsrvd_nfsd+0x1ca nfssvc_nfsd+0x107 sys_nfssvc+0x9c
> amd64_syscall+0x351
Well, most of the threads are stuck like this one, waiting for
a
vnode
lock in ZFS. All of them appear to be in zfs_fhtovp().
I`m not a ZFS guy, so I can`t help much. I`ll try changing the
subject line
to include ZFS vnode lock, so maybe the ZFS guys will take a
look.
The only thing I`ve seen suggested is trying:
sysctl vfs.lookup_shared=0
to disable shared vop_lookup()s. Apparently zfs_lookup()
doesn`t
obey the vnode locking rules for lookup and rename, according
to
the posting I saw.
I`ve added a couple of comments about the other threads below,
but
they are all either waiting for an RPC request or waiting for
the
threads stuck on the ZFS vnode lock to complete.
rick
> 918 100564 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
Fyi, this thread is just waiting for an RPC to arrive. (Normal)
> 918 100565 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100566 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100567 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100568 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100569 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100570 nfsd nfsd: service mi_switch+0xe1
> sleepq_catch_signals+0xab sleepq_wait_sig+0xf
> _cv_wait_sig+0x16a
> svc_run_internal+0x87e svc_thread_start+0xb fork_exit+0x9a
> fork_trampoline+0xe
> 918 100571 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
> 918 100572 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrv_setclient+0xbd nfsrvd_setclientid+0x3c8
> nfsrvd_dorpc+0xc76
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
This one (and a few others) are waiting for the nfsv4_lock.
This
happens
because other threads are stuck with RPCs in progress. (ie. The
ones
waiting on the vnode lock in zfs_fhtovp().)
For these, the RPC needs to lock out other threads to do the
operation,
so it waits for the nfsv4_lock() which can exclusively lock the
NFSv4
data structures once all other nfsd threads complete their RPCs
in
progress.
> 918 100573 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
> nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
Same as above.
> 918 100574 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100575 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100576 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100577 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100578 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100579 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100580 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100581 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100582 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100583 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100584 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100585 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100586 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100587 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100588 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100589 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100590 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100591 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100592 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100593 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100594 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100595 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100596 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100597 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100598 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100599 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100600 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100601 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100602 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100603 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100604 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100605 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100606 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
> 918 100607 nfsd nfsd: service mi_switch+0xe1
> sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
> vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
> zfs_fhtovp+0x38d
> nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
> nfssvc_program+0x554 svc_run_internal+0xc77
> svc_thread_start+0xb
> fork_exit+0x9a fork_trampoline+0xe
Lots more waiting for the ZFS vnode lock in zfs_fhtovp().
918 100608 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrv_getlockfile+0x179 nfsrv_lockctrl+0x21f nfsrvd_lock+0x5b1
nfsrvd_dorpc+0xec6 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100609 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100610 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0xc9e
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
nfsvno_advlock+0x119 nfsrv_dolocal+0x84 nfsrv_lockctrl+0x14ad
nfsrvd_locku+0x283 nfsrvd_dorpc+0xec6 nfssvc_program+0x554
svc_run_internal+0xc77 svc_thread_start+0xb fork_exit+0x9a
fork_trampoline+0xe
918 100611 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100612 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100613 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100614 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100615 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100616 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100617 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100618 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100619 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100620 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100621 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100622 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100623 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a _sleep+0x287 nfsmsleep+0x66 nfsv4_lock+0x9b
nfsrvd_dorpc+0x316 nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb fork_exit+0x9a fork_trampoline+0xe
918 100624 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100625 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100626 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100627 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100628 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100629 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100630 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100631 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100632 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100633 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100634 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100635 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100636 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100637 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100638 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100639 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100640 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100641 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100642 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100643 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100644 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100645 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100646 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100647 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100648 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100649 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100650 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100651 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100652 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100653 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100654 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100655 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100656 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100657 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
918 100658 nfsd nfsd: service mi_switch+0xe1
sleepq_wait+0x3a sleeplk+0x15d __lockmgr_args+0x902
vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43
zfs_fhtovp+0x38d
nfsvno_fhtovp+0x7c nfsd_fhtovp+0xc8 nfsrvd_dorpc+0x917
nfssvc_program+0x554 svc_run_internal+0xc77
svc_thread_start+0xb
fork_exit+0x9a fork_trampoline+0xe
Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr
15 décembre 2014 13:29 "Loïc Blot"
<loic.blot at unix-experience.fr>
a
écrit:
Hmmm...
now i'm experiencing a deadlock.
0 918 915 0 21 0 12352 3372 zfs D - 1:48.64 nfsd: server
(nfsd)
the only issue was to reboot the server, but after rebooting
deadlock arrives a second time when i
start my jails over NFS.
Regards,
Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr
15 décembre 2014 10:07 "Loïc Blot"
<loic.blot at unix-experience.fr>
a
écrit:
Hi Rick,
after talking with my N+1, NFSv4 is required on our
infrastructure.
I tried to upgrade NFSv4+ZFS
server from 9.3 to 10.1, i hope this will resolve some
issues...
Regards,
Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr
10 décembre 2014 15:36 "Loïc Blot"
<loic.blot at unix-experience.fr>
a
écrit:
Hi Rick,
thanks for your suggestion.
For my locking bug, rpc.lockd is stucked in rpcrecv state on
the
server. kill -9 doesn't affect the
process, it's blocked.... (State: Ds)
for the performances
NFSv3: 60Mbps
NFSv4: 45Mbps
Regards,
Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr
10 décembre 2014 13:56 "Rick Macklem" <rmacklem at uoguelph.ca>
a
écrit:
Loic Blot wrote:
> Hi Rick,
> I'm trying NFSv3.
> Some jails are starting very well but now i have an issue
> with
> lockd
> after some minutes:
>
> nfs server 10.10.X.8:/jails: lockd not responding
> nfs server 10.10.X.8:/jails lockd is alive again
>
> I look at mbuf, but i seems there is no problem.
Well, if you need locks to be visible across multiple
clients,
then
I'm afraid you are stuck with using NFSv4 and the
performance
you
get
from it. (There is no way to do file handle affinity for
NFSv4
because
the read and write ops are buried in the compound RPC and
not
easily
recognized.)
If the locks don't need to be visible across multiple
clients,
I'd
suggest trying the "nolockd" option with nfsv3.
> Here is my rc.conf on server:
>
> nfs_server_enable="YES"
> nfsv4_server_enable="YES"
> nfsuserd_enable="YES"
> nfsd_server_flags="-u -t -n 256"
> mountd_enable="YES"
> mountd_flags="-r"
> nfsuserd_flags="-usertimeout 0 -force 20"
> rpcbind_enable="YES"
> rpc_lockd_enable="YES"
> rpc_statd_enable="YES"
>
> Here is the client:
>
> nfsuserd_enable="YES"
> nfsuserd_flags="-usertimeout 0 -force 20"
> nfscbd_enable="YES"
> rpc_lockd_enable="YES"
> rpc_statd_enable="YES"
>
> Have you got an idea ?
>
> Regards,
>
> Loïc Blot,
> UNIX Systems, Network and Security Engineer
> http://www.unix-experience.fr
>
> 9 décembre 2014 04:31 "Rick Macklem" <rmacklem at uoguelph.ca>
> a
> écrit:
>> Loic Blot wrote:
>>
>>> Hi rick,
>>>
>>> I waited 3 hours (no lag at jail launch) and now I do:
>>> sysrc
>>> memcached_flags="-v -m 512"
>>> Command was very very slow...
>>>
>>> Here is a dd over NFS:
>>>
>>> 601062912 bytes transferred in 21.060679 secs (28539579
>>> bytes/sec)
>>
>> Can you try the same read using an NFSv3 mount?
>> (If it runs much faster, you have probably been bitten by
>> the
>> ZFS
>> "sequential vs random" read heuristic which I've been told
>> things
>> NFS is doing "random" reads without file handle affinity.
>> File
>> handle affinity is very hard to do for NFSv4, so it isn't
>> done.)
I was actually suggesting that you try the "dd" over nfsv3
to
see
how
the performance compared with nfsv4. If you do that, please
post
the
comparable results.
Someday I would like to try and get ZFS's sequential vs
random
read
heuristic modified and any info on what difference in
performance
that
might make for NFS would be useful.
rick
rick
This is quite slow...
You can found some nfsstat below (command isn't finished
yet)
nfsstat -c -w 1
GtAttr Lookup Rdlink Read Write Rename Access Rddir
0 0 0 0 0 0 0 0
4 0 0 0 0 0 16 0
2 0 0 0 0 0 17 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 4 0 0 0 0 4 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
4 0 0 0 0 0 3 0
0 0 0 0 0 0 3 0
37 10 0 8 0 0 14 1
18 16 0 4 1 2 4 0
78 91 0 82 6 12 30 0
19 18 0 2 2 4 2 0
0 0 0 0 2 0 0 0
0 0 0 0 0 0 0 0
GtAttr Lookup Rdlink Read Write Rename Access Rddir
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 1 0 0 0 0 1 0
4 6 0 0 6 0 3 0
2 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 1 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
6 108 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
GtAttr Lookup Rdlink Read Write Rename Access Rddir
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
98 54 0 86 11 0 25 0
36 24 0 39 25 0 10 1
67 8 0 63 63 0 41 0
34 0 0 35 34 0 0 0
75 0 0 75 77 0 0 0
34 0 0 35 35 0 0 0
75 0 0 74 76 0 0 0
33 0 0 34 33 0 0 0
0 0 0 0 5 0 0 0
0 0 0 0 0 0 6 0
11 0 0 0 0 0 11 0
0 0 0 0 0 0 0 0
0 17 0 0 0 0 1 0
GtAttr Lookup Rdlink Read Write Rename Access Rddir
4 5 0 0 0 0 12 0
2 0 0 0 0 0 26 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 4 0 0 0 0 4 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
4 0 0 0 0 0 2 0
2 0 0 0 0 0 24 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
GtAttr Lookup Rdlink Read Write Rename Access Rddir
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
4 0 0 0 0 0 7 0
2 1 0 0 0 0 1 0
0 0 0 0 2 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 6 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
4 6 0 0 0 0 3 0
0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
GtAttr Lookup Rdlink Read Write Rename Access Rddir
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
4 71 0 0 0 0 0 0
0 1 0 0 0 0 0 0
2 36 0 0 0 0 1 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
1 0 0 0 0 0 1 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
79 6 0 79 79 0 2 0
25 0 0 25 26 0 6 0
43 18 0 39 46 0 23 0
36 0 0 36 36 0 31 0
68 1 0 66 68 0 0 0
GtAttr Lookup Rdlink Read Write Rename Access Rddir
36 0 0 36 36 0 0 0
48 0 0 48 49 0 0 0
20 0 0 20 20 0 0 0
0 0 0 0 0 0 0 0
3 14 0 1 0 0 11 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 4 0 0 0 0 4 0
0 0 0 0 0 0 0 0
4 22 0 0 0 0 16 0
2 0 0 0 0 0 23 0
Regards,
Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr
8 décembre 2014 09:36 "Loïc Blot"
<loic.blot at unix-experience.fr> a
écrit:
> Hi Rick,
> I stopped the jails this week-end and started it this
> morning,
> i'll
> give you some stats this week.
>
> Here is my nfsstat -m output (with your rsize/wsize
> tweaks)
nfsv4,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negna
etimeo=60,rsize=32768,wsize=32768,readdirsize=32768,readahead=1,wcommitsize=773136,timeout=120,retra
s=2147483647
On server side my disks are on a raid controller which show a
512b
volume and write performances
are very honest (dd if=/dev/zero of=/jails/test.dd bs=4096
count=100000000 => 450MBps)
Regards,
Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr
5 décembre 2014 15:14 "Rick Macklem" <rmacklem at uoguelph.ca> a
écrit:
Loic Blot wrote:
Hi,
i'm trying to create a virtualisation environment based on
jails.
Those jails are stored under a big ZFS pool on a FreeBSD
9.3
which
export a NFSv4 volume. This NFSv4 volume was mounted on a
big
hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but
only 1
was
used at this time).
The problem is simple, my hypervisors runs 6 jails (used 1%
cpu
and
10GB RAM approximatively and less than 1MB bandwidth) and
works
fine at start but the system slows down and after 2-3 days
become
unusable. When i look at top command i see 80-100% on
system
and
commands are very very slow. Many process are tagged with
nfs_cl*.
To be honest, I would expect the slowness to be because of
slow
response
from the NFSv4 server, but if you do:
# ps axHl
on a client when it is slow and post that, it would give us
some
more
information on where the client side processes are sitting.
If you also do something like:
# nfsstat -c -w 1
and let it run for a while, that should show you how many
RPCs
are
being done and which ones.
# nfsstat -m
will show you what your mount is actually using.
The only mount option I can suggest trying is
"rsize=32768,wsize=32768",
since some network environments have difficulties with 64K.
There are a few things you can try on the NFSv4 server side,
if
it
appears
that the clients are generating a large RPC load.
- disabling the DRC cache for TCP by setting
vfs.nfsd.cachetcp=0
- If the server is seeing a large write RPC load, then
"sync=disabled"
might help, although it does run a risk of data loss when
the
server
crashes.
Then there are a couple of other ZFS related things (I'm not
a
ZFS
guy,
but these have shown up on the mailing lists).
- make sure your volumes are 4K aligned and ashift=12 (in
case a
drive
that uses 4K sectors is pretending to be 512byte sectored)
- never run over 70-80% full if write performance is an
issue
- use a zil on an SSD with good write performance
The only NFSv4 thing I can tell you is that it is known that
ZFS's
algorithm for determining sequential vs random I/O fails for
NFSv4
during writing and this can be a performance hit. The only
workaround
is to use NFSv3 mounts, since file handle affinity
apparently
fixes
the problem and this is only done for NFSv3.
rick
I saw that there are TSO issues with igb then i'm trying to
disable
it with sysctl but the situation wasn't solved.
Someone has got ideas ? I can give you more informations if
you
need.
Thanks in advance.
Regards,
Loïc Blot,
UNIX Systems, Network and Security Engineer
http://www.unix-experience.fr
_______________________________________________
freebsd-fs at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to
"freebsd-fs-unsubscribe at freebsd.org"
_______________________________________________
freebsd-fs at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to
"freebsd-fs-unsubscribe at freebsd.org"
_______________________________________________
freebsd-fs at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to
"freebsd-fs-unsubscribe at freebsd.org"
_______________________________________________
freebsd-fs at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to
"freebsd-fs-unsubscribe at freebsd.org"
_______________________________________________
freebsd-fs at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to
"freebsd-fs-unsubscribe at freebsd.org"
_______________________________________________
freebsd-fs at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
More information about the freebsd-fs
mailing list