deadlock with zfs?
Thomas Vogt
freebsdlists at bsdunix.ch
Fri Nov 21 00:56:08 PST 2008
Hello
Am 21.11.2008 um 01:01 schrieb Thomas:
> Hello
>
> I encounter a deadlock while running a few rsync processes mirroring
> remote data to my local zfs pool. After a few hours my system is
> starting more and more vsftpd sessions without closing any inactive
> ftp sessions. I can't kill any rsync and vsftpd processes with "kill
> -9". Even shutdown -r now does not work.
>
> I got a few hunderts vsftpd processes like this
>
> 61346 root 1 57 0 7880K 1692K zfs 1 0:00 0.00%
> vsftpd
> 61481 root 1 68 0 7880K 1696K zfs 1 0:00 0.00%
> vsftpd
> 61354 root 1 65 0 7880K 1692K zfs 1 0:00 0.00%
> vsftpd
> 61480 root 1 68 0 7880K 1696K zfs 0 0:00 0.00%
> vsftpd
> 61600 root 1 69 0 7880K 1704K zfs 1 0:00 0.00%
> vsftpd
> 61599 root 1 68 0 7880K 1704K zfs 1 0:00 0.00%
> vsftpd
>
> Right now i'm building a debug kernel. Whats the best way to get
> usefull information from this deadlock? The system itself is not
> crashing and response well to ssh and i also have a serial console.
>
> The system is running 8.0-CURRENT Thu Nov 20 00:15:46 UTC 2008
> (64bit) with zpool version 13. I use zfs only as data pool. The base
> system is running on ufs2:
>
> Filesystem Size Used Avail Capacity Mounted on
> /dev/da0s1a 496M 220M 236M 48% /
> devfs 1.0K 1.0K 0B 100% /dev
> /dev/da0s1g 169G 15G 141G 9% /disk1
> /dev/da0s1f 3.9G 17M 3.5G 0% /tmp
> /dev/da0s1e 29G 3.7G 23G 14% /usr
> /dev/da0s1d 19G 7.1G 11G 40% /var
> pool 853G 0B 853G 0% /usr/local/data
> pool/cvsup 858G 5.7G 853G 1% /usr/local/data/cvsup
> pool/ftp 3.3T 2.5T 853G 75% /usr/local/data/ftp
> pool/portsnap 853G 633M 853G 0% /usr/local/data/
> portsnap
> pool/www 853G 90M 853G 0% /usr/local/data/www
>
> loader.conf:
> vm.kmem_size="1G"
> kern.maxfiles="65536"
> kern.maxproc="20480"
> net.inet.tcp.tcbhashsize="4096"
> net.inet.tcp.hostcache.hashsize="1024"
> vfs.zfs.arc_min="64M"
> vfs.zfs.arc_max="768M"
> vfs.zfs.prefetch_disable="1"
>
> I also tried to set vfs.zfs.zil_disable=1 but the problem still exist.
>
> I know there are few deadlock reports listed in the freebsd wiki.
> Maybe we can trigger the root cause of the problem and fix it :)
>
> Regards
> Thomas
I recompiled my kernel with debug option. Maybe i have some useful
data. I did most commands listed at http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-deadlocks.html
. alltrace created a ridiculous long list. I added all output to a
text file. http://www.bsdunix.ch/deadlock_2008-11-21.txt
Just the ps output and a trace output as example:
db> ps
pid ppid pgrp uid state wmesg wchan cmd
21909 2551 21909 0 R+ CPU 0 sysctl
21865 21848 21831 1001 S select 0xffffff002a8164c0 rsync
21863 21844 21827 1001 S zfs 0xffffff00038a7cc8 rsync
21861 21847 21838 1001 S zfs 0xffffff00038a7cc8 rsync
21858 21845 21835 1001 S zfs 0xffffff00038a7cc8 rsync
21852 21841 21841 1001 S zfs 0xffffff00038a7cc8 perl
21850 21837 21837 1001 S zfs 0xffffff00038a7cc8 perl5.8.8
21848 21831 21831 1001 S wait 0xffffff0106a82430 sh
21847 21838 21838 1001 S wait 0xffffff001ec22860 sh
21846 21839 21839 0 S db->db_c 0xffffff0119b826b0 newsyslog
21845 21835 21835 1001 S wait 0xffffff003094f860 sh
21844 21827 21827 1001 S wait 0xffffff0138091000 sh
21841 21830 21841 1001 Ss wait 0xffffff0030936430 sh
21839 21832 21839 0 Ss wait 0xffffff01428a8860 sh
21838 21825 21838 1001 Ss wait 0xffffff0030935000 sh
21837 21829 21837 1001 Ss wait 0xffffff003094b430 sh
21835 21828 21835 1001 Ss wait 0xffffff01428a8430 sh
21832 1016 1016 0 S piperd 0xffffff003097fb20 cron
21831 21826 21831 1001 Ss wait 0xffffff01428a3430 sh
21830 1016 1016 0 S piperd 0xffffff01383f4858 cron
21829 1016 1016 0 S piperd 0xffffff0023dd42c8 cron
21828 1016 1016 0 S piperd 0xffffff003098b590 cron
21827 21824 21827 1001 Ss wait 0xffffff0030936000 sh
21826 1016 1016 0 S piperd 0xffffff01383f4000 cron
21825 1016 1016 0 S piperd 0xffffff000383e590 cron
21824 1016 1016 0 S piperd 0xffffff0023dd5858 cron
19454 19448 19448 0 S db->db_c 0xffffff0119b826b0 newsyslog
19448 19445 19448 0 Ss wait 0xffffff0106a80000 sh
19445 1016 1016 0 S piperd 0xffffff000383e858 cron
14894 14893 14894 1007 Ss db->db_c 0xffffff0119b826b0 cat
14893 14891 14891 1007 S select 0xffffff0018caa1c0 sshd
14891 997 14891 0 Ss sbwait 0xffffff0106a048b4 sshd
13493 13479 13479 1001 S zfs 0xffffff00038a7cc8 perl5.8.8
13490 13483 13483 0 S db->db_c 0xffffff0119b826b0 newsyslog
13483 13477 13483 0 Ss wait 0xffffff0138090860 sh
13479 13476 13479 1001 Ss wait 0xffffff0138296860 sh
13477 1016 1016 0 S piperd 0xffffff01383f62c8 cron
13476 1016 1016 0 S piperd 0xffffff0105ed82c8 cron
9316 9311 9311 0 S db->db_c 0xffffff0119b826b0 newsyslog
9311 9308 9311 0 Ss wait 0xffffff0138294430 sh
9308 1016 1016 0 S piperd 0xffffff000383cb20 cron
7843 943 943 0 S zfs 0xffffff00038a7cc8 rsync
7767 7764 7764 1001 S zfs 0xffffff00038a7cc8 perl
7764 7761 7764 1001 Ss wait 0xffffff001ec21860 sh
7761 1016 1016 0 S piperd 0xffffff0105ed8590 cron
7693 943 943 0 S zfs 0xffffff00038a7cc8 rsync
7490 943 943 0 S zfs 0xffffff00038a7cc8 rsync
7263 943 943 0 S zfs 0xffffff00038a7cc8 rsync
6758 943 943 0 S zfs 0xffffff00038a7cc8 rsync
6630 943 943 0 S zfs 0xffffff00038a7cc8 rsync
6582 943 943 0 S zfs 0xffffff00038a7cc8 rsync
6529 6524 6524 0 S db->db_c 0xffffff0119b826b0 newsyslog
6527 6520 6520 1001 S zfs 0xffffff00038a7cc8 perl5.8.8
6524 6518 6524 0 Ss wait 0xffffff001ec23860 sh
6520 6517 6520 1001 Ss wait 0xffffff001ec05430 sh
6518 1016 1016 0 S piperd 0xffffff0023dd5000 cron
6517 1016 1016 0 S piperd 0xffffff000383c590 cron
6478 943 943 0 S zfs 0xffffff00038a7cc8 rsync
6053 943 943 0 S zfs 0xffffff00038a7cc8 rsync
5788 5786 5786 0 S zfs 0xffffff00038a7cc8 vsftpd
5786 823 5786 0 Ss sbwait 0xffffff0138272b34 vsftpd
5785 5783 5783 0 S zfs 0xffffff00038a7cc8 vsftpd
5783 823 5783 0 Ss sbwait 0xffffff01382f1634 vsftpd
5766 5764 5764 0 S zfs 0xffffff00038a7cc8 vsftpd
5764 823 5764 0 Ss sbwait 0xffffff01381b63b4 vsftpd
5753 5751 5751 0 S zfs 0xffffff00038a7cc8 vsftpd
5751 823 5751 0 Ss sbwait 0xffffff01382703b4 vsftpd
5744 5742 5742 0 S zfs 0xffffff00038a7cc8 vsftpd
5742 823 5742 0 Ss sbwait 0xffffff01381b6134 vsftpd
5733 5731 5731 0 S zfs 0xffffff00038a7cc8 vsftpd
5731 823 5731 0 Ss sbwait 0xffffff0138142db4 vsftpd
5730 5728 5728 0 S zfs 0xffffff00038a7cc8 vsftpd
5728 823 5728 0 Ss sbwait 0xffffff01381b5634 vsftpd
5727 5724 5720 1001 S zfs 0xffffff00038a7cc8 rsync
5724 5720 5720 1001 S wait 0xffffff013808d000 sh
5720 5717 5720 1001 Ss wait 0xffffff0037e6b000 sh
5717 1016 1016 0 S piperd 0xffffff00038d2000 cron
5714 943 943 0 S zfs 0xffffff00038a7cc8 rsync
5713 5711 5711 0 S zfs 0xffffff00038a7cc8 vsftpd
5711 823 5711 0 Ss sbwait 0xffffff01381b7634 vsftpd
5709 5707 5707 0 S zfs 0xffffff00038a7cc8 vsftpd
5707 823 5707 0 Ss sbwait 0xffffff01380d2b34 vsftpd
5706 5704 5704 0 S zfs 0xffffff00038a7cc8 vsftpd
5704 823 5704 0 Ss sbwait 0xffffff01381413b4 vsftpd
5696 5694 5694 0 S zfs 0xffffff00038a7cc8 vsftpd
5694 823 5694 0 Ss sbwait 0xffffff0138142634 vsftpd
5669 943 943 0 S zfs 0xffffff00038a7cc8 rsync
5627 5625 5598 1007 S zio->io_ 0xffffff00372c3df8 sh
5626 5600 5598 1007 S piperd 0xffffff00038d2858 grep
5625 5600 5598 1007 S wait 0xffffff013808e430 sh
5610 5597 5610 1007 Ss piperd 0xffffff000383e2c8 sendmail
5600 5599 5598 1007 S wait 0xffffff0037fec860 sh
5599 5598 5598 1007 S wait 0xffffff0037ca2430 lockf
5598 5597 5598 1007 Ss wait 0xffffff0037e67000 sh
5597 1016 1016 0 S piperd 0xffffff0105ed7b20 cron
5596 5594 5594 0 S zfs 0xffffff00038a7cc8 vsftpd
5594 823 5594 0 Ss sbwait 0xffffff00376a93b4 vsftpd
5584 5582 5582 0 S zfs 0xffffff00038a7cc8 vsftpd
5582 823 5582 0 Ss sbwait 0xffffff0037f66634 vsftpd
5570 5568 5568 0 S zfs 0xffffff00038a7cc8 vsftpd
5568 823 5568 0 Ss sbwait 0xffffff0037f6eb34 vsftpd
5472 5470 5470 0 S zfs 0xffffff00038a7cc8 vsftpd
5470 823 5470 0 Ss sbwait 0xffffff0037ff6634 vsftpd
5459 5457 5457 0 S zfs 0xffffff00038a7cc8 vsftpd
5457 823 5457 0 Ss sbwait 0xffffff0037ff9db4 vsftpd
5442 5440 5440 0 S zfs 0xffffff00038a7cc8 vsftpd
5440 823 5440 0 Ss sbwait 0xffffff0037ec3b34 vsftpd
5399 5397 5397 0 S zfs 0xffffff00038a7cc8 vsftpd
5397 823 5397 0 Ss sbwait 0xffffff0037ec48b4 vsftpd
5375 5358 5358 0 S zfs 0xffffff00038a7cc8 vsftpd
5374 5372 5372 0 S zfs 0xffffff00038a7cc8 vsftpd
5372 823 5372 0 Ss sbwait 0xffffff0037eb73b4 vsftpd
5367 5362 5362 0 S zfs 0xffffff00038a7cc8 vsftpd
5366 5364 5364 1004 S zio->io_ 0xffffff0003993b28 vsftpd
5364 823 5364 0 Ss sbwait 0xffffff0037eb6634 vsftpd
5362 823 5362 0 Ss sbwait 0xffffff0037eb6db4 vsftpd
5358 823 5358 0 Ss sbwait 0xffffff0037eb8134 vsftpd
5356 5353 5353 1004 S zio->io_ 0xffffff00372c82b8 vsftpd
5353 823 5353 0 Ss sbwait 0xffffff0037e0e634 vsftpd
5349 5347 5347 1004 S db->db_c 0xffffff0121effb10 vsftpd
5347 823 5347 0 Ss sbwait 0xffffff0037dc8db4 vsftpd
5344 5342 5342 1004 S db->db_c 0xffffff0121effb10 vsftpd
5342 823 5342 0 Ss sbwait 0xffffff0037e0edb4 vsftpd
5328 5326 5326 1004 S zfs 0xffffff0007de0308 vsftpd
5326 823 5326 0 Ss sbwait 0xffffff0037e10134 vsftpd
5323 5321 5321 1004 S db->db_c 0xffffff0121effb10 vsftpd
5321 823 5321 0 Ss sbwait 0xffffff0037d2db34 vsftpd
5318 5301 5301 1004 S db->db_c 0xffffff0121effb10 vsftpd
5301 823 5301 0 Ss sbwait 0xffffff0037dc7db4 vsftpd
5297 5295 5295 1004 S db->db_c 0xffffff0121effb10 vsftpd
5295 823 5295 0 Ss sbwait 0xffffff0037cb68b4 vsftpd
5288 5286 5286 1004 S db->db_c 0xffffff0121effb10 vsftpd
5286 823 5286 0 Ss sbwait 0xffffff0037d2cb34 vsftpd
5285 5283 5283 1004 S db->db_c 0xffffff0121effb10 vsftpd
5283 823 5283 0 Ss sbwait 0xffffff0037d2c8b4 vsftpd
5258 5256 5256 1004 S db->db_c 0xffffff0121effb10 vsftpd
5256 823 5256 0 Ss sbwait 0xffffff0037d2c3b4 vsftpd
5251 5249 5249 1004 S zio->io_ 0xffffff0037368b28 vsftpd
5249 823 5249 0 Ss sbwait 0xffffff0037ce78b4 vsftpd
5246 5244 5244 1004 S db->db_c 0xffffff0121effb10 vsftpd
5244 823 5244 0 Ss sbwait 0xffffff0037cb43b4 vsftpd
5226 5224 5224 1004 S zfs 0xffffff0007de0308 vsftpd
5224 823 5224 0 Ss sbwait 0xffffff0037ce7634 vsftpd
5223 5221 5221 1004 S db->db_c 0xffffff0121effb10 vsftpd
5221 823 5221 0 Ss sbwait 0xffffff0037afadb4 vsftpd
5189 5187 5187 1004 S db->db_c 0xffffff0121effb10 vsftpd
5187 823 5187 0 Ss sbwait 0xffffff0037c98db4 vsftpd
5186 5184 5184 1004 S zfs 0xffffff0007de0308 vsftpd
5184 823 5184 0 Ss sbwait 0xffffff0037cd8134 vsftpd
5134 5132 5132 1004 S db->db_c 0xffffff0121effb10 vsftpd
5132 823 5132 0 Ss sbwait 0xffffff0037bc0b34 vsftpd
5131 5129 5129 1004 S zfs 0xffffff0007de0308 vsftpd
5129 823 5129 0 Ss sbwait 0xffffff0037bc73b4 vsftpd
5128 5126 5126 1004 S zio->io_ 0xffffff01103f7858 vsftpd
5126 823 5126 0 Ss sbwait 0xffffff0037bc0634 vsftpd
5092 5090 5090 1004 S db->db_c 0xffffff0121effb10 vsftpd
5090 823 5090 0 Ss sbwait 0xffffff0037bd4634 vsftpd
5073 5071 5071 1004 S zio->io_ 0xffffff0003a7a2b8 vsftpd
5071 823 5071 0 Ss sbwait 0xffffff0037af8b34 vsftpd
5066 5064 5064 1004 S db->db_c 0xffffff0121effb10 vsftpd
5064 823 5064 0 Ss sbwait 0xffffff0037aa2db4 vsftpd
5044 5042 5042 1004 S db->db_c 0xffffff0121effb10 vsftpd
5042 823 5042 0 Ss sbwait 0xffffff0037aa2134 vsftpd
5041 943 943 65534 S db->db_c 0xffffff012a1b1e90 rsync
5040 5038 5038 1004 S db->db_c 0xffffff0121effb10 vsftpd
5038 823 5038 0 Ss sbwait 0xffffff0037aa2634 vsftpd
5007 5005 5005 1004 S db->db_c 0xffffff0121effb10 vsftpd
5005 823 5005 0 Ss sbwait 0xffffff00379c28b4 vsftpd
5003 5001 5001 1004 S db->db_c 0xffffff0121effb10 vsftpd
5001 823 5001 0 Ss sbwait 0xffffff0037a20634 vsftpd
5000 4998 4998 1004 S db->db_c 0xffffff0121effb10 vsftpd
4998 823 4998 0 Ss sbwait 0xffffff0037a20db4 vsftpd
4983 4981 4981 1004 S db->db_c 0xffffff0121effb10 vsftpd
4981 823 4981 0 Ss sbwait 0xffffff003787d8b4 vsftpd
4979 4977 4977 1004 S db->db_c 0xffffff0121effb10 vsftpd
4977 823 4977 0 Ss sbwait 0xffffff00379f0db4 vsftpd
4973 4970 4973 1003 S zio->io_ 0xffffff0133e8d858 cvsup
4970 4967 4958 0 S wait 0xffffff00379eb860 su
4967 4966 4958 0 S wait 0xffffff0007c11430 sh
4966 4960 4958 0 S wait 0xffffff0018402000 lockf
4960 4958 4958 0 S wait 0xffffff003786a000 sh
4958 4956 4958 0 Ss wait 0xffffff003787b000 sh
4956 1016 1016 0 S piperd 0xffffff000383eb20 cron
4952 4950 4950 1004 S db->db_c 0xffffff0130177e90 vsftpd
4950 823 4950 0 Ss sbwait 0xffffff003787c134 vsftpd
4933 943 943 65534 S db->db_c 0xffffff012a1b1e90 rsync
4894 4892 4892 1004 S db->db_c 0xffffff0121effb10 vsftpd
4892 823 4892 0 Ss sbwait 0xffffff0037957db4 vsftpd
4882 4880 4880 1004 S zfsvfs-> 0xffffff00079b1800 vsftpd
4880 823 4880 0 Ss sbwait 0xffffff0123edb3b4 vsftpd
4861 4860 4799 0 S zio->io_ 0xffffff0133bb72b8 df
4860 4816 4799 0 S wait 0xffffff002393a000 sh
4818 4817 4799 0 S piperd 0xffffff0105ed62c8 mail
4817 4802 4799 0 S wait 0xffffff0037868430 sh
4816 4802 4799 0 S wait 0xffffff0037868000 sh
4802 4799 4799 0 S wait 0xffffff00631cb860 sh
4799 4796 4799 0 Ss wait 0xffffff003787a000 sh
4796 1016 1016 0 S piperd 0xffffff0105ed6858 cron
4770 4765 4765 0 S zio->io_ 0xffffff0003ae7858 newsyslog
4765 4762 4765 0 Ss wait 0xffffff00631b4860 sh
4762 1016 1016 0 S piperd 0xffffff000383c2c8 cron
4756 970 970 1002 S db->db_c 0xffffff0103cef250 cvsupd
4755 970 970 1002 S zio->io_ 0xffffff0133bad588 cvsupd
4754 970 970 1002 S db->db_c 0xffffff0103cef250 cvsupd
4753 970 970 1002 S db->db_c 0xffffff0103cef250 cvsupd
4740 970 970 1002 S zio->io_ 0xffffff0133dcfb28 cvsupd
4739 970 970 1002 S db->db_c 0xffffff0103cef250 cvsupd
4710 4708 4708 1004 S db->db_c 0xffffff0130177e90 vsftpd
4708 823 4708 0 Ss sbwait 0xffffff00376f8634 vsftpd
4676 4674 4674 1004 S db->db_c 0xffffff0121effb10 vsftpd
4674 823 4674 0 Ss sbwait 0xffffff00376f93b4 vsftpd
4673 4671 4671 1004 S db->db_c 0xffffff0121effb10 vsftpd
4671 823 4671 0 Ss sbwait 0xffffff0037653db4 vsftpd
4670 4668 4668 1004 S db->db_c 0xffffff0121effb10 vsftpd
4668 823 4668 0 Ss sbwait 0xffffff00376a7b34 vsftpd
4667 4665 4665 1004 S db->db_c 0xffffff0121effb10 vsftpd
4665 823 4665 0 Ss sbwait 0xffffff00376a9134 vsftpd
4664 4662 4662 1004 S db->db_c 0xffffff0121effb10 vsftpd
4662 823 4662 0 Ss sbwait 0xffffff0037651b34 vsftpd
4661 4659 4659 1004 S db->db_c 0xffffff0121effb10 vsftpd
4659 823 4659 0 Ss sbwait 0xffffff00376523b4 vsftpd
4647 4645 4645 1004 S db->db_c 0xffffff0121effb10 vsftpd
4645 823 4645 0 Ss sbwait 0xffffff00375e3634 vsftpd
4644 4642 4642 1004 S db->db_c 0xffffff0121effb10 vsftpd
4642 823 4642 0 Ss sbwait 0xffffff00375f4134 vsftpd
4640 4638 4638 1004 S db->db_c 0xffffff0121effb10 vsftpd
4638 823 4638 0 Ss sbwait 0xffffff00375f4db4 vsftpd
4637 4635 4635 1004 S zio->io_ 0xffffff0003a8ddf8 vsftpd
4635 823 4635 0 Ss sbwait 0xffffff0088393634 vsftpd
4588 4586 4586 1004 S db->db_c 0xffffff0121effb10 vsftpd
4586 823 4586 0 Ss sbwait 0xffffff0037560634 vsftpd
4567 4565 4565 1004 S db->db_c 0xffffff0121effb10 vsftpd
4565 823 4565 0 Ss sbwait 0xffffff003753cb34 vsftpd
4553 943 943 65534 S zio->io_ 0xffffff0133bf8b28 rsync
4528 4526 4526 1004 S db->db_c 0xffffff0121effb10 vsftpd
4526 823 4526 0 Ss sbwait 0xffffff0007c318b4 vsftpd
4514 4512 4512 1004 S db->db_c 0xffffff0121effb10 vsftpd
4512 823 4512 0 Ss sbwait 0xffffff00883958b4 vsftpd
4509 4505 4505 1004 S db->db_c 0xffffff0121effb10 vsftpd
4505 823 4505 0 Ss sbwait 0xffffff00883938b4 vsftpd
4499 4497 4497 1004 S db->db_c 0xffffff0121effb10 vsftpd
4497 823 4497 0 Ss sbwait 0xffffff00181f7134 vsftpd
4486 4484 4484 1004 S db->db_c 0xffffff0121effb10 vsftpd
4484 823 4484 0 Ss sbwait 0xffffff0007e78134 vsftpd
4472 970 970 1002 S zio->io_ 0xffffff0133c12b28 cvsupd
4421 4419 4419 1004 S db->db_c 0xffffff0121effb10 vsftpd
4419 823 4419 0 Ss sbwait 0xffffff0123edd634 vsftpd
4412 4410 4410 1004 S db->db_c 0xffffff0121effb10 vsftpd
4410 823 4410 0 Ss sbwait 0xffffff0123739134 vsftpd
4395 4393 4393 1004 S db->db_c 0xffffff0121effb10 vsftpd
4393 823 4393 0 Ss sbwait 0xffffff0018c49b34 vsftpd
4363 4358 4358 1004 S db->db_c 0xffffff0121effb10 vsftpd
4362 4360 4360 1004 S zio->io_ 0xffffff0133b282b8 vsftpd
4360 823 4360 0 Ss sbwait 0xffffff0123739634 vsftpd
4358 823 4358 0 Ss sbwait 0xffffff014e6b3634 vsftpd
4354 4341 4341 1004 S zio->io_ 0xffffff01104612b8 vsftpd
4341 823 4341 0 Ss sbwait 0xffffff0123738134 vsftpd
4337 4335 4335 1004 S zio->io_ 0xffffff0133ae0858 vsftpd
4335 823 4335 0 Ss sbwait 0xffffff00181f63b4 vsftpd
4322 943 943 65534 S zio->io_ 0xffffff003700b588 rsync
4304 970 970 1002 SL zio->io_ 0xffffff011040d2b8 cvsupd
4298 4296 4296 1004 S zio->io_ 0xffffff01103b6df8 vsftpd
4296 823 4296 0 Ss sbwait 0xffffff00181f6134 vsftpd
2551 2549 2551 0 Ss+ pause 0xffffff0007c134d0 csh
2549 997 2549 0 Ss select 0xffffff00079a0dc0 sshd
2470 2469 2466 1001 S+ zfs 0xffffff0139a87a58 rsync
2469 2467 2466 1001 S+ zio->io_ 0xffffff00372c7df8 rsync
2467 2466 2466 1001 S+ wait 0xffffff0018403430 sh
2466 2465 2466 1001 S+ wait 0xffffff0018403000 sh
2465 1206 2465 0 S+ wait 0xffffff00038b2430 su
1206 1190 1206 0 Ss+ pause 0xffffff0018402900 csh
1190 997 1190 0 Ss select 0xffffff0023914240 sshd
1089 1 1089 0 Ss+ tty inpu 0xffffff00036310b8 getty
1088 1 1088 0 Ss+ tty inpu 0xffffff000372d8b8 getty
1087 1 1087 0 Ss+ tty inpu 0xffffff00037318b8 getty
1086 1 1086 0 Ss+ tty inpu 0xffffff000373bcb8 getty
1085 1 1085 0 Ss+ tty inpu 0xffffff000373b8b8 getty
1084 1 1084 0 Ss+ tty inpu 0xffffff000373c8b8 getty
1083 1 1083 0 Ss+ tty inpu 0xffffff000373c4b8 getty
1082 1 1082 0 Ss+ tty inpu 0xffffff000373a8b8 getty
1081 1 1081 0 Ss+ tty inpu 0xffffff000373a4b8 getty
1047 1 1047 0 Ss select 0xffffff0007ac0e40 inetd
1016 1 1016 0 Ss nanslp 0xffffffff808423e8 cron
1010 1 1010 25 Ss pause 0xffffff000383d4d0 sendmail
1006 1 1006 0 Ss select 0xffffff000741d240 sendmail
997 1 997 0 Ss select 0xffffff0007ac0540 sshd
970 1 970 1002 Ss select 0xffffff00038abbc0 cvsupd
949 1 948 80 S zio->io_ 0xffffff0133e0fb28 lighttpd
943 1 943 0 Ss select 0xffffff0003951640 rsync
899 1 899 0 Ss select 0xffffff0007a3d4c0 ntpd
841 1 840 0 S select 0xffffff000741c340 snmpd
823 1 823 0 Ss accept 0xffffff0007c2aa5e vsftpd
703 1 703 0 Ss select 0xffffff0007ac1140 syslogd
534 1 534 0 Ss select 0xffffff00079890c0 devd
171 0 0 0 SL tq->tq_d 0xffffff0007be5600
[zil_clean]
170 0 0 0 SL tq->tq_d 0xffffff00039232a0
[zil_clean]
169 0 0 0 SL tq->tq_d 0xffffff0003923180
[zil_clean]
168 0 0 0 SL tq->tq_d 0xffffff0003923060
[zil_clean]
167 0 0 0 SL tq->tq_d 0xffffff000390e4e0
[zil_clean]
165 0 0 0 SL tx->tx_q 0xffffff0003b6e668
[txg_thread_enter]
164 0 0 0 SL tx->tx_c 0xffffff00039124b0
[txg_thread_enter]
163 0 0 0 SL vgeom:io 0xffffff0003bc5110
[vdev:worker da7]
162 0 0 0 SL vgeom:io 0xffffff000394f490
[vdev:worker da6]
161 0 0 0 SL vgeom:io 0xffffff000394f510
[vdev:worker da5]
160 0 0 0 SL vgeom:io 0xffffff000394f690
[vdev:worker da4]
159 0 0 0 SL vgeom:io 0xffffff000394f710
[vdev:worker da3]
158 0 0 0 SL vgeom:io 0xffffff000394f810
[vdev:worker da2]
157 0 0 0 SL vgeom:io 0xffffff000394f910
[vdev:worker da1]
156 0 0 0 SL tq->tq_d 0xffffff000390e600 [spa_zio]
155 0 0 0 SL tq->tq_d 0xffffff000390e720 [spa_zio]
154 0 0 0 SL tq->tq_d 0xffffff000390e840 [spa_zio]
153 0 0 0 SL tq->tq_d 0xffffff000390e960 [spa_zio]
152 0 0 0 SL tq->tq_d 0xffffff000390f4e0 [spa_zio]
151 0 0 0 SL tq->tq_d 0xffffff000390f180 [spa_zio]
150 0 0 0 SL zfsvfs-> 0xffffff00079b11e0 [spa_zio]
149 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio]
148 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio]
147 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio]
146 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio]
145 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio]
144 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio]
143 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio]
142 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio]
141 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio]
140 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio]
139 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio]
138 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio]
137 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio]
136 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio]
135 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio]
134 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio]
133 0 0 0 SL tq->tq_d 0xffffff000390ede0 [spa_zio]
132 0 0 0 SL tq->tq_d 0xffffff000390f2a0 [spa_zio]
131 0 0 0 SL tq->tq_d 0xffffff000390f3c0 [spa_zio]
97 0 0 0 SL l2arc_fe 0xffffffff80d134e0
[l2arc_feed_thread]
96 0 0 0 SL arc_recl 0xffffffff80d0af00
[arc_reclaim_thread]
93 0 0 0 SL tq->tq_d 0xffffff000390f060
[system_taskq]
92 0 0 0 SL tq->tq_d 0xffffff000390f060
[system_taskq]
24 0 0 0 SL sdflush 0xffffffff80a130b8
[softdepflush]
23 0 0 0 SL zio->io_ 0xffffff0133c0e588 [syncer]
22 0 0 0 SL vlruwt 0xffffff000365d000 [vnlru]
21 0 0 0 SL psleep 0xffffffff80a02fdc
[bufdaemon]
9 0 0 0 SL pgzero 0xffffffff80a14a2c [pagezero]
8 0 0 0 SL psleep 0xffffffff80a13dc8 [vmdaemon]
7 0 0 0 SL psleep 0xffffffff80a13d8c
[pagedaemon]
6 0 0 0 SL waiting_ 0xffffffff80a062a8
[sctp_iterator]
20 0 0 0 SL usbevt 0xffffff00015a9420 [usb4]
19 0 0 0 SL usbevt 0xfffffffe4046c420 [usb3]
18 0 0 0 SL usbevt 0xfffffffe4046a420 [usb2]
17 0 0 0 SL usbevt 0xfffffffe40468420 [usb1]
16 0 0 0 SL usbtsk 0xffffffff8083e0e8 [usbtask-
dr]
15 0 0 0 SL usbtsk 0xffffffff8083e0c0 [usbtask-
hc]
14 0 0 0 SL usbevt 0xfffffffe40466420 [usb0]
5 0 0 0 SL ccb_scan 0xffffffff80822e60 [xpt_thrd]
13 0 0 0 SL - 0xffffffff808420c4 [yarrow]
4 0 0 0 SL - 0xffffffff8083e848 [g_down]
3 0 0 0 SL - 0xffffffff8083e840 [g_up]
2 0 0 0 SL - 0xffffffff8083e830 [g_event]
12 0 0 0 RL (threaded) intr
100040 I [swi0:
uart uart]
100039 I [irq1:
atkbd0]
100038 I [irq20:
atapci1]
100037 I [irq14:
ata0]
100032 I [irq22:
uhci1 uhci3]
100028 I [irq23:
uhci0 uhci+]
100025 I [irq18:
arcmsr0]
100024 I [irq9:
acpi0]
100022 I [swi5: +]
100021 I [swi2:
cambio]
100015 I [swi6:
task queue]
100014 I [swi6:
Giant taskq]
100008 I [swi1: net]
100007 I [swi4:
clock]
100006 Run CPU 1 [swi4:
clock]
100005 I [swi3: vm]
11 0 0 0 RL (threaded) idle
100004 CanRun [idle:
cpu0]
100003 CanRun [idle:
cpu1]
1 0 1 0 SLs wait 0xffffff0001442860 [init]
10 0 0 0 SL audit_wo 0xffffffff80a124f0 [audit]
0 0 0 0 SLs (threaded) kernel
100027 D - 0xffffff000159f280 [em1 taskq]
100026 D - 0xffffff00015a0000 [em0 taskq]
100023 D - 0xffffff000152b180 [thread
taskq]
100019 D - 0xffffff000152b900
[acpi_task_2]
100018 D - 0xffffff000152b900
[acpi_task_1]
100017 D - 0xffffff000152b900
[acpi_task_0]
100016 D - 0xffffff000152b980 [kqueue
taskq]
100012 D - 0xffffff0001455080 [firmware
taskq]
100000 D sched 0xffffffff8083e940 [swapper]
db> trace 4509
Tracing pid 4509 tid 100186 td 0xffffff0063ba4720
sched_switch() at sched_switch+0x180
mi_switch() at mi_switch+0x21d
sleepq_switch() at sleepq_switch+0x123
sleepq_wait() at sleepq_wait+0x4d
_cv_wait() at _cv_wait+0x17a
dbuf_read() at dbuf_read+0x33b
dmu_buf_hold() at dmu_buf_hold+0xcc
zap_lockdir() at zap_lockdir+0x53
zap_lookup_norm() at zap_lookup_norm+0x45
zap_lookup() at zap_lookup+0x2e
zfs_dirent_lock() at zfs_dirent_lock+0x501
zfs_dirlook() at zfs_dirlook+0x69
zfs_lookup() at zfs_lookup+0x1f4
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x81
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf
vfs_cache_lookup() at vfs_cache_lookup+0xf0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
lookup() at lookup+0x516
namei() at namei+0x53d
vn_open_cred() at vn_open_cred+0x1a7
kern_openat() at kern_openat+0x169
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (5, FreeBSD ELF64, open), rip = 0x800a4dd0c, rsp =
0x7fffffffe988, rbp = 0xfa0 ---
I'm not a kernel debug expert. Kust let me know if you need additional
information.
Regards
Thomas
More information about the freebsd-current
mailing list