kern/179932: [ciss] ciss i/o stall problem with HP Bl Gen8 (and HP Bl Gen7 Storage Blade)
Philipp Maechler
philipp.maechler at hostpoint.ch
Wed Jun 26 11:20:01 UTC 2013
The following reply was made to PR kern/179932; it has been noted by GNATS.
From: Philipp Maechler <philipp.maechler at hostpoint.ch>
To: sbruno at freebsd.org
Cc: bug-followup at FreeBSD.org, Sean Bruno <sean_bruno at yahoo.com>
Subject: Re: kern/179932: [ciss] ciss i/o stall problem with HP Bl Gen8 (and
HP Bl Gen7 Storage Blade)
Date: Wed, 26 Jun 2013 13:12:40 +0200
This is a multi-part message in MIME format.
--------------070005030706090006050108
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
hi Sean
At the moment we are running the serial console at 9600 baud; we have to
tune this to 115200 baud and also then an alltrace will take "forever"
because of hundreds of apache threads; because this machines are
productive and non-redundant we have to figure out a procedure how we
can handle to do this long-running alltrace without interrupting the
services for such a time.
As an attachement, I can provide you an old debugger session with a
(part) of a ps and some backtraces. It is from one of our HP
Gen7-StorageBlade-Systems running on 9.1-RELEASE or higher, hanging
earlier this year with the same problem.
If you find there enough information to say "alltrace is not needed",
we'd be happy. But until then we will search for possibilities to do
such an alltrace...
The break possibility we already enabled by setting the sysctl
debug.kdb.break_to_debugger=1
As soon as you send us the mentioned patch we will begin to test it and
introduce it to the affected systems.
Philipp
p.s. please note; even there are a lot of processes in some zfs states,
we also tried to do dd to some raw partitions on disk and were not
successful, so we think it's not a zfs problem. In gstat we saw the wait
queue's stall at specific values and nothing going on anymore.
--------------070005030706090006050108
Content-Type: text/plain; charset=UTF-8;
name="20130626_olddebugsession.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="20130626_olddebugsession.txt"
Notes:
* the ps is not complete, we think there is a part missing; but it gives you an overview
* are you looking for specific processes or threads? please tell us!
</>hpiLO-> vsp
Virtual Serial Port Active: COM1
Starting virtual serial port.
Press 'ESC (' to return to the CLI Session.
Tracing pid 12 tid 100043 td 0xfffffe0019ce0000
kdb_enter() at kdb_enter+0x3b
kdb_break() at kdb_break+0x27
scgetc() at scgetc+0x361
sckbdevent() at sckbdevent+0xed
kbdmux_intr() at kbdmux_intr+0x43
kbdmux_kbd_intr() at kbdmux_kbd_intr+0x20
taskqueue_run_locked() at taskqueue_run_locked+0x85
taskqueue_run() at taskqueue_run+0x3a
intr_event_execute_handlers() at intr_event_execute_handlers+0x104
ithread_loop() at ithread_loop+0xa6
fork_exit() at fork_exit+0x11f
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80003ffcf0, rbp = 0 ---
db> ps
pid ppid pgrp uid state wmesg wchan cmd
87125 87123 87125 0 Ds+ zfs 0xfffffe05c0c80db8 bash
87123 1683 87123 0 Ss select 0xfffffe0a5d29a8c0 sshd
87122 87120 87122 0 Ds+ zfs 0xfffffe05c0c80db8 bash
87120 1683 87120 0 Ss select 0xfffffe0a5d4603c0 sshd
86929 86927 86929 0 Ds zio->io_ 0xfffffe02a13b4d70 atrun
86928 86926 86928 2 Ds zio->io_ 0xfffffe08a73d5a00 sh
86927 1687 1687 0 S piperd 0xfffffe02f036a5b0 cron
86926 1687 1687 0 S piperd 0xfffffe050c0cb5b0 cron
86738 86736 86738 0 Ds+ zio->io_ 0xfffffe03f1d81690 bash
86736 1683 86736 0 Ss select 0xfffffe0b7f10b7c0 sshd
86729 86728 86729 181 Ds zfs 0xfffffe026a16d458 python
86728 86726 86726 181 S select 0xfffffe02996c7a40 sshd
86726 1683 86726 0 Ss sbwait 0xfffffe03a1d6ee8c sshd
86725 86724 86725 181 Ds zfs 0xfffffe026a16d458 python
86724 86722 86722 181 S select 0xfffffe0aa6b7e840 sshd
86722 1683 86722 0 Ss sbwait 0xfffffe0c4f90ae8c sshd
86721 86719 86718 181 S piperd 0xfffffe064c9d8b60 wc
86720 86719 86718 181 D spa_name 0xffffffff817f7d00 zpool
86719 86718 86718 181 S wait 0xfffffe0e9fe53000 bash
86718 86717 86718 181 Ss piperd 0xfffffe0604ecd2d8 bash
86717 86715 86715 181 S select 0xfffffe03761605c0 sshd
86715 1683 86715 0 Ss sbwait 0xfffffe0d3d8a83ec sshd
86535 86534 86535 181 Ds zfs 0xfffffe026a16d458 python
86534 86528 86528 181 S select 0xfffffe081b6a1040 sshd
86533 86532 86533 181 Ds zfs 0xfffffe026a16d458 python
86532 86526 86526 181 S select 0xfffffe02815b85c0 sshd
86531 86530 86531 181 Ds zfs 0xfffffe026a16d458 python
86530 86524 86524 181 S select 0xfffffe048360e7c0 sshd
86528 1683 86528 0 Ss sbwait 0xfffffe035cbb0e8c sshd
86526 1683 86526 0 Ss sbwait 0xfffffe006acb03ec sshd
86524 1683 86524 0 Ss sbwait 0xfffffe0030653694 sshd
86328 86327 86328 181 Ds zfs 0xfffffe026a16d458 python
86327 86323 86323 181 S select 0xfffffe0a8855a4c0 sshd
86325 86324 86325 181 Ds zfs 0xfffffe026a16d458 python
86324 86315 86315 181 S select 0xfffffe09ed0b7cc0 sshd
86323 1683 86323 0 Ss sbwait 0xfffffe07afe3d144 sshd
86322 86321 86322 181 Ds zfs 0xfffffe026a16d458 python
86321 86311 86311 181 S select 0xfffffe0e3265b840 sshd
86320 86319 86320 181 Ds zfs 0xfffffe026a16d458 python
86319 86304 86304 181 S select 0xfffffe088678d2c0 sshd
86318 86317 86318 181 Ds zfs 0xfffffe026a16d458 python
86317 86303 86303 181 S select 0xfffffe02418f3640 sshd
86315 1683 86315 0 Ss sbwait 0xfffffe02a72e23ec sshd
86313 86312 86313 181 Ds zfs 0xfffffe026a16d458 python
86312 86297 86297 181 S select 0xfffffe009dcc10c0 sshd
86311 1683 86311 0 Ss sbwait 0xfffffe013d385144 sshd
86310 86309 86310 181 Ds zfs 0xfffffe026a16d458 python
86309 86301 86301 181 S select 0xfffffe01fd72d3c0 sshd
86308 86307 86308 181 Ds zfs 0xfffffe026a16d458 python
86307 86298 86298 181 S select 0xfffffe0483fa1a40 sshd
86304 1683 86304 0 Ss sbwait 0xfffffe02544f5144 sshd
86303 1683 86303 0 Ss sbwait 0xfffffe0ddf800694 sshd
86301 1683 86301 0 Ss sbwait 0xfffffe0b4ef6b144 sshd
86298 1683 86298 0 Ss sbwait 0xfffffe0beda78e8c sshd
86297 1683 86297 0 Ss sbwait 0xfffffe0a36823144 sshd
86292 86291 86292 181 Ds zfs 0xfffffe026a16d458 python
86291 86285 86285 181 S select 0xfffffe0d0524b640 sshd
86290 86289 86290 181 Ds zfs 0xfffffe026a16d458 python
86289 86283 86283 181 S select 0xfffffe0f480e1cc0 sshd
86288 86287 86288 181 Ds zio->io_ 0xfffffe039861f690 python
86287 86281 86281 181 S select 0xfffffe04cf9c80c0 sshd
86285 1683 86285 0 Ss sbwait 0xfffffe03fe528694 sshd
86283 1683 86283 0 Ss sbwait 0xfffffe0311e9593c sshd
86281 1683 86281 0 Ss sbwait 0xfffffe0d3d8a8be4 sshd
86279 1668 1668 0 D zio->io_ 0xfffffe028063ad70 exim-4.80.1-0
86091 86089 86088 181 S piperd 0xfffffe06da459b60 wc
86090 86089 86088 181 D spa_name 0xffffffff817f7d00 zpool
86089 86088 86088 181 S wait 0xfffffe0a88c16000 bash
86088 86087 86088 181 Ss piperd 0xfffffe050c0cb888 bash
86087 86085 86085 181 S select 0xfffffe0bb51f8cc0 sshd
86085 1683 86085 0 Ss sbwait 0xfffffe02ab86e3ec sshd
86080 86078 83614 0 S piperd 0xfffffe0e58ef85b0 grep
86079 86078 83614 0 D spa_name 0xffffffff817f7d00 zfs
86078 83628 83614 0 S wait 0xfffffe0a99074000 bash
85728 83566 49671 0 D tx->tx_s 0xfffffe00209eee20 zfs
85332 85321 85332 0 Ss+ ttyin 0xfffffe0524a144a8 bash
85321 1683 85321 0 Ss select 0xfffffe0b122beb40 sshd
83628 83621 83614 0 S piperd 0xfffffe013ff055b0 bash
83621 83614 83614 0 S wait 0xfffffe0ef81f74a0 bash
83614 83610 83614 0 Ss piperd 0xfffffe0ad77c7000 sh
83610 1687 1687 0 S piperd 0xfffffe03d14ffb60 cron
83566 49854 49671 0 S wait 0xfffffe0eae6ab000 bash
49854 49685 49671 0 S wait 0xfffffe0d80e8b4a0 bash
49685 49671 49671 0 S wait 0xfffffe0caf48d940 bash
49671 49665 49671 0 Ss piperd 0xfffffe0cd7726b60 sh
49665 1687 1687 0 S piperd 0xfffffe0b5d9482d8 cron
34311 34306 34311 0 Ss+ ttyin 0xfffffe002c9388a8 bash
34306 1683 34306 0 Ss select 0xfffffe079c3c0d40 sshd
24912 24564 24912 0 Ss+ ttyin 0xfffffe03e79d2ca8 bash
24564 1683 24564 0 Ss select 0xfffffe0f6a802540 sshd
64399 63540 34618 88 S (threaded) mysqld
105150 S uwait 0xfffffe022c939900 mysqld
105149 S uwait 0xfffffe0603a95280 mysqld
105148 S uwait 0xfffffe0e6d323300 mysqld
105147 S uwait 0xfffffe0f11b16680 mysqld
105146 S sigwait 0xfffffe086602d000 mysqld
105145 S uwait 0xfffffe03c8eff880 mysqld
105144 S uwait 0xfffffe07370e5100 mysqld
105143 S uwait 0xfffffe0ef7fa9b00 mysqld
105142 S uwait 0xfffffe02d415ba00 mysqld
105140 S uwait 0xfffffe0e58e9a100 mysqld
105139 S uwait 0xfffffe06c9cfd000 mysqld
105138 S uwait 0xfffffe01f5eb7e00 mysqld
105137 S uwait 0xfffffe0de3773100 mysqld
105136 S uwait 0xfffffe077c241900 mysqld
105135 S uwait 0xfffffe03dae38800 mysqld
105134 S uwait 0xfffffe09069d7400 mysqld
105133 S uwait 0xfffffe045cc16980 mysqld
105132 S uwait 0xfffffe083cacc000 mysqld
105131 S uwait 0xfffffe02ece6fa80 mysqld
102539 S select 0xfffffe099d9a6040 mysqld
63540 1 34618 0 S wait 0xfffffe004fa1d940 sh
47863 1 47863 0 Ss+ ttyin 0xfffffe001e47e8a8 getty
11143 1 11143 88 DsJ (threaded) mysqld
105622 S uwait 0xfffffe028377db00 mysqld
100617 S uwait 0xfffffe0d3a2d7080 mysqld
100557 S sbwait 0xfffffe00ae8d5144 mysqld
105432 S sbwait 0xfffffe0466736694 mysqld
105431 S sbwait 0xfffffe05950073ec mysqld
105401 S sbwait 0xfffffe069c2dbe8c mysqld
105365 D zio->io_ 0xfffffe0cbafe1a00 mysqld
104814 S uwait 0xfffffe08b0699180 mysqld
104813 S uwait 0xfffffe0b4ee56900 mysqld
104812 S sbwait 0xfffffe017e6fd144 mysqld
104811 S sbwait 0xfffffe094050e694 mysqld
104810 S sbwait 0xfffffe004e27abe4 mysqld
104773 S sbwait 0xfffffe06b343e144 mysqld
104772 S sbwait 0xfffffe059500be8c mysqld
104771 S sbwait 0xfffffe06a97ba93c mysqld
104766 S sbwait 0xfffffe059500ae8c mysqld
104765 S uwait 0xfffffe052bc88c00 mysqld
104764 S sbwait 0xfffffe02ad61a694 mysqld
104763 S sbwait 0xfffffe01fab54694 mysqld
104762 S sbwait 0xfffffe0912318694 mysqld
104759 S sbwait 0xfffffe06b343ebe4 mysqld
104756 S uwait 0xfffffe084b1e4400 mysqld
104738 S umtxn 0xfffffe07bbb1b700 mysqld
103653 S uwait 0xfffffe0393be2280 mysqld
103643 S uwait 0xfffffe0b4e03a600 mysqld
103633 S sbwait 0xfffffe04b04b693c mysqld
103534 D zio->io_ 0xfffffe0d5d939690 mysqld
103217 S sbwait 0xfffffe0e588afe8c mysqld
103216 S sbwait 0xfffffe077e407be4 mysqld
103128 S uwait 0xfffffe043b164c00 mysqld
103126 S uwait 0xfffffe0426a71b00 mysqld
103125 S sbwait 0xfffffe006a0f63ec mysqld
103105 S sbwait 0xfffffe02ad61a93c mysqld
102497 D zio->io_ 0xfffffe0b5de39320 mysqld
102496 S sbwait 0xfffffe006acb0e8c mysqld
102487 S sbwait 0xfffffe077e4063ec mysqld
102475 S sbwait 0xfffffe040f987e8c mysqld
102427 S sbwait 0xfffffe00a35643ec mysqld
102394 S uwait 0xfffffe09ed0c0e00 mysqld
102360 S sbwait 0xfffffe0beda7b694 mysqld
102357 S sbwait 0xfffffe0e6673ee8c mysqld
102356 S sigwait 0xfffffe0b7e213000 mysqld
102355 D zio->io_ 0xfffffe03c317cd70 mysqld
102354 S uwait 0xfffffe0bad460880 mysqld
102353 D zio->io_ 0xfffffe0d09658690 mysqld
102352 S uwait 0xfffffe0bad379b00 mysqld
102350 S uwait 0xfffffe079c3b5680 mysqld
102349 S uwait 0xfffffe09ed530400 mysqld
102348 S uwait 0xfffffe09edae6100 mysqld
102347 S uwait 0xfffffe0c4f119580 mysqld
102346 S uwait 0xfffffe0de1a35780 mysqld
102345 S uwait 0xfffffe0c4f11a900 mysqld
102344 S uwait 0xfffffe0c4f116c00 mysqld
102343 S uwait 0xfffffe079c2f1480 mysqld
102342 S uwait 0xfffffe09ed935d80 mysqld
102341 S uwait 0xfffffe079c1e6880 mysqld
102299 S select 0xfffffe0b1cec4140 mysqld
4970 4968 4970 0 Ss+ ttyin 0xfffffe02b83830a8 bash
4968 1683 4968 0 Ss select 0xfffffe0bad48b840 sshd
2777 2183 2777 0 S+ kqread 0xfffffe027cd92900 tail
2183 2181 2183 0 Ss+ wait 0xfffffe002c5a1940 bash
2181 1683 2181 0 Ss select 0xfffffe01e42f4140 sshd
2180 1 1 0 S ttydcd 0xfffffe001e4758e8 getty
2178 1 2178 0 Ss+ ttyin 0xfffffe001e47bca8 getty
2177 1 2177 0 Ss+ ttyin 0xfffffe001e47b4a8 getty
2176 1 2176 0 Ss+ ttyin 0xfffffe001e47b0a8 getty
2175 1 2175 0 Ss+ ttyin 0xfffffe001e47a8a8 getty
2174 1 2174 0 Ss+ ttyin 0xfffffe001e47a4a8 getty
2173 1 2173 0 Ss+ ttyin 0xfffffe001e4768a8 getty
2172 1 2172 0 Ss+ ttyin 0xfffffe001e476ca8 getty
2171 1 2171 0 DLs+ zio->io_ 0xfffffe0325905d70 login
2129 1 2129 88 DsJ (threaded) mysqld
108446 S umtxn 0xfffffe0d99f8c380 mysqld
108436 S umtxn 0xfffffe05f7e9cb00 mysqld
108431 S umtxn 0xfffffe0a23934200 mysqld
108413 S umtxn 0xfffffe09e6cedb00 mysqld
108397 S umtxn 0xfffffe02d4458080 mysqld
108389 S umtxn 0xfffffe0ab413ea00 mysqld
108352 S umtxn 0xfffffe07612efe00 mysqld
108350 S umtxn 0xfffffe0ddf748180 mysqld
108332 S umtxn 0xfffffe0d89b09100 mysqld
108326 S umtxn 0xfffffe0254472880 mysqld
108322 S umtxn 0xfffffe09dd542300 mysqld
108318 S umtxn 0xfffffe09371be200 mysqld
108316 S umtxn 0xfffffe09371be380 mysqld
108311 S umtxn 0xfffffe0e95eff700 mysqld
108294 S umtxn 0xfffffe0a886e0680 mysqld
108263 S umtxn 0xfffffe0ee0175780 mysqld
108253 S umtxn 0xfffffe0ee0175a80 mysqld
108239 S umtxn 0xfffffe0573530a80 mysqld
108236 S umtxn 0xfffffe0573530d80 mysqld
108234 S umtxn 0xfffffe096063cc80 mysqld
108214 S umtxn 0xfffffe07afb6ac80 mysqld
108213 S umtxn 0xfffffe018c8dbc00 mysqld
108204 S umtxn 0xfffffe0ec8cffc80 mysqld
108192 S umtxn 0xfffffe0aec5fd500 mysqld
108167 S umtxn 0xfffffe020ce9a600 mysqld
108164 S umtxn 0xfffffe094f908e00 mysqld
108145 S umtxn 0xfffffe01361ccc80 mysqld
108113 S umtxn 0xfffffe03b7288200 mysqld
108104 S umtxn 0xfffffe0543900500 mysqld
108086 S umtxn 0xfffffe0575f95a80 mysqld
108077 S umtxn 0xfffffe02f9862c80 mysqld
108069 S umtxn 0xfffffe0f6b7d3480 mysqld
108055 S umtxn 0xfffffe0967e77300 mysqld
108054 S umtxn 0xfffffe0d397a8080 mysqld
108050 S umtxn 0xfffffe013f554980 mysqld
108038 S umtxn 0xfffffe01248d0780 mysqld
108035 S umtxn 0xfffffe06e288f100 mysqld
108025 S umtxn 0xfffffe0299997880 mysqld
108022 S umtxn 0xfffffe0444f7ed80 mysqld
107991 S umtxn 0xfffffe0b0769bb00 mysqld
107969 S umtxn 0xfffffe01bf108480 mysqld
107944 S umtxn 0xfffffe0a230e2b80 mysqld
107916 S umtxn 0xfffffe017e2b4c80 mysqld
107884 S umtxn 0xfffffe017e2b4a80 mysqld
107883 S umtxn 0xfffffe0418fc7480 mysqld
107871 S umtxn 0xfffffe03acb51b00 mysqld
107855 S umtxn 0xfffffe08ed201c80 mysqld
107824 S umtxn 0xfffffe0aece27d00 mysqld
107820 S umtxn 0xfffffe0bb1988c00 mysqld
107809 S umtxn 0xfffffe0de2182a00 mysqld
107794 S umtxn 0xfffffe06ad810380 mysqld
107787 S umtxn 0xfffffe0f56112100 mysqld
107781 S umtxn 0xfffffe0f56112380 mysqld
107736 S umtxn 0xfffffe05a29c6100 mysqld
107717 S umtxn 0xfffffe0c179dd980 mysqld
107716 S umtxn 0xfffffe0650e87a00 mysqld
107709 S umtxn 0xfffffe0650e87c80 mysqld
107690 S umtxn 0xfffffe0ed40b6880 mysqld
107674 S umtxn 0xfffffe02c3e7bc80 mysqld
107664 S umtxn 0xfffffe02c3e7b080 mysqld
107645 S umtxn 0xfffffe04ff252e00 mysqld
107641 S umtxn 0xfffffe01d77a3800 mysqld
107640 S umtxn 0xfffffe04bd459400 mysqld
107629 S umtxn 0xfffffe02cfeba700 mysqld
107600 S umtxn 0xfffffe03406e8980 mysqld
107598 S umtxn 0xfffffe0d6cdd5c00 mysqld
107581 S umtxn 0xfffffe03406e8880 mysqld
107580 S umtxn 0xfffffe0b1cee4d80 mysqld
107569 S umtxn 0xfffffe09e6c14580 mysqld
107567 S umtxn 0xfffffe08b2a70500 mysqld
107563 S umtxn 0xfffffe03df90e100 mysqld
107562 S umtxn 0xfffffe0cdbee6580 mysqld
107556 S umtxn 0xfffffe0bbab0d200 mysqld
107527 S umtxn 0xfffffe0fb315cd00 mysqld
107526 S umtxn 0xfffffe057338e580 mysqld
107525 S umtxn 0xfffffe0bbab0db00 mysqld
107517 S umtxn 0xfffffe01db5b7700 mysqld
107516 S umtxn 0xfffffe01db5b7900 mysqld
107507 S umtxn 0xfffffe0e7bc38c80 mysqld
107472 S umtxn 0xfffffe0e7bc38a00 mysqld
107460 S umtxn 0xfffffe02f05c9000 mysqld
107449 S umtxn 0xfffffe02cbde6d00 mysqld
107443 S umtxn 0xfffffe0e64653d80 mysqld
107422 S umtxn 0xfffffe0dc6963700 mysqld
107420 S umtxn 0xfffffe06d8f20680 mysqld
107418 S umtxn 0xfffffe02a812e480 mysqld
107417 S umtxn 0xfffffe035e4cdb00 mysqld
107403 S umtxn 0xfffffe016ca31300 mysqld
107399 S umtxn 0xfffffe0d73b0bb00 mysqld
107382 S umtxn 0xfffffe03d4208980 mysqld
107352 S umtxn 0xfffffe0eb2423d00 mysqld
107340 S umtxn 0xfffffe05c0dc6280 mysqld
107337 S umtxn 0xfffffe0b0f27c600 mysqld
107333 S umtxn 0xfffffe0a519d8700 mysqld
107312 S umtxn 0xfffffe0c5451da80 mysqld
107308 S umtxn 0xfffffe0f0df60b00 mysqld
107305 S umtxn 0xfffffe06487eb980 mysqld
107303 S umtxn 0xfffffe0a3df35980 mysqld
107299 S umtxn 0xfffffe0f58fea700 mysqld
107291 S umtxn 0xfffffe0a5f6b0300 mysqld
107278 S umtxn 0xfffffe020095be00 mysqld
107277 S umtxn 0xfffffe03783a2200 mysqld
107264 S umtxn 0xfffffe0595e24d00 mysqld
107260 S umtxn 0xfffffe0d989d6400 mysqld
107255 S umtxn 0xfffffe0c25a91380 mysqld
107254 S umtxn 0xfffffe0dfc812080 mysqld
107253 S umtxn 0xfffffe08ed8a6a80 mysqld
107251 S umtxn 0xfffffe039ee19700 mysqld
107249 S umtxn 0xfffffe039ee19980 mysqld
107221 S umtxn 0xfffffe0d807c2e00 mysqld
107215 S umtxn 0xfffffe0a4d784380 mysqld
107207 S umtxn 0xfffffe06e90da980 mysqld
107204 S umtxn 0xfffffe0cc3992780 mysqld
107200 S umtxn 0xfffffe0512754d80 mysqld
107190 S umtxn 0xfffffe0d7a47b100 mysqld
107173 S umtxn 0xfffffe03a16c5c80 mysqld
107171 S umtxn 0xfffffe0404607c00 mysqld
107170 S umtxn 0xfffffe0cc8c81980 mysqld
107162 S umtxn 0xfffffe0ad8a51580 mysqld
107157 S umtxn 0xfffffe0631cc2380 mysqld
107152 S umtxn 0xfffffe0de3bd9500 mysqld
107129 S umtxn 0xfffffe0e32d8d600 mysqld
107121 S umtxn 0xfffffe0e32d8dc00 mysqld
107120 S umtxn 0xfffffe012738b780 mysqld
107110 S umtxn 0xfffffe0e589c6880 mysqld
107107 S umtxn 0xfffffe088617f880 mysqld
107101 S umtxn 0xfffffe0937ba2c80 mysqld
107098 S umtxn 0xfffffe022e48bd80 mysqld
107085 S umtxn 0xfffffe0886039980 mysqld
107084 S umtxn 0xfffffe094f042080 mysqld
107069 S umtxn 0xfffffe03df31b180 mysqld
107045 S umtxn 0xfffffe023a19a780 mysqld
107036 S umtxn 0xfffffe07a002ed80 mysqld
107035 S umtxn 0xfffffe021bcee280 mysqld
107022 S umtxn 0xfffffe0a51abcc80 mysqld
107007 S umtxn 0xfffffe0da93f9400 mysqld
107006 S umtxn 0xfffffe0687561300 mysqld
106997 S umtxn 0xfffffe0b76c23280 mysqld
106995 S umtxn 0xfffffe0faf70b800 mysqld
106985 S umtxn 0xfffffe04ff405680 mysqld
106980 S umtxn 0xfffffe0b7f5ba680 mysqld
106973 S umtxn 0xfffffe0393be2c00 mysqld
106965 S umtxn 0xfffffe03f50fcc00 mysqld
106962 S umtxn 0xfffffe0e8034bb80 mysqld
106933 S umtxn 0xfffffe03b4bb0680 mysqld
106932 S umtxn 0xfffffe0dc1ac0680 mysqld
106916 S umtxn 0xfffffe0abcaab980 mysqld
106883 S umtxn 0xfffffe043db24300 mysqld
106876 S umtxn 0xfffffe072773be00 mysqld
106871 S umtxn 0xfffffe0e1d447100 mysqld
106863 S umtxn 0xfffffe07a052b780 mysqld
106847 S umtxn 0xfffffe0ad78da800 mysqld
106844 S umtxn 0xfffffe0445acfc00 mysqld
106842 S umtxn 0xfffffe025bba2e00 mysqld
106841 S umtxn 0xfffffe036539c280 mysqld
106810 S umtxn 0xfffffe058719b800 mysqld
106786 S umtxn 0xfffffe0e32703880 mysqld
106781 S umtxn 0xfffffe03dfed7c00 mysqld
106779 S umtxn 0xfffffe0c686f7a00 mysqld
106769 S umtxn 0xfffffe01bf889c00 mysqld
106760 S umtxn 0xfffffe07da3bb800 mysqld
106754 S umtxn 0xfffffe044e35a800 mysqld
106744 S umtxn 0xfffffe03d6e88900 mysqld
106714 S umtxn 0xfffffe075f052180 mysqld
106710 S umtxn 0xfffffe0d80cd7a80 mysqld
106692 S umtxn 0xfffffe03df96a500 mysqld
106690 S umtxn 0xfffffe04bd2fcc80 mysqld
106673 S umtxn 0xfffffe0cc75d1300 mysqld
106656 S umtxn 0xfffffe03b58d0180 mysqld
106650 S umtxn 0xfffffe0516d16480 mysqld
106647 S umtxn 0xfffffe056ea70d80 mysqld
106643 S umtxn 0xfffffe0f8a322b00 mysqld
106641 S umtxn 0xfffffe0d9834e480 mysqld
106633 S umtxn 0xfffffe02306f7900 mysqld
106604 S umtxn 0xfffffe034dc04000 mysqld
106583 S umtxn 0xfffffe0d7aad5880 mysqld
106579 S umtxn 0xfffffe0abc16a600 mysqld
106561 S umtxn 0xfffffe03d6188680 mysqld
106537 S umtxn 0xfffffe01afebe000 mysqld
106521 S umtxn 0xfffffe0360772700 mysqld
106516 S umtxn 0xfffffe025a7a4900 mysqld
106506 S umtxn 0xfffffe03423fb480 mysqld
106495 S umtxn 0xfffffe06ea2d2280 mysqld
106488 S umtxn 0xfffffe0a2f142d00 mysqld
106487 S umtxn 0xfffffe02f0a14c80 mysqld
106479 S umtxn 0xfffffe0ddf63c780 mysqld
106438 S umtxn 0xfffffe07be277100 mysqld
106433 S umtxn 0xfffffe037a37cc00 mysqld
106428 S umtxn 0xfffffe0d0eb3eb80 mysqld
106397 S umtxn 0xfffffe0d7a07c200 mysqld
106390 S umtxn 0xfffffe094f76d600 mysqld
106379 S umtxn 0xfffffe014423ab00 mysqld
106377 S umtxn 0xfffffe0721e68000 mysqld
106376 S umtxn 0xfffffe0d47d1a400 mysqld
106364 S umtxn 0xfffffe07a07d2980 mysqld
106348 S umtxn 0xfffffe0293963b00 mysqld
106311 S umtxn 0xfffffe03264fa080 mysqld
106302 S umtxn 0xfffffe02306f6780 mysqld
106274 S umtxn 0xfffffe0f60f58200 mysqld
106272 S umtxn 0xfffffe084f6f9100 mysqld
106250 S umtxn 0xfffffe09304f3300 mysqld
106242 S umtxn 0xfffffe005fa65180 mysqld
106241 S umtxn 0xfffffe009d3aea00 mysqld
106236 S umtxn 0xfffffe0fb3ffbc80 mysqld
106235 S umtxn 0xfffffe058d8a7480 mysqld
106218 S umtxn 0xfffffe0aefd36180 mysqld
106191 S umtxn 0xfffffe009d683880 mysqld
106187 S umtxn 0xfffffe024d76dc80 mysqld
106186 S umtxn 0xfffffe09bda65400 mysqld
106185 S umtxn 0xfffffe0ae627d780 mysqld
106175 S umtxn 0xfffffe0432633600 mysqld
106144 S umtxn 0xfffffe048381ae00 mysqld
106143 S umtxn 0xfffffe09bfb5c980 mysqld
db> bt 86738
Tracing pid 86738 tid 102823 td 0xfffffe0cc7ecc470
sched_switch() at sched_switch+0x13d
mi_switch() at mi_switch+0x196
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x121
zio_wait() at zio_wait+0x61
dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x22b
dmu_buf_hold_array() at dmu_buf_hold_array+0x67
dmu_read_uio() at dmu_read_uio+0x3f
zfs_freebsd_read() at zfs_freebsd_read+0x3e3
VOP_READ_APV() at VOP_READ_APV+0x44
vn_read() at vn_read+0x316
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x6c
sys_read() at sys_read+0x64
amd64_syscall() at amd64_syscall+0x5ea
Xfast_syscall() at Xfast_syscall+0xf7
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x4eeafc, rsp = 0x7fffffffd888, rbp = 0x3 ---
db> bt 86279
Tracing pid 86279 tid 104231 td 0xfffffe0360fd3470
sched_switch() at sched_switch+0x13d
mi_switch() at mi_switch+0x196
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x121
zio_wait() at zio_wait+0x61
dmu_tx_count_write() at dmu_tx_count_write+0x34b
dmu_tx_hold_write() at dmu_tx_hold_write+0x4a
zfs_freebsd_write() at zfs_freebsd_write+0x43b
VOP_WRITE_APV() at VOP_WRITE_APV+0xc6
vn_write() at vn_write+0x38c
dofilewrite() at dofilewrite+0x8b
kern_writev() at kern_writev+0x6c
sys_write() at sys_write+0x64
amd64_syscall() at amd64_syscall+0x5ea
Xfast_syscall() at Xfast_syscall+0xf7
--- syscall (4, FreeBSD ELF64, sys_write), rip = 0x802aa0b2c, rsp = 0x7fffffffc488, rbp = 0x7fffffffc4b0 ---
db> bt 102355
Tracing pid 11143 tid 102355 td 0xfffffe0c8214a8e0
sched_switch() at sched_switch+0x13d
mi_switch() at mi_switch+0x196
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x121
zio_wait() at zio_wait+0x61
zil_commit() at zil_commit+0x3e5
zfs_freebsd_fsync() at zfs_freebsd_fsync+0xdc
VOP_FSYNC_APV() at VOP_FSYNC_APV+0x4a
sys_fsync() at sys_fsync+0x187
amd64_syscall() at amd64_syscall+0x5ea
Xfast_syscall() at Xfast_syscall+0xf7
--- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x80237848c, rsp = 0x7ffffdfefd98, rbp = 0x7ffffdfefdb0 ---
db> bt 86292
Tracing pid 86292 tid 101376 td 0xfffffe006152c000
sched_switch() at sched_switch+0x13d
mi_switch() at mi_switch+0x196
sleepq_wait() at sleepq_wait+0x42
__lockmgr_args() at __lockmgr_args+0x610
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x52
_vn_lock() at _vn_lock+0x47
vget() at vget+0x70
cache_lookup_times() at cache_lookup_times+0x5f5
vfs_cache_lookup() at vfs_cache_lookup+0xc8
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x4c
lookup() at lookup+0x424
namei() at namei+0x535
vn_open_cred() at vn_open_cred+0x3cb
kern_openat() at kern_openat+0x1f9
amd64_syscall() at amd64_syscall+0x5ea
Xfast_syscall() at Xfast_syscall+0xf7
--- syscall (5, FreeBSD ELF64, sys_open), rip = 0x8010a00ac, rsp = 0x7fffffffbd28, rbp = 0x7fffffffbdc0 ---
db> bt 87125
Tracing pid 87125 tid 103485 td 0xfffffe0cb6330000
sched_switch() at sched_switch+0x13d
mi_switch() at mi_switch+0x196
sleepq_wait() at sleepq_wait+0x42
__lockmgr_args() at __lockmgr_args+0x610
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x52
_vn_lock() at _vn_lock+0x47
vget() at vget+0x70
cache_lookup_times() at cache_lookup_times+0x5f5
vfs_cache_lookup() at vfs_cache_lookup+0xc8
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x4c
lookup() at lookup+0x424
namei() at namei+0x535
vn_open_cred() at vn_open_cred+0x3cb
kern_openat() at kern_openat+0x1f9
amd64_syscall() at amd64_syscall+0x5ea
Xfast_syscall() at Xfast_syscall+0xf7
--- syscall (5, FreeBSD ELF64, sys_open), rip = 0x4e1a1c, rsp = 0x7fffffffd8b8, rbp = 0 ---
db> bt 86928
Tracing pid 86928 tid 104507 td 0xfffffe0e2ffdb000
sched_switch() at sched_switch+0x13d
mi_switch() at mi_switch+0x196
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x121
zio_wait() at zio_wait+0x61
dbuf_read() at dbuf_read+0x5e5
dmu_buf_hold() at dmu_buf_hold+0xe0
zap_lockdir() at zap_lockdir+0x58
zap_lookup_norm() at zap_lookup_norm+0x45
zap_lookup() at zap_lookup+0x2e
zfs_dirent_lock() at zfs_dirent_lock+0x4ff
zfs_dirlook() at zfs_dirlook+0x69
zfs_lookup() at zfs_lookup+0x26b
zfs_freebsd_lookup() at zfs_freebsd_lookup+0x81
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x44
vfs_cache_lookup() at vfs_cache_lookup+0xf8
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x4c
lookup() at lookup+0x424
namei() at namei+0x535
kern_statat_vnhook() at kern_statat_vnhook+0xb6
kern_statat() at kern_statat+0x15
sys_stat() at sys_stat+0x2a
amd64_syscall() at amd64_syscall+0x5ea
Xfast_syscall() at Xfast_syscall+0xf7
--- syscall (188, FreeBSD ELF64, sys_stat), rip = 0x800da971c, rsp = 0x7fffffffd6c8, rbp = 0x7fffffffd7f0 ---
db> bt 86720
Tracing pid 86720 tid 104539 td 0xfffffe0ad6cce8e0
sched_switch() at sched_switch+0x13d
mi_switch() at mi_switch+0x196
sleepq_wait() at sleepq_wait+0x42
_sx_xlock_hard() at _sx_xlock_hard+0x4f1
_sx_xlock() at _sx_xlock+0x75
spa_all_configs() at spa_all_configs+0x5e
zfs_ioc_pool_configs() at zfs_ioc_pool_configs+0x29
zfsdev_ioctl() at zfsdev_ioctl+0xe6
devfs_ioctl_f() at devfs_ioctl_f+0x7b
kern_ioctl() at kern_ioctl+0x115
sys_ioctl() at sys_ioctl+0xfd
amd64_syscall() at amd64_syscall+0x5ea
Xfast_syscall() at Xfast_syscall+0xf7
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x801df0aec, rsp = 0x7fffffff83b8, rbp = 0x7fffffff8400 ---
db> bt 86079
Tracing pid 86079 tid 104370 td 0xfffffe06f80cd8e0
sched_switch() at sched_switch+0x13d
mi_switch() at mi_switch+0x196
sleepq_wait() at sleepq_wait+0x42
_sx_xlock_hard() at _sx_xlock_hard+0x4f1
_sx_xlock() at _sx_xlock+0x75
spa_open_common() at spa_open_common+0x7a
pool_status_check() at pool_status_check+0x19
zfsdev_ioctl() at zfsdev_ioctl+0x223
devfs_ioctl_f() at devfs_ioctl_f+0x7b
kern_ioctl() at kern_ioctl+0x115
sys_ioctl() at sys_ioctl+0xfd
amd64_syscall() at amd64_syscall+0x5ea
Xfast_syscall() at Xfast_syscall+0xf7
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x801befaec, rsp = 0x7fffffff8ed8, rbp = 0x7fffffff8f20 ---
db> bt 85728
Tracing pid 85728 tid 105053 td 0xfffffe0294bc5470
sched_switch() at sched_switch+0x13d
mi_switch() at mi_switch+0x196
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x121
txg_wait_synced() at txg_wait_synced+0x85
dsl_sync_task_group_wait() at dsl_sync_task_group_wait+0x128
dsl_dataset_user_release() at dsl_dataset_user_release+0x1c9
dsl_dataset_user_release_tmp() at dsl_dataset_user_release_tmp+0xee
dsl_dataset_user_release_onexit() at dsl_dataset_user_release_onexit+0x21
zfs_onexit_destroy() at zfs_onexit_destroy+0x56
zfsdev_close() at zfsdev_close+0x4f
devfs_destroy_cdevpriv() at devfs_destroy_cdevpriv+0x69
devfs_close_f() at devfs_close_f+0x54
_fdrop() at _fdrop+0x23
closef() at closef+0x52
kern_close() at kern_close+0x172
amd64_syscall() at amd64_syscall+0x5ea
Xfast_syscall() at Xfast_syscall+0xf7
--- syscall (6, FreeBSD ELF64, sys_close), rip = 0x801befb0c, rsp = 0x7fffffffc5a8, rbp = 0x7fffffffcff0 ---
--------------070005030706090006050108--
More information about the freebsd-scsi
mailing list