ZFS crashing while zfs recv in progress
Pascal Braun, Continum
pascal.braun at continum.net
Fri Jun 7 12:38:00 UTC 2013
first I'd like to thank you for your time and effort.
> - Disk da3 has a different drive firmware (A580) than the A800
> drives.
Somehow I did miss that. I can replace this disk with a A800 one, although I don't think this will change much.
> - I have not verified if any of these disks use 4KByte sectors (dmesg
> is
> not going to tell you the entire truth). I would appreciate seeing
> "smartctl -x" output from {da0,da1,da3} so I could get an idea.
> Your
> pools use gpt labelling so I am left with the hope that your labels
> refer to the partition with proper 4KB alignment regardless.
The 'tank' disks are real 512bytes disks. The zpool currently in use is ashift=9. I've also tried ashift=12 in the past, but it didn't help. You'll find the output of smartctl in the attachment.
> Can you tell me what exact disk (e.g. daXX) in the above list you
> used
> for swap, and what kind of both system and disk load were going on at
> the time you saw the swap message?
>
> I'm looking for a capture of "gstat -I500ms" output (you will need a
> VERY long/big terminal window to capture this given how many disks
> you
> have) while I/O is happening, as well as "top -s 1" in another
> window.
> I would also like to see "zpool iostat -v 1" output while things are
> going on, to help possibly narrow down if there is a single disk
> causing
> the entire I/O subsystem for that controller to choke.
The swap disk in use is da28.
The last output of top -s 1 that could be writen to disk was:
---
last pid: 3653; load averages: 0.03, 0.19, 0.30 up 0+15:55:50 03:04:33
43 processes: 1 running, 41 sleeping, 1 zombie
CPU: 0.3% user, 0.0% nice, 0.6% system, 0.1% interrupt, 99.0% idle
Mem: 7456K Active, 27M Inact, 6767M Wired, 3404K Cache, 9053M Free
Swap: 256G Total, 5784K Used, 256G Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
1917 root 1 22 0 33420K 2356K piperd 2 41:24 3.96% zfs
1913 root 1 21 0 71980K 5248K select 4 288:50 3.27% sshd
1853 root 1 20 0 29484K 2788K nanslp 0 3:13 0.00% gstat
1803 root 1 20 0 35476K 2128K nanslp 1 2:44 0.00% zpool
1798 root 1 20 0 16560K 2240K CPU0 7 1:07 0.00% top
1780 root 1 20 0 67884K 1792K select 2 0:23 0.00% sshd
1800 root 1 20 0 12052K 1484K select 6 0:17 0.00% script
1747 root 1 20 0 71980K 1868K select 1 0:13 0.00% sshd
3148 root 1 20 -20 21140K 8956K pause 7 0:11 0.00% atop
1850 root 1 20 0 12052K 1412K select 4 0:06 0.00% script
1784 root 1 20 0 67884K 1772K select 7 0:05 0.00% sshd
1652 nagios 1 20 0 12012K 1044K select 7 0:02 0.00% nrpe2
1795 root 1 20 0 12052K 1408K select 1 0:02 0.00% script
1538 root 1 20 0 11996K 960K nanslp 1 0:01 0.00% ipmon
1670 root 1 20 0 20272K 1876K select 1 0:01 0.00% sendmail
1677 root 1 20 0 14128K 1548K nanslp 2 0:00 0.00% cron
1547 root 1 20 0 12052K 1172K select 5 0:00 0.00% syslogd
---
The last output of zpool iostat -v 1
capacity operations bandwidth
pool alloc free read write read write
-------------- ----- ----- ----- ----- ----- -----
tank 1.19T 63.8T 95 0 360K 0
raidz2 305G 16.0T 25 0 92.2K 0
gpt/disk3 - - 16 0 8.47K 0
gpt/disk9 - - 17 0 18.9K 0
gpt/disk15 - - 12 0 6.98K 0
gpt/disk19 - - 12 0 6.48K 0
gpt/disk23 - - 21 0 14.0K 0
gpt/disk27 - - 18 0 10.5K 0
gpt/disk31 - - 18 0 9.47K 0
gpt/disk36 - - 16 0 18.4K 0
gpt/disk33 - - 12 0 15.5K 0
raidz2 305G 16.0T 25 0 103K 0
gpt/disk1 - - 16 0 8.47K 0
gpt/disk4 - - 24 0 16.0K 0
gpt/disk7 - - 17 0 10.5K 0
gpt/disk10 - - 17 0 8.97K 0
gpt/disk13 - - 25 0 15.5K 0
gpt/disk16 - - 15 0 8.97K 0
gpt/disk24 - - 15 0 7.98K 0
gpt/disk32 - - 25 0 16.9K 0
gpt/disk37 - - 16 0 9.47K 0
raidz2 305G 16.0T 20 0 81.3K 0
gpt/disk2 - - 9 0 4.98K 0
gpt/disk5 - - 20 0 14.0K 0
gpt/disk8 - - 18 0 10.5K 0
gpt/disk11 - - 18 0 9.47K 0
gpt/disk17 - - 20 0 11.5K 0
gpt/disk21 - - 12 0 6.48K 0
gpt/disk25 - - 12 0 6.48K 0
gpt/disk29 - - 20 0 13.0K 0
gpt/disk38 - - 9 0 4.98K 0
raidz2 305G 16.0T 22 0 83.7K 0
gpt/disk12 - - 15 0 7.98K 0
gpt/disk14 - - 18 0 19.4K 0
gpt/disk18 - - 14 0 16.0K 0
gpt/disk22 - - 15 0 7.98K 0
gpt/disk26 - - 19 0 13.0K 0
gpt/disk30 - - 10 0 5.98K 0
gpt/disk34 - - 10 0 5.48K 0
gpt/disk35 - - 18 0 17.9K 0
gpt/disk39 - - 15 0 7.98K 0
-------------- ----- ----- ----- ----- ----- -----
zroot 2.67G 925G 0 0 0 0
mirror 2.67G 925G 0 0 0 0
gpt/disk0 - - 0 0 0 0
gpt/disk6 - - 0 0 0 0
-------------- ----- ----- ----- ----- ----- -----
and the last output of gstat -I500ms
[gstat file]
dT: 0.503s w: 0.500s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 0 0 0 0.0 0 0 0.0 0.0| da0
0 10 10 5 4.6 0 0 0.0 4.6| da1
0 34 34 18 0.1 0 0 0.0 0.4| da2
0 8 8 6 0.2 0 0 0.0 0.1| da3
0 0 0 0 0.0 0 0 0.0 0.0| da0p1
0 0 0 0 0.0 0 0 0.0 0.0| da0p2
0 10 10 5 4.7 0 0 0.0 4.6| da1p1
0 34 34 18 0.2 0 0 0.0 0.5| da2p1
0 8 8 6 0.2 0 0 0.0 0.2| da3p1
0 18 18 12 0.2 0 0 0.0 0.2| da4
0 52 52 31 0.5 0 0 0.0 2.5| da5
0 0 0 0 0.0 0 0 0.0 0.0| da6
0 12 12 8 3.3 0 0 0.0 3.9| da7
0 32 32 17 0.1 0 0 0.0 0.4| da8
0 10 10 7 0.2 0 0 0.0 0.1| da9
0 12 12 7 0.1 0 0 0.0 0.2| da10
0 32 32 16 0.1 0 0 0.0 0.4| da11
0 42 42 22 0.1 0 0 0.0 0.5| da12
0 18 18 12 0.2 0 0 0.0 0.2| da13
0 62 62 38 0.1 0 0 0.0 0.8| da14
0 6 6 4 0.2 0 0 0.0 0.1| da15
0 14 14 8 0.2 0 0 0.0 0.2| da16
0 52 52 32 0.3 0 0 0.0 1.4| da17
0 40 40 21 0.1 0 0 0.0 0.5| da18
0 6 6 4 0.1 0 0 0.0 0.1| da19
0 0 0 0 0.0 0 0 0.0 0.0| da20
0 38 38 21 1.3 0 0 0.0 5.1| da21
0 40 40 20 0.1 0 0 0.0 0.5| da22
0 10 10 7 0.1 0 0 0.0 0.1| da23
0 14 14 8 3.4 0 0 0.0 4.7| da24
0 38 38 20 1.5 0 0 0.0 5.8| da25
0 62 62 39 0.1 0 0 0.0 0.8| da26
0 6 6 4 0.2 0 0 0.0 0.1| da27
0 0 0 0 0.0 0 0 0.0 0.0| da28
0 52 52 4 0.2 0 0 0.0 0.1| da29
0 70 70 36 0.1 0 0 0.0 0.9| da30
0 38 38 19 0.1 0 0 0.0 0.5| da31
0 0 0 0 0.0 0 0 0.0 0.0| da32
0 40 40 20 1.1 0 0 0.0 4.5| da33
0 70 70 35 0.1 0 0 0.0 0.9| da34
0 87 87 51 0.6 0 0 0.0 4.9| da35
0 54 54 32 0.1 0 0 0.0 0.7| da36
0 0 0 0 0.0 0 0 0.0 0.0| da37
0 8 8 4 18.8 0 0 0.0 3.8| da38
0 56 56 28 0.1 0 0 0.0 0.7| da39
[...]
---
> Next: are you using compression or dedup on any of your filesystems?
> If not, have you ever in the past?
No, this pool was build from scratch without any compression or dedup.
> Next: could we have your loader.conf and sysctl.conf please?
loader.conf
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"
console=comconsole
sysctl.conf is empty
> If you could put a swap disk on a dedicated controller (and no other
> disks on it), that would be ideal. Please do not use USB for this
> task
> (the USB stack may introduce its own set of complexities pertaining
> to
> interrupt usage).
I can't easily do this in the current setup. I would have to recreate the primary pool differently.
Thanks again,
Pascal
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: da0.txt
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20130607/bd8891ee/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: da1.txt
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20130607/bd8891ee/attachment-0001.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: da3.txt
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20130607/bd8891ee/attachment-0002.txt>
More information about the freebsd-stable
mailing list