From nobody Wed Apr 03 11:17:50 2024 X-Original-To: freebsd-drivers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V8j0X2F49z5Gb9R for ; Wed, 3 Apr 2024 11:18:04 +0000 (UTC) (envelope-from bsd.m0osk@gmail.com) Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4V8j0W1Fk2z4GMf for ; Wed, 3 Apr 2024 11:18:03 +0000 (UTC) (envelope-from bsd.m0osk@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=LrZ0BsYS; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of bsd.m0osk@gmail.com designates 2607:f8b0:4864:20::1031 as permitted sender) smtp.mailfrom=bsd.m0osk@gmail.com Received: by mail-pj1-x1031.google.com with SMTP id 98e67ed59e1d1-29ddfada0d0so4665750a91.3 for ; Wed, 03 Apr 2024 04:18:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712143081; x=1712747881; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=StEaxiwNhT4pawoYBRomj/XdrL4RF9niskkFU0AJmCM=; b=LrZ0BsYS0GxxnEqZKHjjQX2aswNCmMaRHCJF53pJfFc8gWru1EH6W4zuRWQoJycshj 16hhtUWwbsvUroND/ETzbp4Ed3GG9yL454cwHd6ka8CnNLZ3Zad+vaxWIozQzbTLnHWD 6hyuiqgM8Wf+msIM4bL5u3SDwOSBgLDTCiKoNLgZOtX4ExqyFlKu6Uc91lLDJM07nmlj Cav5mupkUt7bMWvwQi0naJsM/EyomUyr5VcA2pRMqR9N7c3IoLM9DwH8vCS+x8FucnLW 7Oxi1wIHKQiDkAfoYbl75XiIi0E19Td3oSmYCp5WYCjnPc8C/z7qqtvlTfOlmZOurHrL FHhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712143081; x=1712747881; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=StEaxiwNhT4pawoYBRomj/XdrL4RF9niskkFU0AJmCM=; b=d4kdauA/Ra3NAOSS/WBWxp90F48N6U1tEi/Epl1J3nBtTrBvEZpqbMzAjXJrLK+5zP oE90pxmE3SsaHlZ0SbioyAfIPHWsS2WceuuzqFn5FDKWpC5Vsn8VIRNF6rmLEwSicNaq RM4JBRloeEpWYQlq9Xr8Yl2qZKDU/MiIyQQ8lvn7h6BedjYLAiHG10hk6/916JXmyrIn gEnZE8rBiU9rT+ZxapaqmjkKXq+kIWs/IeNeCZ2PiDZGErf98XuZfWJ43f2/yP+DDuO3 cEo/5X8elhaeBP87NfRuqp+kL6Q/8pbht1hnCUpKtd69ES8ruOkVvGcVuQ9eSgGMyP/J fREg== X-Gm-Message-State: AOJu0Yx931lvvdooaNdT2E6FMdYpMDpFip/U5NF4/gl396mAJCwIPJKM c5zIHwINzefsel4T3R6cGJGqESp09GTkA2vbMFqT0W5OuKxvzD3A+CKRspg21MDxAOike+H1jPp CMa5pOrbOy+SujrWp/GZFs+5IfjmpjU0Ni1v3tAbe X-Google-Smtp-Source: AGHT+IFozdt9xnxHm8sVUQiDaJyuXzbiD5zBQbxwWXy2KxmaCbO+rSp4Qjeah/2o0UiLuqpOwTeELHM0CbqGHznSAnA= X-Received: by 2002:a17:90a:ead1:b0:2a2:9843:927c with SMTP id ev17-20020a17090aead100b002a29843927cmr1112614pjb.27.1712143080922; Wed, 03 Apr 2024 04:18:00 -0700 (PDT) List-Id: Writing device drivers for FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-drivers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-drivers@freebsd.org MIME-Version: 1.0 From: Sarder Kamal Date: Wed, 3 Apr 2024 12:17:50 +0100 Message-ID: Subject: Error with mpi3mr driver To: freebsd-drivers@freebsd.org Content-Type: multipart/alternative; boundary="00000000000024a7f606152f6009" X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.86 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.997]; NEURAL_HAM_SHORT(-0.86)[-0.861]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TAGGED_FROM(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; PREVIOUSLY_DELIVERED(0.00)[freebsd-drivers@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-drivers@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1031:from] X-Rspamd-Queue-Id: 4V8j0W1Fk2z4GMf --00000000000024a7f606152f6009 Content-Type: text/plain; charset="UTF-8" Good afternoon everyone We recently procured a Dell PowerEdge R760xd2 server with PERC H965i Adapter, which has 24 SAS disks attached to it. I have set the disks to be non-raid (even though the controller is marked as RAID), and I can see the disks when I load mpi3mr driver. I am also able to create a raidz2 array with the disks (3 arrays of 7 disks and 3 hot spares). I have setup bhyve on the server and the relevant zpool is used for virtual machines. All seem to work, until I try to write back data from within the VMs. For example, if I try to get the ports using git clone -- it hangs after a while. There are no errors, it just hangs. I usually run the commands from a screen session, so I can open another screen session, but most of the commands that need to access the disks hangs -- even Ctrl+C does not work. In the meanwhile, in the host OS, I see these errors Apr 3 10:43:58 newDellServer kernel: mpi3mr0: bus_dmamap_load(): retcode = 36 Apr 3 10:43:58 newDellServer kernel: mpi3mr0: request load in progress Apr 3 10:43:58 newDellServer kernel: func: mpi3mr_action_scsiio line: 1143 Build SGLs failed Apr 3 10:43:58 newDellServer kernel: (da6:mpi3mr0:0:281:0): WRITE(10). CDB: 2a 00 15 6d f6 b8 00 07 e0 00 Apr 3 10:43:58 newDellServer kernel: (da6:mpi3mr0:0:281:0): CAM status: CCB request was invalid Apr 3 10:43:58 newDellServer kernel: (da6:mpi3mr0:0:281:0): Error 22, Unretryable error at different times it picks on different disks, but the essence of the error is the same My google search brought me to this earlier bug report https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272469 but I'm not sure if this is relevant -- I do not have that many cores (the server has only 96 cores in two physical CPUs). here are some of the details from the system # cat /etc/os-release NAME=FreeBSD VERSION="14.0-RELEASE-p6" VERSION_ID="14.0" ID=freebsd ANSI_COLOR="0;31" PRETTY_NAME="FreeBSD 14.0-RELEASE-p6" CPE_NAME="cpe:/o:freebsd:freebsd:14.0" HOME_URL="https://FreeBSD.org/" BUG_REPORT_URL="https://bugs.FreeBSD.org/" # dmesg | grep SMP FreeBSD/SMP: Multiprocessor System Detected: 96 CPUs FreeBSD/SMP: 2 package(s) x 24 core(s) x 2 hardware threads # sysctl kern.sched.topology_spec kern.sched.topology_spec: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47 NUMA node 0, 1 THREAD groupSMT group 2, 3 THREAD groupSMT group 4, 5 THREAD groupSMT group 6, 7 THREAD groupSMT group 8, 9 THREAD groupSMT group 10, 11 THREAD groupSMT group 12, 13 THREAD groupSMT group 14, 15 THREAD groupSMT group 16, 17 THREAD groupSMT group 18, 19 THREAD groupSMT group 20, 21 THREAD groupSMT group 22, 23 THREAD groupSMT group 24, 25 THREAD groupSMT group 26, 27 THREAD groupSMT group 28, 29 THREAD groupSMT group 30, 31 THREAD groupSMT group 32, 33 THREAD groupSMT group 34, 35 THREAD groupSMT group 36, 37 THREAD groupSMT group 38, 39 THREAD groupSMT group 40, 41 THREAD groupSMT group 42, 43 THREAD groupSMT group 44, 45 THREAD groupSMT group 46, 47 THREAD groupSMT group 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95 NUMA node 48, 49 THREAD groupSMT group 50, 51 THREAD groupSMT group 52, 53 THREAD groupSMT group 54, 55 THREAD groupSMT group 56, 57 THREAD groupSMT group 58, 59 THREAD groupSMT group 60, 61 THREAD groupSMT group 62, 63 THREAD groupSMT group 64, 65 THREAD groupSMT group 66, 67 THREAD groupSMT group 68, 69 THREAD groupSMT group 70, 71 THREAD groupSMT group 72, 73 THREAD groupSMT group 74, 75 THREAD groupSMT group 76, 77 THREAD groupSMT group 78, 79 THREAD groupSMT group 80, 81 THREAD groupSMT group 82, 83 THREAD groupSMT group 84, 85 THREAD groupSMT group 86, 87 THREAD groupSMT group 88, 89 THREAD groupSMT group 90, 91 THREAD groupSMT group 92, 93 THREAD groupSMT group 94, 95 THREAD groupSMT group # zpool status -v pool: Data02 state: ONLINE config: NAME STATE READ WRITE CKSUM Data02 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 da7 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10 ONLINE 0 0 0 da11 ONLINE 0 0 0 da12 ONLINE 0 0 0 da13 ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 da14 ONLINE 0 0 0 da15 ONLINE 0 0 0 da16 ONLINE 0 0 0 da17 ONLINE 0 0 0 da18 ONLINE 0 0 0 da19 ONLINE 0 0 0 da20 ONLINE 0 0 0 spares da21 AVAIL da22 AVAIL da23 AVAIL errors: No known data errors # pcicon -lv (truncated) mpi3mr0@pci0:80:0:0: class=0x010400 rev=0x01 hdr=0x00 vendor=0x1000 device=0x00a5 subvendor=0x1028 subdevice=0x2114 vendor = 'Broadcom / LSI' device = 'Fusion-MPT 24GSAS/PCIe SAS40xx' class = mass storage subclass = RAID I am happy to provide more data/details if that helps, please let me also know how to obtain them. Any suggestion to remedy the situation is greatly appreciated. Kind regards SK --00000000000024a7f606152f6009 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Good afternoon everyone

We recently pro= cured a Dell=C2=A0PowerEdge R760xd2 server with=C2=A0PERC H965i Adapter, wh= ich has 24 SAS disks attached to it. I have=C2=A0set the disks to be non-ra= id (even though the controller is marked as RAID), and I can see the disks = when I load mpi3mr driver.

I am also able to creat= e a raidz2 array with the disks (3 arrays of 7 disks and 3 hot spares). I h= ave setup bhyve on the server and the relevant zpool is used for virtual ma= chines.

All seem to work, until I try to write bac= k data from within the VMs. For example, if I try to get the ports using gi= t clone -- it hangs after a while. There are no errors, it just hangs. I us= ually run the commands from a screen session, so I can open another screen = session, but most of the commands that need to access the disks hangs -- ev= en Ctrl+C does not work.

In the meanwhile, in the = host OS, I see these errors
Apr =C2=A03 10:43:58 newDellServer ke= rnel: mpi3mr0: bus_dmamap_load(): retcode =3D 36
Apr =C2=A03 10:43:58 ne= wDellServer kernel: mpi3mr0: request load in progress
Apr =C2=A03 10:43:= 58 newDellServer kernel: func: mpi3mr_action_scsiio line: 1143 Build SGLs f= ailed
Apr =C2=A03 10:43:58 newDellServer kernel: (da6:mpi3mr0:0:281:0): = WRITE(10). CDB: 2a 00 15 6d f6 b8 00 07 e0 00
Apr =C2=A03 10:43:58 newD= ellServer kernel: (da6:mpi3mr0:0:281:0): CAM status: CCB request was invali= d
Apr =C2=A03 10:43:58 newDellServer kernel: (da6:mpi3mr0:0:281:0): Erro= r 22, Unretryable error

at different times it picks on di= fferent disks, but the essence of the error is the same

My google search brought me to this earlier bug report
but I&= #39;m not sure if this is relevant -- I do not have that many cores (the se= rver has only 96 cores in two physical CPUs).=C2=A0

here are some of the details from the system
# cat /etc/os-rele= ase
NAME=3DFreeBSD
VERSION=3D"14.0-RELEASE-p6"
VERSION_= ID=3D"14.0"
ID=3Dfreebsd
ANSI_COLOR=3D"0;31"
P= RETTY_NAME=3D"FreeBSD 14.0-RELEASE-p6"
CPE_NAME=3D"cpe:/o= :freebsd:freebsd:14.0"
HOME_URL=3D"https://FreeBSD.org/"
BUG_REPORT_URL=3D"https://bugs.FreeBSD.org/"

# dmes= g | grep SMP
FreeBSD/SMP: Multiprocessor System Detected: 96 CPUs
Fre= eBSD/SMP: 2 package(s) x 24 core(s) x 2 hardware threads

# sysctl ke= rn.sched.topology_spec
kern.sched.topology_spec: <groups>
=C2= =A0<group level=3D"1" cache-level=3D"0">
=C2= =A0 <cpu count=3D"96" mask=3D"ffffffffffffffff,ffffffff,0= ,0,0,0,0,0,0,0,0,0,0,0,0,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1= 2, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, = 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49,= 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68= , 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 8= 7, 88, 89, 90, 91, 92, 93, 94, 95</cpu>
=C2=A0 <children>=C2=A0 =C2=A0<group level=3D"2" cache-level=3D"3"&g= t;
=C2=A0 =C2=A0 <cpu count=3D"48" mask=3D"fffffffffff= f,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, = 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29,= 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47<= /cpu>
=C2=A0 =C2=A0 <flags><flag name=3D"NODE">= NUMA node</flag></flags>
=C2=A0 =C2=A0 <children>
= =C2=A0 =C2=A0 =C2=A0<group level=3D"3" cache-level=3D"2&q= uot;>
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"= ;3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">0, 1</cpu>
=C2=A0 =C2=A0= =C2=A0 <flags><flag name=3D"THREAD">THREAD group<= /flag><flag name=3D"SMT">SMT group</flag></flag= s>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<grou= p level=3D"3" cache-level=3D"2">
=C2=A0 =C2=A0 = =C2=A0 <cpu count=3D"2" mask=3D"c,0,0,0,0,0,0,0,0,0,0,0,0= ,0,0,0">2, 3</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><f= lag name=3D"THREAD">THREAD group</flag><flag name=3D&= quot;SMT">SMT group</flag></flags>
=C2=A0 =C2=A0 =C2= =A0</group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"3" ca= che-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"= ;2" mask=3D"30,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">4, 5</cp= u>
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"THREAD&quo= t;>THREAD group</flag><flag name=3D"SMT">SMT group= </flag></flags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0= =C2=A0 =C2=A0<group level=3D"3" cache-level=3D"2"&g= t;
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"c0,0,= 0,0,0,0,0,0,0,0,0,0,0,0,0,0">6, 7</cpu>
=C2=A0 =C2=A0 =C2= =A0 <flags><flag name=3D"THREAD">THREAD group</fla= g><flag name=3D"SMT">SMT group</flag></flags>= ;
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group le= vel=3D"3" cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0= <cpu count=3D"2" mask=3D"300,0,0,0,0,0,0,0,0,0,0,0,0,0,0= ,0">8, 9</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><flag = name=3D"THREAD">THREAD group</flag><flag name=3D"= ;SMT">SMT group</flag></flags>
=C2=A0 =C2=A0 =C2=A0&= lt;/group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"3" cache-= level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2&q= uot; mask=3D"c00,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">10, 11</cpu= >
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"THREAD"= ;>THREAD group</flag><flag name=3D"SMT">SMT group&= lt;/flag></flags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 = =C2=A0 =C2=A0<group level=3D"3" cache-level=3D"2">= ;
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"3000,0= ,0,0,0,0,0,0,0,0,0,0,0,0,0,0">12, 13</cpu>
=C2=A0 =C2=A0 = =C2=A0 <flags><flag name=3D"THREAD">THREAD group</= flag><flag name=3D"SMT">SMT group</flag></flags= >
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group= level=3D"3" cache-level=3D"2">
=C2=A0 =C2=A0 =C2= =A0 <cpu count=3D"2" mask=3D"c000,0,0,0,0,0,0,0,0,0,0,0,0= ,0,0,0">14, 15</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><= ;flag name=3D"THREAD">THREAD group</flag><flag name= =3D"SMT">SMT group</flag></flags>
=C2=A0 =C2=A0= =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"3"= ; cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D&= quot;2" mask=3D"30000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">16, = 17</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"T= HREAD">THREAD group</flag><flag name=3D"SMT">= SMT group</flag></flags>
=C2=A0 =C2=A0 =C2=A0</group><= br>=C2=A0 =C2=A0 =C2=A0<group level=3D"3" cache-level=3D"= 2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D&q= uot;c0000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">18, 19</cpu>
=C2= =A0 =C2=A0 =C2=A0 <flags><flag name=3D"THREAD">THREAD= group</flag><flag name=3D"SMT">SMT group</flag>= ;</flags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2= =A0<group level=3D"3" cache-level=3D"2">
=C2= =A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"300000,0,0,0,0= ,0,0,0,0,0,0,0,0,0,0,0">20, 21</cpu>
=C2=A0 =C2=A0 =C2=A0 = <flags><flag name=3D"THREAD">THREAD group</flag>= ;<flag name=3D"SMT">SMT group</flag></flags>=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group level= =3D"3" cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 &l= t;cpu count=3D"2" mask=3D"c00000,0,0,0,0,0,0,0,0,0,0,0,0,0,0= ,0">22, 23</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><fla= g name=3D"THREAD">THREAD group</flag><flag name=3D&qu= ot;SMT">SMT group</flag></flags>
=C2=A0 =C2=A0 =C2= =A0</group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"3" ca= che-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"= ;2" mask=3D"3000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">24, 25= </cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"THR= EAD">THREAD group</flag><flag name=3D"SMT">SM= T group</flag></flags>
=C2=A0 =C2=A0 =C2=A0</group>=C2=A0 =C2=A0 =C2=A0<group level=3D"3" cache-level=3D"2&= quot;>
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D&quo= t;c000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">26, 27</cpu>
=C2= =A0 =C2=A0 =C2=A0 <flags><flag name=3D"THREAD">THREAD= group</flag><flag name=3D"SMT">SMT group</flag>= ;</flags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2= =A0<group level=3D"3" cache-level=3D"2">
=C2= =A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"30000000,0,0,0= ,0,0,0,0,0,0,0,0,0,0,0,0">28, 29</cpu>
=C2=A0 =C2=A0 =C2= =A0 <flags><flag name=3D"THREAD">THREAD group</fla= g><flag name=3D"SMT">SMT group</flag></flags>= ;
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group le= vel=3D"3" cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0= <cpu count=3D"2" mask=3D"c0000000,0,0,0,0,0,0,0,0,0,0,0,= 0,0,0,0">30, 31</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags>&l= t;flag name=3D"THREAD">THREAD group</flag><flag name= =3D"SMT">SMT group</flag></flags>
=C2=A0 =C2=A0= =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"3"= ; cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D&= quot;2" mask=3D"300000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">= 32, 33</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D&qu= ot;THREAD">THREAD group</flag><flag name=3D"SMT"= >SMT group</flag></flags>
=C2=A0 =C2=A0 =C2=A0</group&= gt;
=C2=A0 =C2=A0 =C2=A0<group level=3D"3" cache-level=3D&q= uot;2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask= =3D"c00000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">34, 35</cpu>= ;
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"THREAD"&g= t;THREAD group</flag><flag name=3D"SMT">SMT group<= /flag></flags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2= =A0 =C2=A0<group level=3D"3" cache-level=3D"2">=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"300000000= 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">36, 37</cpu>
=C2=A0 =C2= =A0 =C2=A0 <flags><flag name=3D"THREAD">THREAD group&= lt;/flag><flag name=3D"SMT">SMT group</flag></f= lags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<g= roup level=3D"3" cache-level=3D"2">
=C2=A0 =C2=A0= =C2=A0 <cpu count=3D"2" mask=3D"c000000000,0,0,0,0,0,0,0= ,0,0,0,0,0,0,0,0">38, 39</cpu>
=C2=A0 =C2=A0 =C2=A0 <fl= ags><flag name=3D"THREAD">THREAD group</flag><f= lag name=3D"SMT">SMT group</flag></flags>
=C2= =A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group level=3D&q= uot;3" cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu= count=3D"2" mask=3D"30000000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0= ,0">40, 41</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><fla= g name=3D"THREAD">THREAD group</flag><flag name=3D&qu= ot;SMT">SMT group</flag></flags>
=C2=A0 =C2=A0 =C2= =A0</group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"3" ca= che-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"= ;2" mask=3D"c0000000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">42= , 43</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"= ;THREAD">THREAD group</flag><flag name=3D"SMT"&g= t;SMT group</flag></flags>
=C2=A0 =C2=A0 =C2=A0</group>= ;
=C2=A0 =C2=A0 =C2=A0<group level=3D"3" cache-level=3D&quo= t;2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D= "300000000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">44, 45</cpu>= ;
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"THREAD"&g= t;THREAD group</flag><flag name=3D"SMT">SMT group<= /flag></flags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2= =A0 =C2=A0<group level=3D"3" cache-level=3D"2">=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"c00000000= 000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">46, 47</cpu>
=C2=A0 =C2= =A0 =C2=A0 <flags><flag name=3D"THREAD">THREAD group&= lt;/flag><flag name=3D"SMT">SMT group</flag></f= lags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 </childr= en>
=C2=A0 =C2=A0</group>
=C2=A0 =C2=A0<group level=3D&qu= ot;2" cache-level=3D"3">
=C2=A0 =C2=A0 <cpu count= =3D"48" mask=3D"ffff000000000000,ffffffff,0,0,0,0,0,0,0,0,0,= 0,0,0,0,0">48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, = 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80,= 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95</cpu>
= =C2=A0 =C2=A0 <flags><flag name=3D"NODE">NUMA node<= ;/flag></flags>
=C2=A0 =C2=A0 <children>
=C2=A0 =C2=A0= =C2=A0<group level=3D"3" cache-level=3D"2">
= =C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"30000000000= 00,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">48, 49</cpu>
=C2=A0 =C2= =A0 =C2=A0 <flags><flag name=3D"THREAD">THREAD group&= lt;/flag><flag name=3D"SMT">SMT group</flag></f= lags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<g= roup level=3D"3" cache-level=3D"2">
=C2=A0 =C2=A0= =C2=A0 <cpu count=3D"2" mask=3D"c000000000000,0,0,0,0,0,= 0,0,0,0,0,0,0,0,0,0">50, 51</cpu>
=C2=A0 =C2=A0 =C2=A0 <= ;flags><flag name=3D"THREAD">THREAD group</flag>&l= t;flag name=3D"SMT">SMT group</flag></flags>
= =C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group level= =3D"3" cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 &l= t;cpu count=3D"2" mask=3D"30000000000000,0,0,0,0,0,0,0,0,0,0= ,0,0,0,0,0">52, 53</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags>= ;<flag name=3D"THREAD">THREAD group</flag><flag na= me=3D"SMT">SMT group</flag></flags>
=C2=A0 =C2= =A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"3&q= uot; cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu count= =3D"2" mask=3D"c0000000000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0&= quot;>54, 55</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><flag n= ame=3D"THREAD">THREAD group</flag><flag name=3D"= SMT">SMT group</flag></flags>
=C2=A0 =C2=A0 =C2=A0&l= t;/group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"3" cache-l= evel=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2&qu= ot; mask=3D"300000000000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">56,= 57</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"= THREAD">THREAD group</flag><flag name=3D"SMT">= ;SMT group</flag></flags>
=C2=A0 =C2=A0 =C2=A0</group>=
=C2=A0 =C2=A0 =C2=A0<group level=3D"3" cache-level=3D"= ;2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D&= quot;c00000000000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">58, 59</cpu&= gt;
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"THREAD"= >THREAD group</flag><flag name=3D"SMT">SMT group&l= t;/flag></flags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 = =C2=A0 =C2=A0<group level=3D"3" cache-level=3D"2">= ;
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"300000= 0000000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">60, 61</cpu>
=C2= =A0 =C2=A0 =C2=A0 <flags><flag name=3D"THREAD">THREAD= group</flag><flag name=3D"SMT">SMT group</flag>= ;</flags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2= =A0<group level=3D"3" cache-level=3D"2">
=C2= =A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"c0000000000000= 00,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0">62, 63</cpu>
=C2=A0 =C2= =A0 =C2=A0 <flags><flag name=3D"THREAD">THREAD group&= lt;/flag><flag name=3D"SMT">SMT group</flag></f= lags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<g= roup level=3D"3" cache-level=3D"2">
=C2=A0 =C2=A0= =C2=A0 <cpu count=3D"2" mask=3D"0,3,0,0,0,0,0,0,0,0,0,0,= 0,0,0,0">64, 65</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags>&l= t;flag name=3D"THREAD">THREAD group</flag><flag name= =3D"SMT">SMT group</flag></flags>
=C2=A0 =C2=A0= =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"3"= ; cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D&= quot;2" mask=3D"0,c,0,0,0,0,0,0,0,0,0,0,0,0,0,0">66, 67&l= t;/cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"THREA= D">THREAD group</flag><flag name=3D"SMT">SMT = group</flag></flags>
=C2=A0 =C2=A0 =C2=A0</group>
= =C2=A0 =C2=A0 =C2=A0<group level=3D"3" cache-level=3D"2&q= uot;>
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"= ;0,30,0,0,0,0,0,0,0,0,0,0,0,0,0,0">68, 69</cpu>
=C2=A0 =C2= =A0 =C2=A0 <flags><flag name=3D"THREAD">THREAD group&= lt;/flag><flag name=3D"SMT">SMT group</flag></f= lags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<g= roup level=3D"3" cache-level=3D"2">
=C2=A0 =C2=A0= =C2=A0 <cpu count=3D"2" mask=3D"0,c0,0,0,0,0,0,0,0,0,0,0= ,0,0,0,0">70, 71</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags>&= lt;flag name=3D"THREAD">THREAD group</flag><flag name= =3D"SMT">SMT group</flag></flags>
=C2=A0 =C2=A0= =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"3"= ; cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D&= quot;2" mask=3D"0,300,0,0,0,0,0,0,0,0,0,0,0,0,0,0">72, 73= </cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"THR= EAD">THREAD group</flag><flag name=3D"SMT">SM= T group</flag></flags>
=C2=A0 =C2=A0 =C2=A0</group>=C2=A0 =C2=A0 =C2=A0<group level=3D"3" cache-level=3D"2&= quot;>
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D&quo= t;0,c00,0,0,0,0,0,0,0,0,0,0,0,0,0,0">74, 75</cpu>
=C2=A0 = =C2=A0 =C2=A0 <flags><flag name=3D"THREAD">THREAD gro= up</flag><flag name=3D"SMT">SMT group</flag><= ;/flags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0&l= t;group level=3D"3" cache-level=3D"2">
=C2=A0 =C2= =A0 =C2=A0 <cpu count=3D"2" mask=3D"0,3000,0,0,0,0,0,0,0,= 0,0,0,0,0,0,0">76, 77</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags= ><flag name=3D"THREAD">THREAD group</flag><flag= name=3D"SMT">SMT group</flag></flags>
=C2=A0 = =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"= 3" cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu cou= nt=3D"2" mask=3D"0,c000,0,0,0,0,0,0,0,0,0,0,0,0,0,0">= ;78, 79</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D&q= uot;THREAD">THREAD group</flag><flag name=3D"SMT"= ;>SMT group</flag></flags>
=C2=A0 =C2=A0 =C2=A0</group= >
=C2=A0 =C2=A0 =C2=A0<group level=3D"3" cache-level=3D&= quot;2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask= =3D"0,30000,0,0,0,0,0,0,0,0,0,0,0,0,0,0">80, 81</cpu>=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"THREAD">TH= READ group</flag><flag name=3D"SMT">SMT group</fla= g></flags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 = =C2=A0<group level=3D"3" cache-level=3D"2">
= =C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"0,c0000,0,0= ,0,0,0,0,0,0,0,0,0,0,0,0">82, 83</cpu>
=C2=A0 =C2=A0 =C2= =A0 <flags><flag name=3D"THREAD">THREAD group</fla= g><flag name=3D"SMT">SMT group</flag></flags>= ;
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group le= vel=3D"3" cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0= <cpu count=3D"2" mask=3D"0,300000,0,0,0,0,0,0,0,0,0,0,0,= 0,0,0">84, 85</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><= flag name=3D"THREAD">THREAD group</flag><flag name=3D= "SMT">SMT group</flag></flags>
=C2=A0 =C2=A0 = =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"3"= cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D&q= uot;2" mask=3D"0,c00000,0,0,0,0,0,0,0,0,0,0,0,0,0,0">86, = 87</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"T= HREAD">THREAD group</flag><flag name=3D"SMT">= SMT group</flag></flags>
=C2=A0 =C2=A0 =C2=A0</group><= br>=C2=A0 =C2=A0 =C2=A0<group level=3D"3" cache-level=3D"= 2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D&q= uot;0,3000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0">88, 89</cpu>
= =C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D"THREAD">THR= EAD group</flag><flag name=3D"SMT">SMT group</flag= ></flags>
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 = =C2=A0<group level=3D"3" cache-level=3D"2">
= =C2=A0 =C2=A0 =C2=A0 <cpu count=3D"2" mask=3D"0,c000000,0= ,0,0,0,0,0,0,0,0,0,0,0,0,0">90, 91</cpu>
=C2=A0 =C2=A0 =C2= =A0 <flags><flag name=3D"THREAD">THREAD group</fla= g><flag name=3D"SMT">SMT group</flag></flags>= ;
=C2=A0 =C2=A0 =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group le= vel=3D"3" cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0= <cpu count=3D"2" mask=3D"0,30000000,0,0,0,0,0,0,0,0,0,0,= 0,0,0,0">92, 93</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags>&l= t;flag name=3D"THREAD">THREAD group</flag><flag name= =3D"SMT">SMT group</flag></flags>
=C2=A0 =C2=A0= =C2=A0</group>
=C2=A0 =C2=A0 =C2=A0<group level=3D"3"= ; cache-level=3D"2">
=C2=A0 =C2=A0 =C2=A0 <cpu count=3D&= quot;2" mask=3D"0,c0000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0">9= 4, 95</cpu>
=C2=A0 =C2=A0 =C2=A0 <flags><flag name=3D&quo= t;THREAD">THREAD group</flag><flag name=3D"SMT"&= gt;SMT group</flag></flags>
=C2=A0 =C2=A0 =C2=A0</group&g= t;
=C2=A0 =C2=A0 </children>
=C2=A0 =C2=A0</group>
=C2= =A0 </children>
=C2=A0</group>
</groups>

# z= pool status -v
=C2=A0 pool: Data02
=C2=A0state: ONLINE
config:
=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 NAME =C2=A0 =C2=A0 =C2=A0 =C2=A0STATE =C2= =A0 =C2=A0 READ WRITE CKSUM
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Data02 =C2=A0 = =C2=A0 =C2=A0ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0<= br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 raidz2-0 =C2=A0ONLINE =C2=A0 =C2=A0 = =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 da0 =C2=A0 =C2=A0 ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 = =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da1 =C2=A0 =C2= =A0 ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da2 =C2=A0 =C2=A0 ONLINE =C2=A0 =C2=A0 = =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 da3 =C2=A0 =C2=A0 ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 = =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da4 =C2=A0 =C2= =A0 ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da5 =C2=A0 =C2=A0 ONLINE =C2=A0 =C2=A0 = =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 da6 =C2=A0 =C2=A0 ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 = =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 raidz2-1 =C2=A0ONLINE= =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 da7 =C2=A0 =C2=A0 ONLINE =C2=A0 =C2=A0 =C2=A0 0= =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 da8 =C2=A0 =C2=A0 ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 = =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da9 =C2=A0 =C2=A0 ONL= INE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 da10 =C2=A0 =C2=A0ONLINE =C2=A0 =C2=A0 =C2=A0 = 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 da11 =C2=A0 =C2=A0ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 = =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da12 =C2=A0 =C2=A0ONL= INE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 da13 =C2=A0 =C2=A0ONLINE =C2=A0 =C2=A0 =C2=A0 = 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 rai= dz2-2 =C2=A0ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da14 =C2=A0 =C2=A0ONLINE =C2=A0= =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 da15 =C2=A0 =C2=A0ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0= =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da16= =C2=A0 =C2=A0ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da17 =C2=A0 =C2=A0ONLINE =C2= =A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 da18 =C2=A0 =C2=A0ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2= =A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 d= a19 =C2=A0 =C2=A0ONLINE =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2= =A0 0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da20 =C2=A0 =C2=A0ONLINE= =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 0
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 spares
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da21 =C2=A0 =C2= =A0 =C2=A0AVAIL =C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da22 =C2=A0 = =C2=A0 =C2=A0AVAIL =C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da23 =C2= =A0 =C2=A0 =C2=A0AVAIL =C2=A0

errors: No known data errors

<= /div>

# pcicon -lv (truncated)
mpi3mr0@pci0:80:0:0: = =C2=A0 =C2=A0class=3D0x010400 rev=3D0x01 hdr=3D0x00 vendor=3D0x1000 device= =3D0x00a5 subvendor=3D0x1028 subdevice=3D0x2114
=C2=A0 =C2=A0 vendor =C2= =A0 =C2=A0 =3D 'Broadcom / LSI'
=C2=A0 =C2=A0 device =C2=A0 =C2= =A0 =3D 'Fusion-MPT 24GSAS/PCIe SAS40xx'
=C2=A0 =C2=A0 class =C2= =A0 =C2=A0 =C2=A0=3D mass storage
=C2=A0 =C2=A0 subclass =C2=A0 =3D RAID=


I am happy to provide more data/details i= f that helps, please let me also know how to obtain them.

Any suggestion to remedy the situation is greatly appreciated.

Kind regards
SK

--00000000000024a7f606152f6009--