From nobody Mon Jan 24 08:29:59 2022 X-Original-To: freebsd-scsi@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5AB0D198BE86 for ; Mon, 24 Jan 2022 08:30:09 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jj3802YQkz4bBM for ; Mon, 24 Jan 2022 08:30:08 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800:0:0:0:0:a135]) by mx0.gentlemail.de (8.15.2/8.15.2) with ESMTP id 20O8U01O015002 for ; Mon, 24 Jan 2022 09:30:00 +0100 (CET) (envelope-from freebsd@omnilan.de) X-Authentication-Warning: mx0.gentlemail.de: Host ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800:0:0:0:0:a135] claimed to be mh0.gentlemail.de Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 4DAA3D48 for ; Mon, 24 Jan 2022 09:30:00 +0100 (CET) To: "freebsd-scsi@freebsd.org" From: Harry Schmalzbauer Subject: iscsi initiator problems Organization: OmniLAN Message-ID: <4bbb06e0-0e31-1064-b80d-1267bc9ea07d@omnilan.de> Date: Mon, 24 Jan 2022 09:29:59 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 List-Id: SCSI subsystem List-Archive: https://lists.freebsd.org/archives/freebsd-scsi List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4Jj3802YQkz4bBM X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@omnilan.de designates 2a00:e10:2800::a130 as permitted sender) smtp.mailfrom=freebsd@omnilan.de X-Spamd-Result: default: False [-0.91 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-scsi@freebsd.org]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; RCPT_COUNT_ONE(0.00)[1]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[omnilan.de]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.61)[-0.614]; TO_DN_EQ_ADDR_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-scsi]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:61157, ipnet:2a00:e10:2800::/38, country:DE]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Hi all, I'm not sure if this list fit's for iscsi (initiator) problems. But since I first need to get a clue about the following geom messages, I think the scsi experts probably know best about them: g_dev_ioctl: offset=312526336 length=2048 last message repeated 4 times g_dev_ioctl: offset=630456320 length=6707200 last message repeated 4 times g_dev_ioctl: offset=9738045440 length=16777216 last message repeated 4 times g_dev_ioctl: offset=9939372032 length=15710208 last message repeated 4 times g_dev_ioctl: offset=312529408 length=7168 last message repeated 4 times g_dev_ioctl: offset=9723707392 length=14338048 last message repeated 4 times g_dev_ioctl: offset=406888448 length=4283904 : g_dev_ioctl: offset=379645440 length=14749696 last message repeated 4 times g_dev_ioctl: offset=27418112 length=16777216 : g_dev_ioctl: offset=295853568 length=16675840 last message repeated 4 times g_dev_ioctl: offset=16927744 length=4608 last message repeated 4 times g_dev_ioctl: offset=10027428352 length=512 last message repeated 4 times g_dev_ioctl: offset=27418112 length=16777216 : g_dev_ioctl: offset=211967488 length=6787584 last message repeated 4 times g_dev_ioctl: offset=406897152 length=7680 last message repeated 4 times g_dev_ioctl: offset=9738045440 length=16777216 : g_dev_ioctl: offset=10023258112 length=4170240 last message repeated 4 times g_dev_ioctl: offset=630456320 length=6787584 last message repeated 4 times g_dev_ioctl: offset=406904832 length=4267520 last message repeated 4 times g_dev_ioctl: offset=406888448 length=8704 last message repeated 4 times g_dev_ioctl: offset=312529408 length=2567168 last message repeated 4 times g_dev_ioctl: offset=16927744 length=4608 last message repeated 4 times g_dev_ioctl: offset=233229312 length=512 last message repeated 4 times g_dev_ioctl: offset=630456320 length=6792192 last message repeated 4 times g_dev_ioctl: offset=233230336 length=7680 last message repeated 4 times g_dev_ioctl: offset=637248512 length=16777216 Skipped are notably many reading the following numbers: kernel: g_dev_ioctl: offset=NOT4kaligned length=16777216 (many times more often occuring than the excerpt above) The excerpt above is from messages of the iscsi INITIATOR (stable/13, amd64). They occur if a bhyve(4) guest (win2k832) utilizes a virti-blk provided disk. The other side is also FreeBSD stable/13 (amd64) and ctl/ctld(8) providing a ZVOL-dataset block backend. In ctld(8), I define a blocksize of 4k (volblocksize is 64k) for the ctl target. I'm using such kind of provider/_target_ setups for years _without any problems_ using ESXi or/and Windows (soft-)iSCSI-initiator. The failure of the ctld-IP-iscsid->virtio-blk chain is completely unclear to me and the only obvious info from the kernel error messages shown above is that either one of "offset" or "length" is _not_ 4k aligned. The guest OS is aware about 4k drives (passing virtio-blk with .sectorsize=4096) and I can transfer some hundred gigabytes, but at some point, the OS marks the disk as inaccessable. Most likely due to the errors from above (kernel log of the bhyve(4) host acting as iscsi initiator). I also never had such a issue when using the exactly same server/target with the _Windows software iSCSI-initiator_, so I guess it's a problem with the FreeBSD iscsi initiator. Unfortunately, before this error occured, I was hit by another, problem: https://lists.freebsd.org/pipermail/freebsd-scsi/2017-June/007385.html iscsi-initiator transfers over ethernet with JumboFrames (MTU of 9000) dont work at all. Not for the reason that allocating jumbo mbufs failed, but with the same symptoms like the thread discusses. Reducing mtu to 4096 by setting a host-route to each other on both sides solved the problem (at least to some degree). When the messages from above arose, this host route was set to 1500 bytes. I disabled JumboFrames on the NICs but didn't want to change it manually on the NICs since I currently have no maintenance window to check for unwanted side effects. (if_lagg(4) has two i210 and LACP, providing if_vlan(4) children on both sides). Since networking on all my machines using Intel NICs is broken to some degree since the latest August driver updates (stable/13), I can imagine NIC drops to be the culprit in this case too.  But I have no chance to rule that out nor confirm with testing (production env and neither time nor equipment handy to copy)... What do you think, the error messages above are telling? Thanks, -harry