From nobody Wed Jun 19 06:32:04 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4W3v190s4Tz5P3V1 for ; Wed, 19 Jun 2024 06:32:13 +0000 (UTC) (envelope-from lexi@le-fay.org) Received: from fuchsia.eden.le-Fay.ORG (fuchsia.eden.le-fay.org [81.187.47.195]) by mx1.freebsd.org (Postfix) with ESMTP id 4W3v182F2xz58v6 for ; Wed, 19 Jun 2024 06:32:12 +0000 (UTC) (envelope-from lexi@le-fay.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=le-fay.org header.s=fuchsia header.b=bBF9GbOX; dmarc=none; spf=pass (mx1.freebsd.org: domain of lexi@le-fay.org designates 81.187.47.195 as permitted sender) smtp.mailfrom=lexi@le-fay.org Received: from iris.eden.le-Fay.ORG (iris.eden.le-fay.org [IPv6:fd12:8247:3:1::7]) by fuchsia.eden.le-Fay.ORG (Postfix) with ESMTP id 273922002E for ; Wed, 19 Jun 2024 06:32:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=le-fay.org; s=fuchsia; t=1718778724; bh=OJxEbAvGddxxKV8f0nByhVmEr3Xb/eoBOzv6Z8RMQ54=; h=Date:From:To:Subject; b=bBF9GbOXjrkS6wPFGQlkcKIhIMspUYZC/IGpjLWOJ17YgPP96+sd4uKj7Now+RwlS LIDBCIhKzsZfc03/q5hpB2eTMM+Js7qL4ZN3vgYP3i0ojvU/cKZk7GAKtlI8bxgu+e WKWdToD8avyzlN+oN+D8g3506x1hwVCZFtS+i0lo= Received: from ilythia.eden.le-fay.org (ilythia.eden.le-fay.org [IPv6:2001:8b0:aab5:c003:3::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by iris.eden.le-Fay.ORG (Postfix) with ESMTPSA id 855A12C0471 for ; Wed, 19 Jun 2024 07:32:02 +0100 (BST) Date: Wed, 19 Jun 2024 07:32:04 +0100 From: Lexi Winter To: fs@freebsd.org Subject: NFS, intermittent 'RPC struct is bad' errors Message-ID: Mail-Followup-To: fs@freebsd.org List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="wKOJ5TJPneXaYjNl" Content-Disposition: inline X-Spamd-Bar: ----- X-Spamd-Result: default: False [-5.50 / 15.00]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; R_DKIM_ALLOW(-0.20)[le-fay.org:s=fuchsia]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; R_SPF_ALLOW(-0.20)[+ip4:81.187.47.195]; RCVD_NO_TLS_LAST(0.10)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:20712, ipnet:81.187.0.0/16, country:GB]; MISSING_XM_UA(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[le-fay.org:dkim]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DMARC_NA(0.00)[le-fay.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[fs@freebsd.org]; MLMMJ_DEST(0.00)[fs@freebsd.org]; DKIM_TRACE(0.00)[le-fay.org:+] X-Rspamd-Queue-Id: 4W3v182F2xz58v6 --wKOJ5TJPneXaYjNl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline hi, i have a few systems running NFSv4 on FreeBSD, using Kerberos (MIT Kerberos KDC), with the server exporting ZFS filesystems. recently i've noticed intermittent errors of 'RPC struct is bad' when writing to the NFS server, which usually resolves itself after retrying. for example: % rsync -iavP /scratch/Star.Trek.Prodigy.S01E* . sending incremental file list >f++++++++++ Star.Trek.Prodigy.S01E01E02.1080p.WEBRip.x265-KONTRAST.mkv 32,768 0% 0.00kB/s 0:00:00 rsync: [receiver] write failed on "/data/public/TV/Star Trek Prodigy/Season 01/Star.Trek.Prodigy.S01E01E02.1080p.WEBRip.x265-KONTRAST.mkv": RPC struct is bad (72) rsync error: error in file IO (code 11) at receiver.c(380) [receiver=3.3.0] rsync: [sender] write error: Broken pipe (32) % rsync -iavP /scratch/Star.Trek.Prodigy.S01E* . sending incremental file list >f.st....... Star.Trek.Prodigy.S01E01E02.1080p.WEBRip.x265-KONTRAST.mkv 912,704,431 100% 96.51MB/s 0:00:09 (xfr#1, to-chk=18/19) >f++++++++++ Star.Trek.Prodigy.S01E03.1080p.WEBRip.x265-KONTRAST.mkv 477,408,567 100% 100.06MB/s 0:00:04 (xfr#2, to-chk=17/19) [...] the client is running FreeBSD 15.0-CURRENT from around May 24, and the server is running a slightly older 15.0-CURRENT from around May 23. /etc/exports on the server is pretty standard: /data/public -sec=krb5:krb5i:krb5p -network 2001:8b0:aab5::/48 /data/public/Books -sec=krb5:krb5i:krb5p -network 2001:8b0:aab5::/48 /data/public/CalibreLibrary -sec=krb5:krb5i:krb5p -network 2001:8b0:aab5::/48 /data/public/Comics -sec=krb5:krb5i:krb5p -network 2001:8b0:aab5::/48 /data/public/Films -sec=krb5:krb5i:krb5p -network 2001:8b0:aab5::/48 /data/public/Miscellaneous -sec=krb5:krb5i:krb5p -network 2001:8b0:aab5::/48 V4: /data -sec=sys:krb5:krb5i:krb5p -network 2001:8b0:aab5::/48 client mount options: hemlock.eden.le-fay.org:/public /data/public nfs rw,nfsv4,minorversion=2,sec=krb5p,gssname=host,bgnow,proto=tcp6,nconnect=4,rsize=1048576,wsize=1048576,noncontigwr 0 0 is there anything more i can do investigate this? would a tcpdump capture of the error be useful (considering all the RPC traffic is Kerberos-encrypted)? --wKOJ5TJPneXaYjNl Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAABCAAdFiEEuwt6MaPcv/+Mo+ftDHqbqZ41x5kFAmZye2EACgkQDHqbqZ41 x5mcfAv9H6f6bJIhTIQovyJMQGiJO+VZyQtkSmvR9iup7CPVGnaEf1NfPasXnyLY OT7XdooA29sEHsJu5+hVgBj/H5YEN1gU+/jHUA2oqTh/LLXEsc18E8DtSHeHPz/h KBE5f8dA6s5vqcPMPRgh7egfVm0VvvMwzeGlCS4fZWXXNmBOXbh2OMRSXoo7I83e /d3CR819MgQNPpIkWtO2k7aYEHbaGUNJkcln9YvBPdgTgJII8xd7g0SD/mc0SeXe p3CNCbI/eECWngZJwnI6Nm0xn1FBcON4hcjbz5How/MXz8szASEzBBinVx2tBt1F TPeRdwXTyq1rhY8t1w5PCRfyRLKsxNpLAcjwbfpWCGEGFkdZg3oaaECXL0C7OaGD EoydpwsY16WzpK1ctzuZoTBTQ70q2uIweTWqbcy+8wWnZf+Jwg3hmPJUUGMtOlsx XahfBNDirs/m/SYYh6dYHtsSrwwbtJKF7TlPm4iRKCx/4nyKkbZ5/VyYvXRySrAs m6r8ItAQ =r4rW -----END PGP SIGNATURE----- --wKOJ5TJPneXaYjNl--