From nobody Mon Sep 30 20:07:17 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XHXDM1vtxz5XbZl for ; Mon, 30 Sep 2024 20:07:31 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XHXDL3Xz4z4FxL for ; Mon, 30 Sep 2024 20:07:30 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=F1g1ftNj; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2a00:1450:4864:20::62d as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-x62d.google.com with SMTP id a640c23a62f3a-a8a706236bfso398161466b.0 for ; Mon, 30 Sep 2024 13:07:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1727726849; x=1728331649; darn=freebsd.org; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=0gZ4eHJApbB/Bs8jdXLHQhGEk8OIWD1gXoqH6FRSEb0=; b=F1g1ftNjjiJ5hK2ucIfMNGUlcgL08aa538MbDxB5qHEvcCIA4xcQ2CrvjbnL07EJ60 lEg+ATzmG+GheBCsNZvvN2tkBIv35TFMWhzY5+SWIfrLdBIqe3APTI0BnY+X8B0MuQmA nnDSKnm6334nRM06tCEsJDPPkQQjgTGFeX/SXcZS70zEia/h7/BtYEOaSP7jCCnqcdtw +3Ldmkftb59xlbcpWUsnblUThnkAZtvjFY0qjYGhe0B1fiy9RqKcnH8IhwFrIuMAqBB6 fFwpdB1oIzvvfcpXUNWEx5rxGlgw3TP7m347IgrzVpyBIM/zfGutAz1CHeZ2iNIP45Qe y0/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727726849; x=1728331649; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0gZ4eHJApbB/Bs8jdXLHQhGEk8OIWD1gXoqH6FRSEb0=; b=UtK3A5rowEX96edAqwHdi8NJnvp4teMVZhbHgH/gvXR4mMbZGfA8aaLC8eB/1glq42 6oDDAbz4NpO/LIOyNxw73vOwoasvvcvM9A3qyMQzz5q5oooaPXaIlpwdBTlBZAz+TCUO K7tSYlMKD2ChLL3Sb6VfLc1kDGM2VpBx2gC5Up6xB3pGpBNlUW8vIf3ojyikdwl841Di AbAFY60ASVmWxXaDj0AUyjlv4xQ3Ij7VHL6VCqmW9zsbnPfdskXpFDv/dvw16NpmbYB6 4OT/QI03UgjEKXvlM4hZhNCG95UYlpus+8TxfBr3igcd7/JJXUcEmEG/y3coMtzHMIju sEYg== X-Gm-Message-State: AOJu0YzENwgZp/FlLaDnc2aCxN45HsUTC9q+gQx6lPJox3hQUi29pQFO pCNukqM7QFX+TcZUlok8g79HvH70g/iCjBfqGHpJa/94ZydNuwI48YzkAdKO2iAEMiQApIaWAuV LKDs7a6ZjmLm1cwZHhuz7/m9F8VfVBvs= X-Google-Smtp-Source: AGHT+IEmAos5NAthCnc0k3bo4rswlx0rhX0Q2OI7RwDG1aIhWYNM8ShQmjW2gKCDHt/K8S/TfJtc07zEtjJfreJkmgw= X-Received: by 2002:a05:6402:2691:b0:5c8:9ec7:eced with SMTP id 4fb4d7f45d1cf-5c89ec7ef7amr3548270a12.14.1727726848862; Mon, 30 Sep 2024 13:07:28 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Rick Macklem Date: Mon, 30 Sep 2024 13:07:17 -0700 Message-ID: Subject: Re: NFS, intermittent 'RPC struct is bad' errors To: fs@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.78 / 15.00]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_MEDIUM(-0.98)[-0.975]; NEURAL_HAM_SHORT(-0.80)[-0.801]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TAGGED_FROM(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; PREVIOUSLY_DELIVERED(0.00)[fs@freebsd.org]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MLMMJ_DEST(0.00)[fs@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62d:from] X-Rspamd-Queue-Id: 4XHXDL3Xz4z4FxL X-Spamd-Bar: --- On Tue, Jun 18, 2024 at 11:32=E2=80=AFPM Lexi Winter wrot= e: > > hi, > > i have a few systems running NFSv4 on FreeBSD, using Kerberos (MIT > Kerberos KDC), with the server exporting ZFS filesystems. > > recently i've noticed intermittent errors of 'RPC struct is bad' when > writing to the NFS server, which usually resolves itself after retrying. It is possible that commit 5037c6398b (dated Aug 27) fixed this, although it is difficult to say. As you note, since you are using krb5p, a packet trace is pretty usel= ess. If you still see this after an upgrade to a post Aug.27 kernel and could so= mehow run with krb5i instead of krb5p (then you could capture packets, although you'd need to be capturing when the failure occurs, which might result in a large capture), then maybe post again. The only option you are using that doesn't often get used by others is "noncontigwr". (You might try mounts without that option, to see if the problem goes away.) Oh, and if you have set vfs.nfsd.enable_delegations=3D1, try with that set to 0 on the server. That's about all I can think of for you to try, rick > for example: > > % rsync -iavP /scratch/Star.Trek.Prodigy.S01E* . > sending incremental file list > >f++++++++++ Star.Trek.Prodigy.S01E01E02.1080p.WEBRip.x265-KONTRAST.mkv > 32,768 0% 0.00kB/s 0:00:00 rsync: [receiver] write fail= ed on "/data/public/TV/Star Trek Prodigy/Season 01/Star.Trek.Prodigy.S01E01= E02.1080p.WEBRip.x265-KONTRAST.mkv": RPC struct is bad (72) > rsync error: error in file IO (code 11) at receiver.c(380) [receiver=3D3.= 3.0] > > rsync: [sender] write error: Broken pipe (32) > % rsync -iavP /scratch/Star.Trek.Prodigy.S01E* . > sending incremental file list > >f.st....... Star.Trek.Prodigy.S01E01E02.1080p.WEBRip.x265-KONTRAST.mkv > 912,704,431 100% 96.51MB/s 0:00:09 (xfr#1, to-chk=3D18/19) > >f++++++++++ Star.Trek.Prodigy.S01E03.1080p.WEBRip.x265-KONTRAST.mkv > 477,408,567 100% 100.06MB/s 0:00:04 (xfr#2, to-chk=3D17/19) > [...] > > the client is running FreeBSD 15.0-CURRENT from around May 24, and the > server is running a slightly older 15.0-CURRENT from around May 23. > > /etc/exports on the server is pretty standard: > > /data/public -sec=3Dkrb5:krb5i:krb5p -network 2001:8= b0:aab5::/48 > /data/public/Books -sec=3Dkrb5:krb5i:krb5p -network 2001:8= b0:aab5::/48 > /data/public/CalibreLibrary -sec=3Dkrb5:krb5i:krb5p -network 2001:8= b0:aab5::/48 > /data/public/Comics -sec=3Dkrb5:krb5i:krb5p -network 2001:8= b0:aab5::/48 > /data/public/Films -sec=3Dkrb5:krb5i:krb5p -network 2001:8= b0:aab5::/48 > /data/public/Miscellaneous -sec=3Dkrb5:krb5i:krb5p -network 2001:8= b0:aab5::/48 > V4: /data -sec=3Dsys:krb5:krb5i:krb5p -networ= k 2001:8b0:aab5::/48 > > client mount options: > > hemlock.eden.le-fay.org:/public /data/public nfs rw,nfsv4,minorver= sion=3D2,sec=3Dkrb5p,gssname=3Dhost,bgnow,proto=3Dtcp6,nconnect=3D4,rsize= =3D1048576,wsize=3D1048576,noncontigwr 0 0 > > is there anything more i can do investigate this? would a tcpdump > capture of the error be useful (considering all the RPC traffic is > Kerberos-encrypted)?