From nobody Tue Aug 20 15:28:45 2024 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WpCzx753zz5TKRy for ; Tue, 20 Aug 2024 15:29:01 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WpCzx0KzSz42fr for ; Tue, 20 Aug 2024 15:29:01 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=ng2ewLDg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2607:f8b0:4864:20::42e as permitted sender) smtp.mailfrom=jdavidlists@gmail.com Received: by mail-pf1-x42e.google.com with SMTP id d2e1a72fcca58-714186ce2f2so401850b3a.0 for ; Tue, 20 Aug 2024 08:29:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724167739; x=1724772539; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ab2Kxl3mjh2lnSs/DSQoPAjCBl7Mz7JmDUBP/fPHob8=; b=ng2ewLDgguHwE9cuXsmwNUHS3afG9g4x7uOX/HUbKN4bkXOoDiucAsktATmMvnK60X TwX+GVlYknLYcFhkm6fOmv0cETrS0HrcYeZn+Al53WeUTyIyF5Nmmn0kSIUQypvAhqP0 jA7TxalGNJI02Z8foO4y4Q2RJNMxQ2wb6TOCHE21ZiupCfbxs5qhBcUiempjo1YQlbI8 B7a0P6zJX5ypQuNXKu/V95mxDYfXUWzEkp2War+YkVcZEcWapwQvdAJT8kimbziPU42L fjgYHzdFbtFqgUlW/LmlUL8nb0UYo9uw7BBZzNwncZPdxBgTnfcz7HOaGPXlX4amWWw7 KDgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724167739; x=1724772539; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ab2Kxl3mjh2lnSs/DSQoPAjCBl7Mz7JmDUBP/fPHob8=; b=lm69xz+9NkxzQX/uqlhb4XgcZsk07gdxtAEuEv/Fp536orqvIelOQnDhtGFlcsipZx Mth6jKGoqSz0Wwzl5OlhZqmQuBIIGSGxSwb122ui5b+FkirEigJXxiDeRDy5w87wSb87 tLsaUfsSjJ22KeEgcx/9YWBUfP7dtpPXmu9ipksuo6Hr8dDmKCNK9nb0AWM6naAEsHB/ 7IwtUdshV+4R4qOWRBxpXlx8TiS/QKdivLWyi63KN4Tqt4jBTfUd+VoC/X0dSbjCCtI5 iO/KlsqC1ONcYAtlVYhWBKA+mLfUXEE+EKej2AnVirFalvQwBsEOYOQD1FC2ftor7KXE plxQ== X-Gm-Message-State: AOJu0YyG9VPxJ4AaB2d4L3gWtefW9UsbisTpYIFZSZdEkOcJwHvZSIpm HREXvwCf5OTHx6CUfeO7rNRv96uWEiuDywluAC2g0JEBt/rpGUsKm3rLf5pUxq746pLUbZOCL9n pmiUJUK2N9SEHanVN9g6FPeObGkI= X-Google-Smtp-Source: AGHT+IG4C6gZ6P50eJue8Oo9X96uzP6v8pz6klFUjGgDvSbttgHj+20PuVh3IA32UfSQL54IJmZ1h/o0YTwYZa2gK/8= X-Received: by 2002:a05:6a00:148f:b0:70d:34aa:6d51 with SMTP id d2e1a72fcca58-713c4df24e9mr14839168b3a.6.1724167739384; Tue, 20 Aug 2024 08:28:59 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Tue, 20 Aug 2024 11:28:45 -0400 Message-ID: Subject: Re: NFS, intermittent 'RPC struct is bad' errors To: Rick Macklem Cc: fs@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; TAGGED_FROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; PREVIOUSLY_DELIVERED(0.00)[fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[fs@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::42e:from] X-Rspamd-Queue-Id: 4WpCzx0KzSz42fr On Wed, Jun 19, 2024 at 10:05=E2=80=AFAM Rick Macklem wrote: > On Tue, Jun 18, 2024 at 11:32=E2=80=AFPM Lexi Winter wr= ote: > > i have a few systems running NFSv4 on FreeBSD, using Kerberos (MIT > > Kerberos KDC), with the server exporting ZFS filesystems. > > > > recently i've noticed intermittent errors of 'RPC struct is bad' when > > writing to the NFS server, which usually resolves itself after retrying= . > > for example: > [...] > No one else has reported anything like this recently, We are also seeing intermittent "RPC struct is bad" from FreeBSD NFSv4.2 clients accessing ZFS filesystems. There are a few differences between our situation and that reported by Lexi Winter: - NFS servers are Debian 12. - We do not use kerberos, noncontigwr, or delegations. - It does not resolve itself after retrying. In our case, it seems to "infect" directories as viewed from a certain FreeBSD client machine after attempting to modify (usually delete) that directory. once a directory is affected, it can no longer be viewed or removed from that client. Other clients (FreeBSD or otherwise) are not affected. It doesn't seem to be cached or anything. I.e., it doesn't go away if you leave that directory alone for a while (whether a few minutes, few hours, or a full day). Once the directory is removed from another (identical) client, everything is fine. It seems to happen randomly, on the order of once every few billion NFS calls, so whatever it is, it's a very, very edge case. ZFS+NFS seems to have a few of those, and I'm sure cross-platform mounts aren't helping. Thanks!