From nobody Fri May 27 00:04:23 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B1F6E1B4D545 for ; Fri, 27 May 2022 00:04:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4L8Q5g1L43z4VV1 for ; Fri, 27 May 2022 00:04:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0D1BC18007 for ; Fri, 27 May 2022 00:04:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 24R04MDl082206 for ; Fri, 27 May 2022 00:04:22 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 24R04MLn082205 for bugs@FreeBSD.org; Fri, 27 May 2022 00:04:22 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 260011] Unresponsive NFS mount on AWS EFS Date: Fri, 27 May 2022 00:04:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: rmacklem@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1653609863; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W6N+fG7xRNzXXEGJUqPG5XAUNopZvAy7F/4Sns7LINA=; b=d4WY+0IJ8vvNHBtI4vN34dqyiPkYKasdnENn/yG4tUKgrZI8pl0n828Ca+ED9+0LvdWlPC a7b1aPaqcu8jlN5CIqfnYDVfo/PdsTRUaOBMUf1pOyHY3sGeCOtZBSEUOtgvJ+UOgstTQA dwOQxjYpOaqRGBHftiXzE26EmY5VH60h2pKA0sT5tgXBILUUTB2Hx0xSUIf8DKgTjp28rl a/QmiShCZM3uTaCWIeEKr5lRMPmrN9J2vBonynR129mR0nfmqct25EPU4XyirUFHleBqb0 l8jpUsFN+5dYb8Fm0kGul5R2kJMCeT/HCZJXt4HvGq+iTP1LEwzw4GSGFqvbeg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1653609863; a=rsa-sha256; cv=none; b=GZOBrmwsxi4LGdkQ60L+RNYrnrUWoNTkCs56ymWJkCLg7AnJz9XrJhALuEFEts4i3ikFvs ooKiuoHFaHTy7lPlKzQ/K3RKrRk98l4s8LQkMc7O6vvT1MPPS3B4qcrORVfhV4orI02j+I l0rP2ZYX/Ae018R8+pejrYeIQbjB+jumg23meOzxfTLVX1JmTvnkynEHFCZJPXP/ld6NcP /YpfFv8fqBlFgoOiaZ4qnuT4mVV2YuCddO3bmllYIOoLcEFCobWldWnGElDFcBWIJcq9cp 86Yi3KvTnNVJDzzhmhrwdBtgNj8olpCEF5YPRUqmxygRTVxSSJ3KpbYyizp3dg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D260011 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |rmacklem@FreeBSD.org --- Comment #17 from Rick Macklem --- Created attachment 234241 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D234241&action= =3Dedit handle bogus slot# replies for the Sequence op cpercival@ emailed with some diagnostics (that I did not realize were not in 13.0) which indicates that the Amazon EFS server is pretty badly broken. It sometimes (I don't know how frequently) returns the wrong slotid for a session. (It is required by the RFC to be the same as the request.) Once this happens, there is no way to know which slot# the server actually used. This patch (which is rather large and, unfortunately, will not apply to 13.0, but should apply to stable/13 and 13.1, I think?) marks both of the slots (the one in the request and the one in the reply) bad, so they will no longer be used. When all slots get marked "bad", it does a DestroySession operation, which should make subsequent uses of the session fail with NFSERR_BADSESSION. An NFSERR_BADSESSION reply should, in turn, start a recovery cycle which should create a new session that can be used. This patch has been tested against a hacked FreeBSD nfsd that replies with a bogus slot# once every 100 RPCs and seems to work ok. I have no idea if the Amazon EFS server will behave the same way, but I am hoping cpercival@ will be able to test it. I believe this serious bug in the Amazon EFS server would explain your hangs. --=20 You are receiving this mail because: You are the assignee for the bug.=