From nobody Tue Dec 20 00:10:05 2022 X-Original-To: fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NbcQk0k9Mz1GF0d for ; Tue, 20 Dec 2022 00:10:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NbcQj6Gtdz3jZ2 for ; Tue, 20 Dec 2022 00:10:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671495005; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KNEs5fDjnmvbCs8q7w5r09pU4LhgTr89H0SibkWTXXc=; b=jVl4H0FEuSinSQp4liPi661zCDfCJEHbSklQJS3q7aFn5xvaQjS7PrgJXIdJj6JTaSE2EY xE9DlTYHBLj6/FORkiJiN8vyPdd7Ryc7uBdDa/eN2wQ7hS4OUeUlglTeh5q27F0HI6ZgBP FmuuZvORJDJBzgvbxCwfwokciX0KG5c7D/mJoeBfTz/Wfwr4N4ACQyUF3dsaMzJp0Ct7Ic Ta7YvLGl8/ZCYCNLer3TimpJ90DbwxjsGWXVcOxPsJUCes21VpSeisnpWGb+tmi9r+bKLL o0rv/ZHs5Sq5aXgfy7gVuswLfHmNRLL+VrJPtJs9mRTcWFCihAQ8hqWeb115HA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671495005; a=rsa-sha256; cv=none; b=nHs9fimJRVSAOPEnM91/PU5kN47+OQniM7NtkfoPZgC4FiJ2EfAC5mdVtV6DTvRTvMzNR9 w6KzQoIsibLeVkgxr2klD9CLShRQZz4sBEkx3ux4Sf+m0Fm/gq/uMOG1ff6SkGvXYXW90x EAdNz+mZblteFdtVj5lKwPwBvyYYZ+8lWiTgpzc8bgGvxjZpyU1bjMRPFsxv0XSq0GDpCU nyj+qdQu0KCPCFHgiYu/sPfpoUZSXPFmwqQTd0HT/rwtKr142NXYdNaiWgiZ3kZzgZjnW8 tZnU/doY5HOcgK32goToMDhCG+5QDICGYNcFCVIVz0Qq8Jb973XpGfYE3fUNbw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4NbcQj5FWLzFWg for ; Tue, 20 Dec 2022 00:10:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 2BK0A5gi047776 for ; Tue, 20 Dec 2022 00:10:05 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 2BK0A5th047775 for fs@FreeBSD.org; Tue, 20 Dec 2022 00:10:05 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 266302] [zfs][iscsi] Periodic drops by ctl with "failed to allocate soft PDU" since 13.1 Date: Tue, 20 Dec 2022 00:10:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: eborisch+FreeBSD@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D266302 --- Comment #2 from eborisch+FreeBSD@gmail.com --- What's interesting is that, in the latest one I have data for, the free mem= ory never fell below 10GB. (Granted, that is only being sampled every 30s in my logging, so it could have much lower briefly.) I see a spike (monitoring vmstat -z) of a large number of frees just after = the error was detected -- abd_chunk, zio_buf_comb_131072, and dnode_t being the largest, freeing a combined 2.2GB, and a spike in failures for 'vm pgcache' immediately before it, and continuing immediately after the failure. I've p= ut a copy of the logging (1m) of vmstat -z and some analysis here: https://github.com/eborisch/pdu-error/blob/main/pdu_analysis.ipynb The log stops _after_ the error was detected, so it has considerable data leading up to, and then one data point after an instance of this error, if you're interested in looking at it. In the interim, I can further restrict vfs.zfs.arc_max; I already have it s= et to 75% of the 64GB system. --=20 You are receiving this mail because: You are the assignee for the bug.=