From nobody Wed Jan 29 05:13:57 2025 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YjVgX2xxcz5m5Ys for ; Wed, 29 Jan 2025 05:14:00 +0000 (UTC) (envelope-from glebius@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YjVgX2CfVz3JM6; Wed, 29 Jan 2025 05:14:00 +0000 (UTC) (envelope-from glebius@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1738127640; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+g6scScMN2J3fJVNbLEHTJHSzvkykWAHaudlIQNN3Vc=; b=TTZwVJayLh6wKJRxBMKorRCwr2wOQMSsaeQDZolBA7YaD5xRlGAW78rprc0Qr981S2Akdp kX7fc5pDodkagj4HzGgbvly/Bel2nczYzAAqDtCSOTE4FhR5mx2TJPl2u5eg4tsHO/Ax+t DojaQU7F1lFyC762BGGV/dMyo7MdNI5+m7LYjV8BY3wCuccFLv8ENb7rGZCjG802+o+iKr Jwkli0OBYFXA8oT7yuQ/8aXFVMuVgZzBaOcAVhZLBICbYL+5mYh4x692FGXNc7u51/lpUC YBtC0SUOWTqIehOSoaB7xv+qUMyjjYF0mXkPyDKSUj/QdFoVkWacQmkT5DzHNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1738127640; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+g6scScMN2J3fJVNbLEHTJHSzvkykWAHaudlIQNN3Vc=; b=Hfw2xrcYt7xcYrMPgSoftutUdFLWL8dW7CzLZOyyfX2xmJRc1eko+FGiBRyTsM2veVujO9 KMjw4QmWA3jfvHeCxNCXEmzLFo+akSfx73MmBRycNcHM1Twqt8XmrP/uASbXH+xJB2qtbn 4yEGaksoxaBPO8RjbNdlc6q/KgYZjFO/8sGqt3o5v2hxliqAyT8Kr2lXbktVEyvFbsZRsR zyA+E0KWzyaba0J9lxDd02vViWetDHnztm2QZ9OnhAEAXMiJApuEmO7mnO83LEldsarZfV TXNbHbOoa3FPWcBxjT4T/4CduAVinw37YXCJtdRkV2w7ANsKSI1bYUtoBbFIXw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1738127640; a=rsa-sha256; cv=none; b=S79un1KgPVBiaf8snSxFHxrk8lJJok6QIpmOgC+i/aQUFbBKiRM/Ejz6Ij5wjIZh3WuW3s mzITUXrnyuNeZiyo7UsjIwM42pf0nvKJ78iU8MWkam1WACW0l06u5sBglIPRBwhev0Tyyb RfEUiYyjSd9wPmI/YUSshNO56Zmt1My8k4FqporVNxSn3rXM5xVE68x0/HsuOlXMuiUO5w ZDRXxQHYfa2uP3Ay/mU8T9TaAr/UYz3g90LxaC3b6/ZNNQ3OhDK6jstXqfaIwxV3aOoSCy 5yUExGpvxaq4AeQojQZ04mZ9U+iCb2k30pcc3I8C6iIDtAK94MSXhpZ6mroYyw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from cell.glebi.us (glebi.us [162.251.186.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: glebius) by smtp.freebsd.org (Postfix) with ESMTPSA id 4YjVgW5PNqz5jW; Wed, 29 Jan 2025 05:13:59 +0000 (UTC) (envelope-from glebius@freebsd.org) Date: Tue, 28 Jan 2025 21:13:57 -0800 From: Gleb Smirnoff To: Rick Macklem Cc: current@freebsd.org, rmacklem@freebsd.org Subject: Re: HEADS UP: NFS changes coming into CURRENT early February Message-ID: References: List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="8/4KebHVP00ayBWE" Content-Disposition: inline In-Reply-To: --8/4KebHVP00ayBWE Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, Jan 27, 2025 at 06:10:42PM -0800, Rick Macklem wrote: R> I think I've found a memory leak, but it shouldn't be a show stopper. R> R> What I did on the NFS client side is: R> # vmstat -m | fgrep -i rpc R> # mount -t nfs -o nfsv4,tls nfsv4-server:/ /mnt R> # ls --lR /mnt R> --> Then I network partitioned it from the server a few times, until R> the TCP connection closed. R> (My client is in bhyve and the server on the system the bhyve R> instance is running in. I just "ifconfig bridge0 down", waited for R> the TCP connection to close "netstat --a" then "ifconfig bridge0 up". R> Once done, I R> # umount /mnt R> # vmstat -m | fgrep -i rpc R> and say a somewhat larger allocation count R> R> The allocation count only goes up if I do the network partitioning R> and only on the NFS client side. R> R> Since the leak is slow and only happens when the TCP connection R> breaks, I do not think it is a show stopper and one of us can track it down R> someday. I reproduced the recipe and find two problems, but don't have a final solution for either. First, when we create backchannel in sys/fs/nfs/nfs_commonkrpc.c newnfs_connect(): if (nfs_numnfscbd > 0) { nfs_numnfscbd++; NFSD_UNLOCK(); xprt = svc_vc_create_backchannel( nfscbd_pool); CLNT_CONTROL(client, CLSET_BACKCHANNEL, xprt); NFSD_LOCK(); nfs_numnfscbd--; if (nfs_numnfscbd == 0) wakeup(&nfs_numnfscbd); } The svc_vc_create_backchannel() creates xprt with refcount=1. Then we link it to our client, CLNT_CONTROL() makes refcount=2. Then this functions forgets the pointer (but it owns refcount). Whenever the client is destroyed, it will do SVC_RELEASE() on the backchannel, refcount goes 2 to 1 and it is leaked. I made a patch against that and it does reduce amount of M_RPC leaked after your recipe, but once I got panic where the backchannel xprt is actually used after free. So looks like there is a case that my patch doesn't cover. What drives me crazy, can't reproduce it for the second time and can't get any clue from the single core. Patch attached. Second, with the patch the M_RPC leak count for me is 2. And I found that these two items are basically is a clnt_vc that belongs to a closed connection: fffff80029614a80 tcp4 0 0 10.6.6.9.772 10.6.6.9.2049 CLOSED There is no connection peer connection, as the server received a timeout trying to send. But rpc.tlsclntd doesn't try to send anything on the socket, it just keeps it select(2) fd set and doesn't garbage collect. So it is a bigger resource leak than just two pieces of M_RPC. I don't think this is related to my changes. -- Gleb Smirnoff --8/4KebHVP00ayBWE Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="backchannel-leak.diff" commit 1ca54035f1ab46884675fdbba7364624e1217c9e Author: Gleb Smirnoff Date: Tue Jan 28 20:13:19 2025 -0800 try diff --git a/sys/fs/nfs/nfs_commonkrpc.c b/sys/fs/nfs/nfs_commonkrpc.c index e5c658ce76d2..9f3583a09037 100644 --- a/sys/fs/nfs/nfs_commonkrpc.c +++ b/sys/fs/nfs/nfs_commonkrpc.c @@ -262,7 +262,6 @@ newnfs_connect(struct nfsmount *nmp, struct nfssockreq *nrp, struct socket *so; int one = 1, retries, error = 0; struct thread *td = curthread; - SVCXPRT *xprt; struct timeval timo; uint64_t tval; @@ -430,12 +429,15 @@ newnfs_connect(struct nfsmount *nmp, struct nfssockreq *nrp, */ NFSD_LOCK(); if (nfs_numnfscbd > 0) { + SVCXPRT *xprt; + nfs_numnfscbd++; NFSD_UNLOCK(); xprt = svc_vc_create_backchannel( nfscbd_pool); CLNT_CONTROL(client, CLSET_BACKCHANNEL, xprt); + SVC_RELEASE(xprt); NFSD_LOCK(); nfs_numnfscbd--; if (nfs_numnfscbd == 0) --8/4KebHVP00ayBWE--