From nobody Sat Dec 17 17:13:14 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NZCTP4Scrzt682 for ; Sat, 17 Dec 2022 17:22:33 +0000 (UTC) (envelope-from glebius@freebsd.org) Received: from glebi.us (glebi.us [162.251.186.162]) by mx1.freebsd.org (Postfix) with ESMTP id 4NZCTN5HKRz3y1N; Sat, 17 Dec 2022 17:22:32 +0000 (UTC) (envelope-from glebius@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 162.251.186.162 is neither permitted nor denied by domain of glebius@freebsd.org) smtp.mailfrom=glebius@freebsd.org; dmarc=none Received: by glebi.us (Postfix, from userid 1000) id C7C2843545; Sat, 17 Dec 2022 09:13:14 -0800 (PST) Date: Sat, 17 Dec 2022 09:13:14 -0800 From: Gleb Smirnoff To: Zhenlei Huang Cc: "Bjoern A. Zeeb" , "freebsd-jail@freebsd.org" Subject: Re: What's going on with vnets and epairs w/ addresses? Message-ID: References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> X-Spamd-Result: default: False [1.50 / 15.00]; VIOLATED_DIRECT_SPF(3.50)[]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.995]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; MLMMJ_DEST(0.00)[freebsd-jail@freebsd.org]; FREEFALL_USER(0.00)[glebius]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; R_SPF_SOFTFAIL(0.00)[~all:c]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:27348, ipnet:162.251.186.0/24, country:US]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; GREYLIST(0.00)[pass,body] X-Rspamd-Queue-Id: 4NZCTN5HKRz3y1N X-Spamd-Bar: + X-ThisMailContainsUnwantedMimeParts: N Zhenlei, On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote: Z> I managed to repeat this issue on CURRENT/14 with this small snip: Z> Z> ------------------------------------------- Z> #!/bin/sh Z> Z> # test jail name Z> n="test_ref_leak" Z> Z> jail -c name=$n path=/ vnet persist Z> # The following line trigger jail pr_ref leak Z> jexec $n ifconfig lo0 inet 127.0.0.1/8 Z> Z> jail -R $n Z> Z> # wait a moment Z> sleep 1 Z> Z> jls -j $n Z> Z> After DDB debugging and tracing , it seems that is triggered by a combine of [1] and [2] Z> Z> [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 Z> [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b Z> Z> Z> In [1] the per-VNET uma zone is shared with the global one. Z> `pcbinfo->ipi_zone = pcbstor->ips_zone;` Z> Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by uma_zfree_smr() . Z> Z> Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called immediately , Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. Z> Z> And it is also not possible to free up the cache by per-VNET SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. This is known issue and I'd prefer not to call it a problem. The "leak" of a jail happens only if machine is idle wrt the networking activity. Getting back to the problem that started this thread - the epair(4)s not immediately popping back to prison0. IMHO, the problem again lies in the design of if_vmove and epair(4) in particular. The if_vmove shall not exist, instead we should do a full if_attach() and if_detach(). The state of an ifnet when it undergoes if_vmove doesn't carry any useful information. With Alexander melifaro@ we discussed better options for creating or attaching interfaces to jails that if_vmove. Until they are ready the most easy workaround to deal with annoying epair(4) come back problem is to remove it manually before destroying a jail, like I did in 80fc25025ff. -- Gleb Smirnoff