From nobody Sat Dec 17 19:23:15 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NZG8m3v4Nz1Cgbs for ; Sat, 17 Dec 2022 19:23:20 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NZG8m3LHZz4Flh; Sat, 17 Dec 2022 19:23:20 +0000 (UTC) (envelope-from bz@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671305000; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=I76dFbrStqbMe81XkNxRp1pLwKcqskjKj/dACTxwY/0=; b=QMQKaLEOG0Qzp3cO1cIrS83MMfrp7Tc9S6cj6gAs4CtBn3IQ1KvNkqS8hkKJSBKdTywKOv 5qN+a+khqO0s0kACXeJPNVx8TNxFFfyEMEXStQhLZWESiwWkX+3+2g7nB6BXadmkPYHkE5 QVbOMteUmf/umgvO5DCqjNls/or4ceDaof4BCoMlllYdGX3o3dhiuptS91DhLoRjJ3Gcnc E954Y/m3COYXckyuc2Q7HCzZwCEOcSO6pOOYAleml+pQU9c0R6gyb7kjkE3mj86ADDToJw RDKwV+UpIskURXJXikb7K8LuvqGu3VIS/y3F9Tr8tKZTbnjwj5kKr2+RJmCdNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1671305000; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=I76dFbrStqbMe81XkNxRp1pLwKcqskjKj/dACTxwY/0=; b=UJTGHS34nOszTrdWyRB6tncjMhARBc1DIeo903AQHPIUY81gQA/CuPfoPGJAXMQqygBSQ3 MJpxOuizkGbf9/s4CTW5kyNVd8FVOiTORAyXcDL6EGyc9K5R/K8a8QeU+EjnVb8ska6eMz UzwPZa7JIlA0F6csOhcJGVHsqDYMMiWqq6AII/uZ6ccJjM41h/x6qNQ02kJtzX48O3elv0 KhlYaShTU4TjqoACLkxy0yZSnb5fu9uIBE0NtlawJHRi5pI338hg8CCt1sAczzfd9dKQ+K YHFKuc2SQaLz6k3Ylv/h4A0HoSDCxV8xflapKIv6UUrAZJ6IztoKQPwPalm9Ww== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1671305000; a=rsa-sha256; cv=none; b=h1Qp+9Nglk3NNWcOq2SxMsmxDlP2z4HAUIUoxvDI1mhgrCICMfvs8BNHKe0lPPZ6jxBB9B TX2funXij+F/CtcJeTwZFDtrYWX6R7qSBcSyzGYQJ1GSG+wZoW1WlyAMXsYaxnRf24T95H 0yT+M0HR6Mzyoui5hwsETVMWRzokWVM7h8fHwnwxCm13NhVqrGI7LsMAvOfm51HlJSg1Os MKrNWmzfHxud/Pw8KaOffW9exrhEnc7YLFqNy8l9WC7xdAO/SdhOkHFl5N82SvcewEcnff VbeL+SXWMlZlrfeViHspiTLM7WBoBl7YMgy2QZBtOWLl+cr/rU5WLZXWM5nFmw== Received: from mx1.sbone.de (cross.sbone.de [195.201.62.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) (Authenticated sender: bz/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4NZG8m11jRz18JL; Sat, 17 Dec 2022 19:23:20 +0000 (UTC) (envelope-from bz@freebsd.org) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 3D9478D4A129; Sat, 17 Dec 2022 19:23:18 +0000 (UTC) Received: from content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 9B4F25C3A833; Sat, 17 Dec 2022 19:23:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:4902:0:7404:2:1025]) by content-filter.t4-02.sbone.de (content-filter.t4-02.sbone.de [IPv6:fde9:577b:c1a9:4902:0:7404:2:2742]) (amavisd-new, port 10024) with ESMTP id eHZVb-2OWWyW; Sat, 17 Dec 2022 19:23:16 +0000 (UTC) Received: from strong-iwl0.sbone.de (strong-iwl0.sbone.de [IPv6:fde9:577b:c1a9:4902:b66b:fcff:fef3:e3d2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id D44C55C3A830; Sat, 17 Dec 2022 19:23:15 +0000 (UTC) Date: Sat, 17 Dec 2022 19:23:15 +0000 (UTC) From: "Bjoern A. Zeeb" To: Gleb Smirnoff cc: Zhenlei Huang , "freebsd-jail@freebsd.org" Subject: Re: What's going on with vnets and epairs w/ addresses? In-Reply-To: Message-ID: <9p9919q1-n639-p581-6q1o-so48o5ns6717@serrofq.bet> References: <5r22os7n-ro15-27q-r356-rps331o06so5@mnoonqbm.arg> <150A60D6-6757-46DD-988F-05A9FFA36821@FreeBSD.org> X-OpenPGP-Key-Id: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-ThisMailContainsUnwantedMimeParts: N On Sat, 17 Dec 2022, Gleb Smirnoff wrote: > Zhenlei, > > On Fri, Dec 16, 2022 at 06:30:57PM +0800, Zhenlei Huang wrote: > Z> I managed to repeat this issue on CURRENT/14 with this small snip: > Z> > Z> ------------------------------------------- > Z> #!/bin/sh > Z> > Z> # test jail name > Z> n="test_ref_leak" > Z> > Z> jail -c name=$n path=/ vnet persist > Z> # The following line trigger jail pr_ref leak > Z> jexec $n ifconfig lo0 inet 127.0.0.1/8 > Z> > Z> jail -R $n > Z> > Z> # wait a moment > Z> sleep 1 > Z> > Z> jls -j $n > Z> > Z> After DDB debugging and tracing , it seems that is triggered by a combine of [1] and [2] > Z> > Z> [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 > Z> [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b > Z> > Z> > Z> In [1] the per-VNET uma zone is shared with the global one. > Z> `pcbinfo->ipi_zone = pcbstor->ips_zone;` > Z> > Z> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by uma_zfree_smr() . > Z> > Z> Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called immediately , > Z> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. > Z> > Z> And it is also not possible to free up the cache by per-VNET SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. > > This is known issue and I'd prefer not to call it a problem. The "leak" of a jail > happens only if machine is idle wrt the networking activity. > > Getting back to the problem that started this thread - the epair(4)s not immediately > popping back to prison0. IMHO, the problem again lies in the design of if_vmove and > epair(4) in particular. The if_vmove shall not exist, instead we should do a full > if_attach() and if_detach(). The state of an ifnet when it undergoes if_vmove doesn't > carry any useful information. With Alexander melifaro@ we discussed better options > for creating or attaching interfaces to jails that if_vmove. Until they are ready > the most easy workaround to deal with annoying epair(4) come back problem is to > remove it manually before destroying a jail, like I did in 80fc25025ff. Ok, move an em0 or cxl0 into the jail; the problem will be the same I bet and you need the physical interface to not disappear as then you cannot re-create a new jail with it. /bz -- Bjoern A. Zeeb r15:7