Re: What's going on with vnets and epairs w/ addresses?
- In reply to: Bjoern A. Zeeb: "Re: What's going on with vnets and epairs w/ addresses?"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sat, 17 Dec 2022 13:36:10 UTC
> On Dec 17, 2022, at 6:55 AM, Bjoern A. Zeeb <bz@FreeBSD.org <mailto:bz@FreeBSD.org>> wrote: > > On Fri, 16 Dec 2022, Zhenlei Huang wrote: > > Hi, > >> I managed to repeat this issue on CURRENT/14 with this small snip: >> >> ------------------------------------------- >> #!/bin/sh >> >> # test jail name >> n="test_ref_leak" >> >> jail -c name=$n path=/ vnet persist >> # The following line trigger jail pr_ref leak >> jexec $n ifconfig lo0 inet 127.0.0.1/8 >> >> jail -R $n >> >> # wait a moment >> sleep 1 >> >> jls -j $n >> >> >> ------------------------------------------- >> >> >> After DDB debugging and tracing , it seems that is triggered by a combine of [1] and [2] >> >> [1] https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 <https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915><https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915 <https://reviews.freebsd.org/rGfec8a8c7cbe4384c7e61d376f3aa5be5ac895915>> >> [2] https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b <https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b><https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b <https://reviews.freebsd.org/rGeb93b99d698674e3b1cc7139fda98e2b175b8c5b>> >> >> >> In [1] the per-VNET uma zone is shared with the global one. >> `pcbinfo->ipi_zone = pcbstor->ips_zone;` >> >> In [2] unref `inp->inp_cred` is deferred called in inpcb_dtor() by uma_zfree_smr() . >> >> Unfortunately inps freed by uma_zfree_smr() are cached and inpcb_dtor() is not called immediately , >> thus leaking `inp->inp_cred` ref and hence `prison->pr_ref`. >> >> And it is also not possible to free up the cache by per-VNET SYSUNINIT tcp_destroy / udp_destroy / rip_destroy. > > Thanks a lot for tracking it down. > > That seems to be a regression then that needs to be fixed before > 14.0-RELEASE will happen as it'll break management utilities of people. > > Could you open a bug report and flag it as such? While I was trying to open a new bug report Bugzilla prompt an existing PR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264981 <https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264981> opened by olivier. I think this issue is same with that one. > > /bz > > >> >> >> Best regards, >> Zhenlei >> >>> On Dec 14, 2022, at 9:56 AM, Zhenlei Huang <zlei@FreeBSD.org <mailto:zlei@FreeBSD.org>> wrote: >>> >>> >>> Hi, >>> >>> I also encounter this problem while testing gif tunnel between jails. >>> >>> My script is similar but with additional gif tunnels. >>> >>> >>> There are reports in mailing list [1], [2], and another one in forum [3] . >>> >>> Seem to be a long standing issue. >>> >>> [1] https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.html <https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.html><https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.html <https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086126.html>> >>> [2] https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html <https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html> <https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html <https://lists.freebsd.org/pipermail/freebsd-jail/2017-March/003357.html>> >>> [3] https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-starting-networking-et-cetera.84200/ <https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-starting-networking-et-cetera.84200/><https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-starting-networking-et-cetera.84200/ <https://forums.freebsd.org/threads/jails-stopping-prolonged-deaths-starting-networking-et-cetera.84200/>> >>> >>> >>> Best regards, >>> Zhenlei >>> >>>> On Dec 14, 2022, at 7:03 AM, Bjoern A. Zeeb <bz@FreeBSD.org <mailto:bz@FreeBSD.org> <mailto:bz@FreeBSD.org <mailto:bz@FreeBSD.org>>> wrote: >>>> >>>> Hi, >>>> >>>> I have used scripts like the below for almost a decade and a half >>>> (obviously doing more than that in the middle). I haven't used them >>>> much lately but given other questions I just wanted to fire up a test. >>>> >>>> I have an end-November kernel doing the below my eapirs do not come back >>>> to be destroyed (immediately). >>>> I have to start polling for the jid to be no longer alive and not in >>>> dying state (hence added the jls/ifconfig -l lines and removed the >>>> error checking from ifconfig destroy). That seems sometimes rather >>>> unreasonably long (to the point I give up). >>>> >>>> If I don't configure the addresses below this isn't a problem. >>>> >>>> Sorry I am confused by too many incarnations of the code; I know I once >>>> had a version with an async shutdown path but I believe that never made >>>> it into mainline, so why are we holding onto the epairs now and not >>>> nuking the addresses and returning them and are clean? >>>> >>>> It's a bit more funny; I added a twiddle loop at the end and nothing >>>> happened. So I stop the script and start it again and suddenly another >>>> jail or two have cleaned up and their epairs are back. Something feels >>>> very very wonky. Play around with this and see ... and let me know if >>>> you can reproduce this... I quite wonder why some test cases haven't >>>> gone crazy ... >>>> >>>> /bz >>>> >>>> ------------------------------------------------------------------------ >>>> #!/bin/sh >>>> >>>> set -e >>>> set -x >>>> >>>> js=`jail -i -c -n jl host.hostname=left.example.net <http://left.example.net/> <http://left.example.net/ <http://left.example.net/>> vnet persist` >>>> jb=`jail -i -c -n jr host.hostname=right.example.net <http://right.example.net/> <http://right.example.net/ <http://right.example.net/>> vnet persist` >>>> >>>> # Create an epair connecting the two machines (vnet jails). >>>> ep=`ifconfig epair create | sed -e 's/a$//'` >>>> >>>> # Add one end to each vnet jail. >>>> ifconfig ${ep}a vnet ${js} >>>> ifconfig ${ep}b vnet ${jb} >>>> >>>> # Add an IP address on the epairs in each vnet jail. >>>> # XXX Leave these out and the cleanup seems to work fine. >>>> jexec ${js} ifconfig ${ep}a inet 192.0.2.1/24 >>>> jexec ${jb} ifconfig ${ep}b inet 192.0.2.2/24 >>>> >>>> # Clean up. >>>> jail -r ${jb} >>>> jail -r ${js} >>>> >>>> # You want to be able to remove this line ... >>>> set +e >>>> >>>> # No epairs to destroy with addresses configured; fine otherwise. >>>> ifconfig ${ep}a destroy >>>> # echo $? >>>> >>>> # Add this is here only as things are funny ... >>>> # jls -av jid dying >>>> # ifconfig -l >>>> >>>> # end >>>> ------------------------------------------------------------------------ >>>> >>>> -- >>>> Bjoern A. Zeeb r15:7 >>>> >>> >> >> > > -- > Bjoern A. Zeeb r15:7