From nobody Wed Mar 23 17:20:28 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 007FA1A204ED for ; Wed, 23 Mar 2022 17:20:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KNw985Cthz3KFW for ; Wed, 23 Mar 2022 17:20:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8FF0A27B00 for ; Wed, 23 Mar 2022 17:20:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 22NHKSor062628 for ; Wed, 23 Mar 2022 17:20:28 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 22NHKSvl062627 for bugs@FreeBSD.org; Wed, 23 Mar 2022 17:20:28 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 262743] Memory leak in strongswan's charon daemon when communicating over vici socket. Date: Wed, 23 Mar 2022 17:20:28 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mskalski13@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.mimetype attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1648056028; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=0iGNZpCTRnI3hQkBSUCyXy1oa/o2GXfitN6BR3+lDSY=; b=mNn8fOgtUHYwr3O0hOH/jLjBoxWBZ6A5YM8ijJwby4I1YMvceuornQ8gFH59zmFIfjrWzg chww+kQbMD1XJSLfVukDvNvkxveEMd4Y4Vr3Og5G1DtCWAnAvCz4mWpOHJ2+qJcm1NSKoy lIe1kIkqQvVXrDpJ7TWKgHKX7InjJ+U5GZ+1G3U+bnV5W1dAbdtTRLBEUDl2Qng7hA34Ys JZEhqOiLtYR9FWGI1MZ3GPHvZckvcxnmfYdPDEuVsgyxb42xCcNUQCaCVKgZPH5AdliRqc DamE4KrTiNXbNQW9arpKkhcGXMgTmUWCM3mqy+TmeTJHBILTFgYHfPkkdkYnGg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1648056028; a=rsa-sha256; cv=none; b=IJtNTGgDoFcWmUlkeAIEXPtSwa1v1dtzrzTsM3K5znfphlOXpCKfVdH6qpmqLJKAOkw6NX Tdtk4vf+bh1pJ2AnOknFURlWnHFVlezrw9igx+PlGKsECK9rkt8varu5MCxlzfYvfYOFVr XovXrld0N0SSIYccb4zcgKH3r8gTJrf8hvsprkBuIJtl66JAPU3KFSdTU3rqzsvMU5PRMp Xa9pKwPZUEOZgD7/lP8tijmyAY1bzqSR+9tzAPLiK3e39mwqWEcJRUxvbFQ4D4Nj4Sy9k7 bgx0dvYjbcB4ZSkq6vAn/Zr2hQfhO66IZgXjLS+TXCrEaSgczBLRBdvErHPloQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D262743 Bug ID: 262743 Summary: Memory leak in strongswan's charon daemon when communicating over vici socket. Product: Base System Version: 13.1-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: misc Assignee: bugs@FreeBSD.org Reporter: mskalski13@gmail.com Attachment #232660 text/plain mime type: Created attachment 232660 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D232660&action= =3Dedit Dump of statistics of jemalloc library at charon daemon exit On FreeBSD system (amd64, arm64) when communicating over vici socket memory leaks in terms of constantly increasing Virtual and Resident (VMS and RSS) memory of process occur, until all system memory is exhausted, when process charon is killed by kernel with message kernel: pid 903 (charon), jid 0, ui= d 0, was killed: failed to reclaim memory. Any tool for memory leak detection tools (valgrind, ktrace) does not detect= any memory leaks, increasing RSS is the only symptom. The same behaviour was observed on FreeBSD 12.1, 12.2 and 9.3 (the latter is the last release before incorporating jemalloc library to FreeBSD's libc). When running charon daemon on Linux (tested on Ubuntu 20.04 and Debian 10 bookworm/sid) problem does not occur. I think this behaviour is because frequent memory allocation and deallocati= on (malloc/free functions), which is used in vici plugin. And I observed that = this increase can also be caused by SA renegotiations, but that is harder to isolate. And there is no special malloc configuration for charon daemon and on the o= ther hand other applications on FreeBSD box are not affected, which are i.e. some running python daemons (which I believe do massive allocations and use mult= iple threads). I wonder what is specific in a way strongswan allocates memory th= at RSS process memory is increasing so much? To reproduce: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 1. Download any VM image with FreeBSD 12.0+ (was tested also on latest amd64 13.1-BETA2 to confirm) Configure virtual machine; for strongswan compilation give more memory, but= for test 256 MB is enough. 2. Run VM and disable swap (to speed-up failure) # swapoff /dev/gpt/swapfs 3. install required packages for strongswan compilation: # pkg install git autoconf gperf autoconf-archive libtool m4 automake flex bison pkgconf gettext 4. get strongswan: git clone https://github.com/strongswan/strongswan 5. Compile strongswan: cd strongswan ./configure --disable-kernel-netlink --enable-kernel-pfroute --enable-kernel-pfkey --disable-gmp --enable-openssl --enable-mediation --disable-scripts --with-group=3Dwheel --enable-gcm --enable-ccm --enable-p= kcs11 make -j4 make install 6. start strongswan: ipsec start 7. run in loop any command which communicates on vici interface, swanctl --stats is enough to reproduce error: sh -c 'while swanctl --stats >/dev/null; do true; done' 8. Observe increase of VSS and RSS (Virtual and resident) memory of charon process, using e.g. top 9. After few hours charon should be killed by kernel due to not enough memory/swap space. Additional info =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Problem occurred when monitored via vici socket state of charon daemon (tun= nel definitions, SAs, etc), but it was also reproduced using simple swanctl --s= tats command repeated in loop. No change in this beaviour is observed when using different configure's --with-printf-hooks=3D -- according to issue in pfsense: https://redmine.pfsense.org/issues/5149 this could be the reason, but tests with --with-printf-hooks=3Dbuiltin, --with-printf-hooks=3Dglibc and --with-printf-hooks=3Dvstr did not fix the error. I did some tests using various settings of `jemalloc`, attaching results, b= ut I don't know how to interpret the results. It was gathered using following command: sh -c "MALLOC_CONF=3D'stats_print:true,narenas:1' /usr/local/libexec/ipsec/= charon 2>/var/log/charon-memdump-0.log" --=20 You are receiving this mail because: You are the assignee for the bug.=