From nobody Tue Nov 14 13:44:22 2023 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SV6wk0Btmz5127R for ; Tue, 14 Nov 2023 13:44:38 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SV6wj20RGz3N6F for ; Tue, 14 Nov 2023 13:44:37 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net; dmarc=none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 3AEDiUbj040754 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL) for ; Tue, 14 Nov 2023 08:44:30 -0500 (EST) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:a42a:b17e:7016:bb1d] ([IPv6:2607:f3e0:0:4:a42a:b17e:7016:bb1d]) by pyroxene2a.sentex.ca (8.17.1/8.15.2) with ESMTPS id 3AEDiKMb074923 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Tue, 14 Nov 2023 08:44:29 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <02880a36-eec2-4fd4-8693-1f3de382e730@sentex.net> Date: Tue, 14 Nov 2023 08:44:22 -0500 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: FreeBSD-STABLE Mailing List From: mike tancsa Subject: RELENG_14 [process] was killed: failed to reclaim memory Autocrypt: addr=mike@sentex.net; keydata= xsBNBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAHNHW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+wsCOBBMBCAA4FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAl+pQfkCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQeVOEFl5W rMiN6ggAk3H5vk8QnbvGbb4sinxZt/wDetgk0AOR9NRmtTnPaW+sIJEfGBOz47Xih+f7uWJS j+uvc9Ewn2Z7n8z3ZHJlLAByLVLtcNXGoRIGJ27tevfOaNqgJHBPbFOcXCBBFTx4MYMM4iAZ cDT5vsBTSaM36JZFtHZBKkuFEItbA/N8ZQSHKdTYMIA7A3OCLGbJBqloQ8SlW4MkTzKX4u7R yefAYQ0h20x9IqC5Ju8IsYRFacVZconT16KS81IBceO42vXTN0VexbVF2rZIx3v/NT75r6Vw 0FlXVB1lXOHKydRA2NeleS4NEG2vWqy/9Boj0itMfNDlOhkrA/0DcCurMpnpbM7ATQRcsMzk AQgA1Dpo/xWS66MaOJLwA28sKNMwkEk1Yjs+okOXDOu1F+0qvgE8sVmrOOPvvWr4axtKRSG1 t2QUiZ/ZkW/x/+t0nrM39EANV1VncuQZ1ceIiwTJFqGZQ8kb0+BNkwuNVFHRgXm1qzAJweEt RdsCMohB+H7BL5LGCVG5JaU0lqFU9pFP40HxEbyzxjsZgSE8LwkI6wcu0BLv6K6cLm0EiHPO l5G8kgRi38PS7/6s3R8QDsEtbGsYy6O82k3zSLIjuDBwA9GRaeigGppTxzAHVjf5o9KKu4O7 gC2KKVHPegbXS+GK7DU0fjzX57H5bZ6komE5eY4p3oWT/CwVPSGfPs8jOwARAQABwsB2BBgB CAAgFiEEmuvCXT0aY6hs4SbWeVOEFl5WrMgFAl+pQfkCGwwACgkQeVOEFl5WrMiVqwf9GwU8 c6cylknZX8QwlsVudTC8xr/L17JA84wf03k3d4wxP7bqy5AYy7jboZMbgWXngAE/HPQU95NM aukysSnknzoIpC96XZJ0okLBXVS6Y0ylZQ+HrbIhMpuQPoDweoF5F9wKrsHRoDaUK1VR706X rwm4HUzh7Jk+auuMYfuCh0FVlFBEuiJWMLhg/5WCmcRfiuB6F59ZcUQrwLEZeNhF2XJV4KwB Tlg7HCWO/sy1foE5noaMyACjAtAQE9p5kGYaj+DuRhPdWUTsHNuqrhikzIZd2rrcMid+ktb0 NvtvswzMO059z1YGMtGSqQ4srCArju+XHIdTFdiIYbd7+jeehg== Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 on 64.7.153.18 X-Spamd-Result: default: False [-1.37 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_SPAM_LONG(0.95)[0.952]; NEURAL_HAM_SHORT(-0.93)[-0.934]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; XM_UA_NO_VERSION(0.01)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; FROM_HAS_DN(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; FREEFALL_USER(0.00)[mike]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[sentex.net]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4SV6wj20RGz3N6F X-Spamd-Bar: - While testing some new hardware on a recent RELENG_14 image (from Nov 10th), I noticed some of my ssh sessions would get killed off with the errors below (twice in 24hrs) pid 1697 (sshd), jid 0, uid 1001, was killed: failed to reclaim memory pid 6274 (sshd), jid 0, uid 1001, was killed: failed to reclaim memory Nothing fancy bencthmark wise, I am just testing a whole mess of HDDs off a backplane by generating some synthetic traffic on a big pool of disks. 65G of RAM. ARC is not limited and seems to try and take the max possible.  Any ideas what might be going on ? CPU:  1.1% user,  0.0% nice, 20.1% system,  0.0% interrupt, 78.7% idle Mem: 124K Active, 16M Inact, 6156K Laundry, 59G Wired, 3061M Free ARC: 53G Total, 1418M MFU, 50G MRU, 374M Anon, 389M Header, 396M Other      50G Compressed, 211G Uncompressed, 4.22:1 Ratio Swap: 4096M Total, 22M Used, 4074M Free Script is #!/bin/sh while true do bonnie -s 190000 -d /hddpool/test/ md5 /hddpool/junk* bonnie++ -u root -d /hddpool/test date sleep 10 done pool looks like # zpool status   pool: hddpool  state: ONLINE   scan: scrub repaired 0B in 00:01:39 with 0 errors on Mon Nov 13 08:33:53 2023 config:         NAME        STATE     READ WRITE CKSUM         hddpool     ONLINE       0     0     0           raidz1-0  ONLINE       0     0     0             da0p1   ONLINE       0     0     0             da7p1   ONLINE       0     0     0             da12p1  ONLINE       0     0     0             da13p1  ONLINE       0     0     0           raidz1-1  ONLINE       0     0     0             da1p1   ONLINE       0     0     0             da6p1   ONLINE       0     0     0             da9p1   ONLINE       0     0     0             da11p1  ONLINE       0     0     0           raidz1-2  ONLINE       0     0     0             da4p1   ONLINE       0     0     0             da5p1   ONLINE       0     0     0             da8p1   ONLINE       0     0     0             da10p1  ONLINE       0     0     0 errors: No known data errors # pstat -T  94/2090092 files 22M/4096M swap space # cat /etc/fstab # Device                Mountpoint      FStype  Options Dump    Pass# /dev/ada0p2             none    swap    sw              0       0 /dev/ada1p2             none    swap    sw              0       0 #