From nobody Sun Jan 28 14:15:56 2024 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TND516Pfcz58fvL; Sun, 28 Jan 2024 14:16:37 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail.madpilot.net (vogon.madpilot.net [159.69.1.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TND506gXyz57yS; Sun, 28 Jan 2024 14:16:36 +0000 (UTC) (envelope-from mad@madpilot.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=madpilot.net header.s=bjowvop61wgh header.b="D XbesPe"; dmarc=pass (policy=quarantine) header.from=madpilot.net; spf=pass (mx1.freebsd.org: domain of mad@madpilot.net designates 159.69.1.99 as permitted sender) smtp.mailfrom=mad@madpilot.net Received: from mail (mail [IPv6:fd5c:5351:d272::3]) by mail.madpilot.net (Postfix) with ESMTP id 4TND4H1qHzz6fL4; Sun, 28 Jan 2024 15:15:59 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=madpilot.net; h= content-transfer-encoding:content-type:content-type:in-reply-to :content-language:references:from:from:subject:subject:date:date :message-id:received; s=bjowvop61wgh; t=1706451357; x= 1708265758; bh=6i1JPJVn0NDGonSa+fopcBkHZYgpc/ajneNCiXAiIjk=; b=D XbesPeSSwO9WpJtpUhfh7Uwl71osnuTg1fa87GMlZ2BVqTdWwwYlrNfrm0rcCLyT /qmNcYTRAYWt75ICrCuE/WUG5yd0vSdlL4b6x3QEX9KHYmsBJAjqJytizEqy0rJL Pmni2H2U7JvFTBWZPwgK8H23/gucIs4QimrqOfMh6mk9b0YUZ2AJg3d6dcVMlVw4 Er4qV3JfZ0PcLL6O9OB9ZC6xjHNi7a4yikMA+Zux4z7bT5+Vkp7DyRPPv7uWgHLH JnGEgtEQJKV/34pmiKU+7pjnhgKAq06+X0rdEE0J4ueumlZx20DnxQEt28PhGyK3 Pam9ruAL8hUcIvWZFEo1w== Received: from mail.madpilot.net ([IPv6:fd5c:5351:d272::3]) by mail (mail.madpilot.net [IPv6:fd5c:5351:d272::3]) (amavisd-new, port 10026) with ESMTP id c0TspYulRFUS; Sun, 28 Jan 2024 15:15:57 +0100 (CET) Message-ID: <0fc7f929-6e5b-4a33-97d2-8a9c0c07d524@madpilot.net> Date: Sun, 28 Jan 2024 15:15:56 +0100 Subject: qemu-user-static aarch64 lockup/race? (was Re: Python failure in poudriere on arm64 (via qemu-user-static cross compiling)) From: Guido Falsi To: emulation@FreeBSD.org, "freebsd-arm@freebsd.org" References: <6a33726b-eb6f-418e-9fbd-6d0b9b4bfaa8@madpilot.net> Content-Language: en-US Autocrypt: addr=mad@madpilot.net; keydata= xsBNBE+G+l0BCADi/WBQ0aRJfnE7LBPsM0G3m/m3Yx7OPu4iYFvS84xawmRHtCNjWIntsxuX fptkmEo3Rsw816WUrek8dxoUAYdHd+EcpBcnnDzfDH5LW/TZ4gbrFezrHPdRp7wdxi23GN80 qPwHEwXuF0X4Wy5V0OO8B6VT/nA0ADYnBDhXS52HGIJ/GCUjgqJn+phDTdCFLvrSFdmgx4Wl c0W5Z1p5cmDF9l8L/hc959AeyNf7I9dXnjekGM9gVv7UDUYzCifR3U8T0fnfdMmS8NeI9NC+ wuREpRO4lKOkTnj9TtQJRiptlhcHQiAlG1cFqs7EQo57Tqq6cxD1FycZJLuC32bGbgalABEB AAHNHkd1aWRvIEZhbHNpIDxtYWRAbWFkcGlsb3QubmV0PsLAeAQTAQIAIgUCT4b6XQIbAwYL CQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQGuaGDlbL0pOWigf/YVTVf3+ZRnzeGP7CjGV1 Wrrxzjc8h8W64NZasV0XLHGFjl5MYwtm9jJ9gbL8Ubtqstey7lYpjOk2fG6YDhY5eptWCpR6 1QqYrioukhCfKbodSk6PnIZcx719nJVK2P7ihdFEN78TavpBwqIf9hGEcKkMpbRFQv1mYvXD hKVwQGY+8bkH/a/pAWmIyD4qMfKCMurH5DexxEt5SYWu5BB5hd/DWyZ0wuZ+F79KMPzLBPJW 5cpdLNbrvenSqFZGJEGhtTp7GFJJr6lTy8VLBArxmFHiY5jGyR45eZEGDcz86FfGgvPnnpi7 aNCc/ROdF7fnZYPh8uZGGjQbd4EYK4xMzc7BTQRTEHtBARAAoWGsNx6g90r8gcNKaiPpJBiK y8ztV2FyV5LsT0OgQBW3vIxt/odtsxVNNjpyS/BNZCyzLAsFc1WrGBzhYsmPN9SGB5/5YTvk zf5YViU5VAsZlj/MRWCZrWtpic4c0A7N4csOYReNtk/q8YB4PIFsZ9A+kTuoZhnu5t5PdfBA 74+SVwKu84+PZk9wDEY1LbFVT8vM42oKsmoswlIhwJ2xuJI/gbk+cMUe0yiRpNjo4Svw4RB8 4B6uFwdRr/PtS7xi2Zqoof5AaQT9YSBpGpKJOe/Qk5MP4PF6Fqq+go89n77Y2kJkwcHaLoD/ GJ+ZDASIiMRe1y54FHOQ1RCTGGpnJLXdKuGhwv3J21pU8HNlq0ASNQMMQmYAwtUWzjmp/KEy I1qkcmjafcxb8TmiaoK8SQN1Zf96fc/sIrZN6Z5oOCEyyCQ0prH/PTA2jlRkKQ487PTGk2JS KU5VuS57Nlk2DrnvjWp57aV9eFAhpnrrJPuGmFz83/Pc8gC0t7N7i7VVHYRcC5naxYB2UoI1 OUkyxpT/HvQFXXVZ3/KmdXMzrx191AggCPWIwUAP+VcaURSYpeDk6/ZVAOVOe1ChqcJisCD7 wK20/OOvJ2AtkWreGu1CZ9zSx7nK/VYdLr34GxQ4bT1G+9rBQNnFSNbX2TJ431Mdo1GCjDeR K4CtSnrNKYkAEQEAAcLAXwQYAQgACQUCUxB7QQIbDAAKCRAa5oYOVsvSkw3nCADhsKRf+rAR ULTpOh5HoLam62ZJZAyCkNqqu/rke5uj5AaaDY/h7BNhBDiDqhhZLTeofGpVVaErPsWN+tX5 0fypsIt9KAhy90GFrtrIZlWuyK4wsoZvDfp9yaRk+lIM58dw/Rcfxn670JaPTFSRPECVn/uL qBhJSkbYlY212YT9fxVUTJe6wIvDLQrQEjrQD/h1FMhfcLhAqsndltRd6DPvTKeMd/6VAxn0 hkoBKhEy5LkWjM9CHppu+bBkQ91/kj2uJQSXO8euonwHHS3c+6N2i2H7I0emcHGu07wuRB2t Dnw/RLBxohffdPZT2kbxuG7lhVHzwVDw5DRwSw8GkOdy In-Reply-To: <6a33726b-eb6f-418e-9fbd-6d0b9b4bfaa8@madpilot.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: - X-Spamd-Result: default: False [-2.00 / 15.00]; MISSING_MIME_VERSION(2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[madpilot.net,quarantine]; R_DKIM_ALLOW(-0.20)[madpilot.net:s=bjowvop61wgh]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; TO_DN_EQ_ADDR_SOME(0.00)[]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:24940, ipnet:159.69.0.0/16, country:DE]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[madpilot.net:+]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[emulation@FreeBSD.org,freebsd-arm@freebsd.org]; SUBJECT_HAS_QUESTION(0.00)[] X-Rspamd-Queue-Id: 4TND506gXyz57yS List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Hi all, again, I have some more findings about this, I'm top posting because the old message is not really that much relevant anymore. I'm now running a machine with head (commit b32d49cfbaa0437d08e65e7cd7c82c5951b1a852 Jan 25th), poudriere installed in it, machine is amd64, with an arm64 jail, 14.0-RELEASE, installed from official distribution binaries (https download method), with cross tools. To make sure everything is aligned I rebuild everything: updated head, rebuild cross tools in the jail, recompiled all ports for the host architecture and force reinstalled them, especially qemu-user-static, cleaned up all packages for the arm64 jail. If I missed something important please point it out. I have made some more tests and I'm getting python failures in poudriere like the one described below from time to time (don't have hard stats but feels like 50% chance). If I get past that it usually is able to build all the not many packages, but locks up at: Creating repository in /tmp/packages: 0% with nCPUs processes like this: > ps -ax | grep -i pkg 91287 1 I+J 0:00.02 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91288 1 I+J 0:00.02 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91289 1 I+J 0:00.00 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91290 1 I+J 0:00.00 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91291 1 I+J 0:00.00 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91292 1 I+J 0:00.00 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91293 1 I+J 0:00.00 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91294 1 I+J 0:00.00 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91295 1 I+J 0:00.00 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91296 1 I+J 0:00.00 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91297 1 I+J 0:00.00 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91298 1 I+J 0:00.00 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91299 1 I+J 0:00.00 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key 91300 1 I+J 0:00.00 /usr/local/bin/qemu-aarch64-static /.p/pkg-static repo -o /tmp/packages /packages /tmp/repo.key And this has hit me 100% of the time up to now. Looks like it is pkg spawning ncpu processes, I'm looking at reducing them, just in case this can sidestep the race/lockup. My suspect is there is some race with quemu-user-static or the APIs it is using, that is triggered by pkg-repo. How can I investigate this? I'm able to reproduce it 100% of the time. BTW these are the pkgs I'm building at present: dns/unbound net-mgmt/vmutils net/kea sysutils/htop sysutils/node_exporter sysutils/tmux (vmutils and node_exporter are go packages and are being skipped since go fails, but I keep them in the list, since I can grab binaries from the official repos, htop I'm going to drop in the near future) Thanks in advance, any help appreciated, especially any suggestions for where to look at and investigation to understand if this is a local problem, or some issue with base/qemu. On 24/01/24 22:10, Guido Falsi wrote: > Hi, > > I recently see a strange failure with python 3.9 in poudriere, it was > not happening a few weeks ago. > > I'm building in poudriere on a head machine running amd64, with a > poudriere jail for arm64, via qemu-user-static. The jail is running 14.0. > > I'm not sure what is going on. > > It fails in the packaging phase with a bunch of errors like: > > =========================================================================== > =================================================== > ===== env: 'PKG_NOTES=build_timestamp ports_top_git_hash > ports_top_checkout_unclean port_git_hash port_checkout_unclean built_by' > 'PKG_NOTE_build_timestamp=2024-01-24T17:07:52+0000' > 'PKG_NOTE_ports_top_git_hash=0816fdcb6ce8' > 'PKG_NOTE_ports_top_checkout_unclean=no' > 'PKG_NOTE_port_git_hash=0816fdcb6ce8' > 'PKG_NOTE_port_checkout_unclean=no' > 'PKG_NOTE_built_by=poudriere-git-3.4.1' NO_DEPENDS=yes USER=root UID=0 > GID=0 > ===>  Building packages for python39-3.9.18 > ===>   Building python39-3.9.18 > pkg-static: Unable to access file > /wrkdirs/usr/ports/lang/python39/work/stage/usr/local/lib/python3.9/__pycache__/imaplib.cpython-39.opt-2.pyc:No such file or directory > pkg-static: Unable to access file > /wrkdirs/usr/ports/lang/python39/work/stage/usr/local/lib/python3.9/__pycache__/imghdr.cpython-39.opt-2.pyc:No such file or directory > pkg-static: Unable to access file > /wrkdirs/usr/ports/lang/python39/work/stage/usr/local/lib/python3.9/__pycache__/imp.cpython-39.opt-2.pyc:No such file or directory > pkg-static: Unable to access file > /wrkdirs/usr/ports/lang/python39/work/stage/usr/local/lib/python3.9/__pycache__/inspect.cpython-39.opt-2.pyc:No such file or directory > pkg-static: Unable to access file > /wrkdirs/usr/ports/lang/python39/work/stage/usr/local/lib/python3.9/__pycache__/io.cpython-39.opt-2.pyc:No such file or directory > pkg-static: Unable to access file > /wrkdirs/usr/ports/lang/python39/work/stage/usr/local/lib/python3.9/__pycache__/ipaddress.cpython-39.opt-2.pyc:No such file or directory > pkg-static: Unable to access file > /wrkdirs/usr/ports/lang/python39/work/stage/usr/local/lib/python3.9/__pycache__/mailbox.cpython-39.opt-2.pyc:No such file or directory > pkg-static: Unable to access file > /wrkdirs/usr/ports/lang/python39/work/stage/usr/local/lib/python3.9/__pycache__/mailcap.cpython-39.opt-2.pyc:No such file or directory > pkg-static: Unable to access file > /wrkdirs/usr/ports/lang/python39/work/stage/usr/local/lib/python3.9/__pycache__/mimetypes.cpython-39.opt-2.pyc:No such file or directory > > > > (it's all about 'opt-2.pyc' files) > > > What could have changed? Maybe I'm doing something wrong? Maybe I'm > hitting some qemu-user-static issue on head? > > > Any help appreciated. > > > (full log available if needed) > -- Guido Falsi