From nobody Sat May 25 21:44:31 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VmwRg1Z5jz5LjQ1 for ; Sat, 25 May 2024 21:44:47 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VmwRd2k15z40dc for ; Sat, 25 May 2024 21:44:45 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=iitbombay-org.20230601.gappssmtp.com header.s=20230601 header.b=iaq+TFW2; dmarc=pass (policy=quarantine) header.from=iitbombay.org; spf=pass (mx1.freebsd.org: domain of bakul@iitbombay.org designates 2607:f8b0:4864:20::102c as permitted sender) smtp.mailfrom=bakul@iitbombay.org Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-2b9702e05easo4721034a91.1 for ; Sat, 25 May 2024 14:44:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay-org.20230601.gappssmtp.com; s=20230601; t=1716673483; x=1717278283; darn=freebsd.org; h=message-id:in-reply-to:to:references:date:subject:mime-version :content-transfer-encoding:from:from:to:cc:subject:date:message-id :reply-to; bh=T0sDJyARzZHMfwYqm5VgkzDsg/OEcZkYlWPzv2JN8oY=; b=iaq+TFW2/lrOBfhgS94a+idOWAdeSzG+5m/jDtqqtls+fSE4lC/hG5DrFkilNu6N/o g2DPMzCKcDRaqF05eS+CWdSsNIdCofiysTE2elSol6XfFNM3k+uPat118Hq+xFm1CZ5h SoJx3NZgZDRAfJ/9T+3UQuHkyGlglfdV2IYtq3AbBbcSkrp87jqreQCJ3AkpQnRgxu4N Eatw2FZw8uzqWE5iy2Rv1uDzLDYvK/x87lNv9Bx0RV9m+Gmq8t5ZScJK8Arf986HSMha bNagv2GyL10DRu6h+mqnRFXxhNXBd/154B6Q2+LktCWA7c1A94iRcQVgna2GUBMkELwM 38bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716673483; x=1717278283; h=message-id:in-reply-to:to:references:date:subject:mime-version :content-transfer-encoding:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T0sDJyARzZHMfwYqm5VgkzDsg/OEcZkYlWPzv2JN8oY=; b=M8DFbi7PUGLdHLodw5ZXdNN4qe1RikMuohgLYIzCKKkNDwAHAJCLttZ0cl3bZYzPtD PjsJlbmvuezWSk8TxcthKdqnviBC9kEC5XnYUkgBnOPml2Ln5r4Y1Nm+wY9EkYG3YNN1 9EzrnTRlHyYsmXsjw2FQkAoNfcjumYvmfOntRmXD2N1iafM/qz4mK0BN40C97XcDJE9D yCBFX6t4DcJzCYNnF7aOa92FsUjTimanzt0+i/lX+LRvN7xrRKz8R2OC5tFHbuqRb9wq Udv5rcDaCRiftHGA6IEQqBAhUrcE58B3qlZFlI876/Suq0PIthySR+euo8sMY21Hmy9Y XUUw== X-Gm-Message-State: AOJu0YxbTi93aY7NDlZIXvG0PEapUjCQpFLk2zL29QqcNily7PGGh8Fm axTJ4MoNUw7FC0CLEkBwLgTl9gqHkSQJi6m3XPnOFKWfdJ/wj2QJViDY+yHjKeQ17gNmoMsBd50 = X-Google-Smtp-Source: AGHT+IF6X/2CRDPa9etEnk1y+h6i8w1VVzihHn29syfrJWpxDdjKsegj39VkQ4qUELxgjAk3y3kbfA== X-Received: by 2002:a17:902:d2d0:b0:1f4:58aa:3623 with SMTP id d9443c01a7336-1f458aa44bbmr46367875ad.65.1716673482635; Sat, 25 May 2024 14:44:42 -0700 (PDT) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1f44c7580ebsm34136865ad.30.2024.05.25.14.44.41 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 25 May 2024 14:44:42 -0700 (PDT) From: Bakul Shah Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\)) Subject: Re: Question regarding crunchgen(1) binaries Date: Sat, 25 May 2024 14:44:31 -0700 References: <202404150105.43F15VoL068210@donotpassgo.dyslexicfish.net> <7lmqszm7n35b5jitwvzagmlc2lecl6p3dhu2bnhri4unnjtlow@f5txrntbo7yw> <202404151955.43FJtMnU083779@critter.freebsd.dk> To: FreeBSD Hackers In-Reply-To: <202404151955.43FJtMnU083779@critter.freebsd.dk> Message-Id: X-Mailer: Apple Mail (2.3774.600.62) X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[iitbombay.org,quarantine]; R_DKIM_ALLOW(-0.20)[iitbombay-org.20230601.gappssmtp.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[iitbombay-org.20230601.gappssmtp.com:+]; TO_DN_ALL(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MIME_TRACE(0.00)[0:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEFALL_USER(0.00)[bakul]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102c:from] X-Rspamd-Queue-Id: 4VmwRd2k15z40dc On Apr 15, 2024, at 12:55=E2=80=AFPM, Poul-Henning Kamp = wrote: >=20 > -------- > Warner Losh writes: >=20 >> Maybe start there to understand what "LTO" the security thing is = doing and >> why it's either wrong or violates an assumption in crunchgen that can = be >> fixed. >=20 > Crunch binaries were invented 30 years ago, to make FreeBSD=20 > installation program fit on a single floppy disk. >=20 > Note that the goal was saving disk-space rather than RAM. >=20 > The "architecture" of crunchgen is to take a lot of programs, rename > their main() and link them all together with a new main() which > dispatches to the right program's main() based on argv[0] >=20 > Statistically you save half a disk-allocation unit for each program > which was nothing to sneeze at, but the real disk-space dividend > comes from linking the resulting combi-program static. >=20 > Because it is linked static, only those .o files which are referenced > gets pulled in from the libraries, libm::j0.o only gets pulled in > if you Bessel functions, which, countrary to rumours, sysinstall > did not. >=20 > (The goal of shared libraries is saving RAM: Everybody gets the > complete library, but only one copy of it's code ever gets loaded.) >=20 > But the real trick is actually not crunchgen, which was originally = just > a shell script, but rather crunchide(1). >=20 > Crunchide(1) does unnatural acts to an objectfile's symboltabel, > to get around the fact that all the programs have a function called > "main" and that they litter the global symbol namespace with their > private inter-file references. >=20 > To make a crunched binary, the .o files for the individual programs > are first "pre-linked" without libraries so that internal interfile > references are resolved. >=20 > Then crunchide changes all global symbols, except "main" to be local > symbols, so that they become unavailable for symbol resolution in > the final run of the linker. The "main" symbol is also renamed > to a per-program name, something like "cp_main" for cp(1) etc. >=20 > And then all the prelinked .o files, one per program, gets linked > together with the "dispatch main" and this time with libraries. >=20 > I see no reason why crunchgen cannot be done with Link Time > Optimization, but somebody has to write the new crunchide(1), and > I suspect it will have a tougher row to hoe, because pre-linking > cannot be used to take care of the inter-program symbols. >=20 > As I understand it LTO can also link with "normal libraries" > so one option might be to only LTO the final linking step of > the crunch process, treating all the programs as "normal libraries", > but still getting LTO advantage internally in the libraries. I'd asked Jaime Da Silva (the original author of crunchgen) about this. He eventually checked his spam chocked personal domain mbox and saw my message. He had this to say: I haven't touched crunch in ~30 years. No doubt "crunchide" is the problem, zapping symbols needed by CFI and LTO. Assuming these advanced techniques can work with multiple link passes ("ld -r") then it should be possible to modify crunchide to rename symbols rather than zapping them. I am a little surprised crunch is still in use in freebsd. I think the concept, if it were more flexible, would still have traction in embedded systems, but everyone seems to be fine with just using busybox and calling it done. In case this is useful!