From nobody Wed Jul 31 18:51:39 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WZ1R257Xfz5R82l for ; Wed, 31 Jul 2024 18:51:42 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-ot1-x330.google.com (mail-ot1-x330.google.com [IPv6:2607:f8b0:4864:20::330]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4WZ1R216JPz4Ptn for ; Wed, 31 Jul 2024 18:51:42 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-ot1-x330.google.com with SMTP id 46e09a7af769-709485aca4bso2542957a34.1 for ; Wed, 31 Jul 2024 11:51:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; t=1722451901; x=1723056701; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=nZzZx6jVzUCnA0taDbobiwaDjINrS3ymc0NvhAY4LZU=; b=HVzOiNOXxwmnlsZIzR02pi/ArPBh8lOPAUB1jqjn+wnCJt0E+ChEF0IJbZUhxLlWEL 0ZJw7UuquByzh62eWm7z9KIkYkbFlnzU1RXxhjDWuezbA0udTg/8WzW7IeDQ5NF6bwyO RNNkkkZ5Nn6ZFUYp/sdCZQ6cAcT99L2Z1Lc+/re+IvJtX7IK7fV9hbTlQ4IQw2OeRDDm SGToCgIeELP60ASrBZC7LB6EYLT9CERhaEc/5/FYCaMnZv/3UjPeljYa21wUu722lazX No438TzDrbYCuCGVyxi0yP0Vkag//QW5AaAxJqaUCmHXoNbOxL/THRnCIn429QAIeHJR em9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722451901; x=1723056701; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=nZzZx6jVzUCnA0taDbobiwaDjINrS3ymc0NvhAY4LZU=; b=XpjrYk/pr7eVg+KNaoJDUO5w4/n2FCk2f6GmYS9j8MD10GqB4Z6XATSCcE12Zc5gIT pSYDZh0IU2ndvNBPn1wP8bdWnuHeAQZGjxFldJ1OI/HRcu7uCgN+2LLnhHavFwQskR28 p/1FFq0AYbqrD0l2JYAlYZsvFKjrWE6/TXsY2IjmBJP2oeoafQk8qAamWGUbxyPVRWH2 Kuyfbskmalt5A0SKgusjyafCmQZXCbPhfRAhSsYFVdqvnynqjTVLsXvVZQ4CC2RmA2/F 8vPwOY92oY/1XGRpUhve2eSh9u+ViheQ3Yf5i6S7DhwEiiwfbEWEa1Xzv9a38I33ZHdD pZuw== X-Forwarded-Encrypted: i=1; AJvYcCW2wh9LBwUcYRrbkgnOqFO+dYBecvlmd/u+Z2bblaNKJrI/WqhLKchroQGMShf4YrHkck0gyvL44+hEkpYcikRiTnvRDUilv+oSjgw= X-Gm-Message-State: AOJu0Yy0qz/DDShJoUHGT5QKxYzHbLzXNkvPeKzG1YBImJJ5UNe9md5s tYE2gcKq0HWyvw1qinGYDUTLc8ZXXpl32bk4R2/Pb+WOzyEBsarXQrgLzgFYTlA= X-Google-Smtp-Source: AGHT+IEAz0/f3yD5w/czW6go8CJDc6rknfedbtI2Mc/4TwJd1sedY2GkYurCSRWJLbvYE3PNNfvvbg== X-Received: by 2002:a05:6830:2a11:b0:704:2748:5ab6 with SMTP id 46e09a7af769-7096b84669fmr51658a34.20.1722451900756; Wed, 31 Jul 2024 11:51:40 -0700 (PDT) Received: from mutt-hbsd (174-24-87-135.clsp.qwest.net. [174.24.87.135]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-7095ac7ba70sm1014852a34.0.2024.07.31.11.51.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 11:51:40 -0700 (PDT) Date: Wed, 31 Jul 2024 18:51:39 +0000 From: Shawn Webb To: Warner Losh Cc: Alan Somers , FreeBSD Hackers , Scott Long , Goran Meki?? Subject: Re: The Case for Rust (in the base system) Message-ID: X-Operating-System: FreeBSD mutt-hbsd 15.0-CURRENT-HBSD FreeBSD 15.0-CURRENT-HBSD X-PGP-Key: https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/blob/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="l2ox7s2au2w6utkz" Content-Disposition: inline In-Reply-To: X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4WZ1R216JPz4Ptn --l2ox7s2au2w6utkz Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jul 31, 2024 at 11:01:17AM -0600, Warner Losh wrote: > On Wed, Jul 31, 2024, 9:40=E2=80=AFAM Alan Somers w= rote: >=20 > > On Wed, Jul 31, 2024 at 8:37=E2=80=AFAM Shawn Webb > > wrote: > > > > > > On Sat, Jan 20, 2024 at 09:51:25AM -0700, Alan Somers wrote: > > > > In a recent thread on src-committers, we discussed the costs and > > > > benefits of including Rust code in the FreeBSD base system. To > > > > summarize, the cost is that it would double our build times. imp > > > > suggested adding an additional step after buildworld for stuff that > > > > requires an external toolchain. That would ease the build time pai= n. > > > > The benefit is that some tools would become easier to write, or even > > > > become possible. Here is a list of actual and potential Rust proje= cts > > > > that could benefit from being in-tree. If anybody else has items to > > > > add, I suggest moving this into the project wiki: > > > > > > > > Stuff that could only be written in Rust if it were in base > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > > > * ctl-exporter (I started this, but discovered that the CTL stats A= PI > > is > > > > unstable, so it can't live in ports. Instead, I had to do it in = C). > > > > > > https://github.com/freebsd/freebsd-src/commit/1a7f22d9c211f504f6c48a864= 01469181a67ec34 > > > > > > > > * fusefs tests. Absolutely impossible to do in C. I considered Ru= st, > > but went > > > > with C++ so they could live in base. They are too closely couple= d to > > > > fusefs(5) to live out-of-tree. > > > > https://github.com/freebsd/freebsd-src/tree/main/tests/sys/fs/fus= efs > > > > > > > > * devd. Currently C++, but imp suggested a rewrite. > > > > https://github.com/freebsd/freebsd-src/tree/main/sbin/devd > > > > > > > > * zfsd. Currently C++, but I've long pondered a rewrite. Using Ru= st > > would > > > > make it more testable. > > > > https://github.com/freebsd/freebsd-src/tree/main/cddl/usr.sbin/zf= sd > > > > > > > > * nscd. Currently C, but confusing and with no test coverage. I've > > > > contemplated a rewrite myself, but I don't want to do it in C. > > > > https://github.com/freebsd/freebsd-src/tree/main/usr.sbin/nscd > > > > > > > > * The userland portion of the 802.11ac and Lightning stacks. scottl > > suggested > > > > that these were good candidates for Rust. > > > > > > > > * freebsd-kpi-r14-0 . https://crates.io/crates/freebsd-kpi-r14-0 > > > > > > > > Stuff that can live in ports, but would be nicer in base > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > > > * gstat-rs https://crates.io/crates/gstat > > > > > > > > * geom-exporter (I've started this, but haven't published it) > > > > > > > > * nfs-exporter https://crates.io/crates/freebsd-nfs-exporter > > > > > > > > * virtiofsd-rs . Nobody has yet tried to port it to FreeBSD. But = if > > the > > > > connection to bhyve(8) is too intimate, it might be hard to do in > > ports. > > > > https://gitlab.com/virtio-fs/virtiofsd > > > > > > > > * jail-exporter https://crates.io/crates/jail_exporter > > > > > > > > * Various jail managers have been attempted in Rust. I think these > > are fine in > > > > ports, but others like Goran Mekic have opined that they should be > > moved to > > > > base instead. > > > > > > > > * musikid's pjdfstest rewrite. I think it would be great to start > > using this > > > > to test the base system's file systems. If the tests themselves > > lived in > > > > base, they would be easier to sync with file system development. > > > > https://github.com/musikid/pjdfstest > > > > > > > > * pf-rs. I suspect that the API isn't very stable. > > > > https://crates.io/crates/pf-rs > > > > > > > > * benchpmc. The pmc counter names changes between releases. > > > > https://crates.io/crates/benchpmc > > > > > > > > FreeBSD-related applications that are just fine in ports > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > > > * fsx-rs. Unlike pjdfstest, this only tests datapath APIs. Those = are > > usually > > > > more stable than control path APIs, so I think there's little to = be > > gained by > > > > moving this into base. https://crates.io/crates/fsx > > > > > > > > * ztop. It uses ZFS's kstats sysctl interface, which is pretty sta= ble. > > > > https://crates.io/crates/ztop > > > > > > > > * iocage-provision https://crates.io/crates/iocage-provision > > > > > > > > * rsblk https://crates.io/crates/rsblk > > > > > > > > * xfuse https://github.com/KhaledEmaraDev/xfuse > > > > > > > > Other FreeBSD-related libraries in Rust > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > Just see the list at https://crates.io/keywords/freebsd > > > > > > > > > > One new data point: DARPA is looking to rewrite a significant amount > > > of C code to Rust with their "Translating All C to Rust (TRACTOR)" > > > project: > > > https://sam.gov/opp/1e45d648886b4e9ca91890285af77eb7/view > > > > Interesting. And since you bring it up, I have two new data points mys= elf: > > > > * ctld: while working on some bugs in ctld, I had trouble > > understanding the config file parsing. So I rewrote that part in > > Rust, just to help my understanding. Later, I rewrote the XML > > parsing, too. Then I rewrote the LUN creation and deletion, just to > > see how hard it would be. All of those parts take about 5x fewer SLOC > > in Rust than in C, and they're less buggy, too. Config file parsing > > is more consistent, no memory leaks, etc. Alas, I'm not planning to > > finish this project, since the base system doesn't allow Rust and ctld > > is too tightly coupled to ctl to live in ports. > > >=20 > Cool. Still waiting for anybody to take me up on the offer to do build > system integration. Since the Rust advocates can't get even this basic st= ep > done for review, it's going to be impossible to have Rust in the base. Th= is > isn't even integrate rust compiler like we do with llvm, but with external > Rust toolchain. >=20 > Until somebody steps up for this task, the status quo can't possibly chan= ge. Back at the FreeBSD Developer Summit at this last BSDCan, there was interest in supporting optional external toolchains in the src build framework. You had mentioned you would be happy to mentor someone, but not do the nitty gritty yourself. I could carve off some time in September to be the primary developer, doing the nitty gritty work. Would you be comfortable answering my questions, should I have any? Also: what work (or research), if any, has been done on the concept of external toolchain support for optional components in the FreeBSD source tree? Am I starting afresh or building upon existing work? Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50 https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A= 4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc --l2ox7s2au2w6utkz Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmaqh7MACgkQ/y5nonf4 4frCGg//fh37VVA0JqPk72OOOoPW3RmJ+AhJ2F5KHKMVScrgJGl6vrzmrbqfJe0P 0tp+6TCx/nSHZHfqq8RL+S5vimv7vq+BdZgyKoFSnprjsLODe28azIilvaj/z5tJ 2DIUMWVkIOyDHojx8uxe3B/EfSOrryVGWjx1ltwRn2AcG/lEEqedwBej999fd2jo 8zBbrM/Zi/B7A2u+Sp0WM+D4jvdQLr6GHK+hteZNkovAH1O5SxdGg7nZVFIWSP8h tQpCnt7hg8awyE4AT5FPNZmtJdHnMg9K9r8I2peOwUaFleuqiIJ/eW0FP3fZ6QVh ilda8XfPiZ5TYBM5rnhZR/3vsjz68zhKpo8AypAIeLaUqFNqDIHz4VuyYwhSPox0 HDjttwThDizqqx9Pg0j7YaStjVNH8mlIgLYT22hqH8eDFSnQfInLRX7n37HcZ8DU WsSW2Z0tfEwBG587T95xVdwxlM+jZjH2mdXCaRYhb5drsEp1jePY6AAfeK6iPjmo qXF1aOffPfAkA5wqspWlcklfbpUqMh8w2RqgzrWSw5vzE3mK7gU1dRYI1Z7s0Eii lz/3dxstAYJzcwoN/Rnavehitohj9QwNX0BH5Qy2OecZAjiav8IbZ0irfxnr+8Bo ii6yD7FJiWRC7EnGq3Rg82JHlh0/+FNtbLPGzqVhxf93arxRQJ8= =oSjC -----END PGP SIGNATURE----- --l2ox7s2au2w6utkz--