Re: The Case for Rust (in the base system)

On Wed, Jul 31, 2024 at 11:01:17AM -0600, Warner Losh wrote:
> On Wed, Jul 31, 2024, 9:40 AM Alan Somers <asomers@freebsd.org> wrote:
> 
> > On Wed, Jul 31, 2024 at 8:37 AM Shawn Webb <shawn.webb@hardenedbsd.org>
> > wrote:
> > >
> > > On Sat, Jan 20, 2024 at 09:51:25AM -0700, Alan Somers wrote:
> > > > In a recent thread on src-committers, we discussed the costs and
> > > > benefits of including Rust code in the FreeBSD base system.  To
> > > > summarize, the cost is that it would double our build times.  imp
> > > > suggested adding an additional step after buildworld for stuff that
> > > > requires an external toolchain.  That would ease the build time pain.
> > > > The benefit is that some tools would become easier to write, or even
> > > > become possible.  Here is a list of actual and potential Rust projects
> > > > that could benefit from being in-tree.  If anybody else has items to
> > > > add, I suggest moving this into the project wiki:
> > > >
> > > > Stuff that could only be written in Rust if it were in base
> > > > ===========================================================
> > > >
> > > > * ctl-exporter (I started this, but discovered that the CTL stats API
> > is
> > > >   unstable, so it can't live in ports.  Instead, I had to do it in C).
> > > >
> > https://github.com/freebsd/freebsd-src/commit/1a7f22d9c211f504f6c48a86401469181a67ec34
> > > >
> > > > * fusefs tests.  Absolutely impossible to do in C.  I considered Rust,
> > but went
> > > >   with C++ so they could live in base.  They are too closely coupled to
> > > >   fusefs(5) to live out-of-tree.
> > > >   https://github.com/freebsd/freebsd-src/tree/main/tests/sys/fs/fusefs
> > > >
> > > > * devd.  Currently C++, but imp suggested a rewrite.
> > > >   https://github.com/freebsd/freebsd-src/tree/main/sbin/devd
> > > >
> > > > * zfsd.  Currently C++, but I've long pondered a rewrite.  Using Rust
> > would
> > > >   make it more testable.
> > > >   https://github.com/freebsd/freebsd-src/tree/main/cddl/usr.sbin/zfsd
> > > >
> > > > * nscd.  Currently C, but confusing and with no test coverage.  I've
> > > >   contemplated a rewrite myself, but I don't want to do it in C.
> > > >   https://github.com/freebsd/freebsd-src/tree/main/usr.sbin/nscd
> > > >
> > > > * The userland portion of the 802.11ac and Lightning stacks.  scottl
> > suggested
> > > >   that these were good candidates for Rust.
> > > >
> > > > * freebsd-kpi-r14-0 .  https://crates.io/crates/freebsd-kpi-r14-0
> > > >
> > > > Stuff that can live in ports, but would be nicer in base
> > > > ========================================================
> > > >
> > > > * gstat-rs https://crates.io/crates/gstat
> > > >
> > > > * geom-exporter (I've started this, but haven't published it)
> > > >
> > > > * nfs-exporter https://crates.io/crates/freebsd-nfs-exporter
> > > >
> > > > * virtiofsd-rs .  Nobody has yet tried to port it to FreeBSD.  But if
> > the
> > > >   connection to bhyve(8) is too intimate, it might be hard to do in
> > ports.
> > > >   https://gitlab.com/virtio-fs/virtiofsd
> > > >
> > > > * jail-exporter https://crates.io/crates/jail_exporter
> > > >
> > > > * Various jail managers have been attempted in Rust.  I think these
> > are fine in
> > > >   ports, but others like Goran Mekic have opined that they should be
> > moved to
> > > >   base instead.
> > > >
> > > > * musikid's pjdfstest rewrite.  I think it would be great to start
> > using this
> > > >   to test the base system's file systems.  If the tests themselves
> > lived in
> > > >   base, they would be easier to sync with file system development.
> > > >   https://github.com/musikid/pjdfstest
> > > >
> > > > * pf-rs.  I suspect that the API isn't very stable.
> > > >   https://crates.io/crates/pf-rs
> > > >
> > > > * benchpmc.  The pmc counter names changes between releases.
> > > >   https://crates.io/crates/benchpmc
> > > >
> > > > FreeBSD-related applications that are just fine in ports
> > > > =========================================================
> > > >
> > > > * fsx-rs.  Unlike pjdfstest, this only tests datapath APIs.  Those are
> > usually
> > > >   more stable than control path APIs, so I think there's little to be
> > gained by
> > > >   moving this into base. https://crates.io/crates/fsx
> > > >
> > > > * ztop.  It uses ZFS's kstats sysctl interface, which is pretty stable.
> > > >   https://crates.io/crates/ztop
> > > >
> > > > * iocage-provision  https://crates.io/crates/iocage-provision
> > > >
> > > > * rsblk https://crates.io/crates/rsblk
> > > >
> > > > * xfuse  https://github.com/KhaledEmaraDev/xfuse
> > > >
> > > > Other FreeBSD-related libraries in Rust
> > > > =======================================
> > > > Just see the list at https://crates.io/keywords/freebsd
> > > >
> > >
> > > One new data point: DARPA is looking to rewrite a significant amount
> > > of C code to Rust with their "Translating All C to Rust (TRACTOR)"
> > > project:
> > > https://sam.gov/opp/1e45d648886b4e9ca91890285af77eb7/view
> >
> > Interesting.  And since you bring it up, I have two new data points myself:
> >
> > * ctld: while working on some bugs in ctld, I had trouble
> > understanding the config file parsing.  So I rewrote that part in
> > Rust, just to help my understanding.  Later, I rewrote the XML
> > parsing, too.  Then I rewrote the LUN creation and deletion, just to
> > see how hard it would be.  All of those parts take about 5x fewer SLOC
> > in Rust than in C, and they're less buggy, too.  Config file parsing
> > is more consistent, no memory leaks, etc.  Alas, I'm not planning to
> > finish this project, since the base system doesn't allow Rust and ctld
> > is too tightly coupled to ctl to live in ports.
> >
> 
> Cool. Still waiting for anybody to take me up on the offer to do build
> system integration. Since the Rust advocates can't get even this basic step
> done for review, it's going to be impossible to have Rust in the base. This
> isn't even integrate rust compiler like we do with llvm, but with external
> Rust toolchain.
> 
> Until somebody steps up for this task, the status quo can't possibly change.

Back at the FreeBSD Developer Summit at this last BSDCan, there was
interest in supporting optional external toolchains in the src build
framework. You had mentioned you would be happy to mentor someone, but
not do the nitty gritty yourself.

I could carve off some time in September to be the primary developer,
doing the nitty gritty work. Would you be comfortable answering my
questions, should I have any?

Also: what work (or research), if any, has been done on the concept of
external toolchain support for optional components in the FreeBSD
source tree? Am I starting afresh or building upon existing work?

Thanks,

-- 
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50
https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc