Re: The Case for Rust (in the base system)

On Sat, Jan 20, 2024, 7:06 PM Tomoaki AOKI <junchoon@dec.sakura.ne.jp>
wrote:

> On Sat, 20 Jan 2024 15:31:23 -0700
> Warner Losh <imp@bsdimp.com> wrote:
>
> > On Sat, Jan 20, 2024 at 11:45 AM Aleksandr Fedorov <wigneddoom@yandex.ru
> >
> > wrote:
> >
> > > What about external dependencies?
> > >
> > >
> https://github.com/Axcient/freebsd-nfs-exporter/blob/master/Cargo.toml#L19
> > > https://github.com/asomers/gstat-rs/blob/master/gstat/src/main.rs#L20
> > >
> > > Is there any plan for which crates we should take into the base system?
> > >
> > > We have had C++ in base for many years, but I don’t see any good
> libraries
> > > for CLI, logging, JSON, etc.
> > >
> > >
> > >
> https://doc.rust-lang.org/rustc/platform-support.html#tier-1-with-host-tools
> > >
> > > Where is the support for Freebsd as a primary platform? ARM, RISC-V,
> > > Power? Should we rewrite devd?
> > >
> > > I think we need to start by providing official repositories (e.g
> > > git.FreeBSD.org/rust.git or git.FreeBSD.org/go.git)
> > > for different languages that include stable bindings to the system API:
> > > - sysctl
> > > - libgeom
> > > - libifconfig
> > > - netgraph
> > > - jail
> > > - etc.
> > >
> > > So that it’s not just some anonymous on crates.io that represents
> these
> > > bindings, but our community.
> > > Officially, with support for a stable ABI for releases, security
> patches,
> > > etc.
> > >
> > > After this, it will be possible to think about which components to
> include
> > > in the base system.
> > >
> > > I would be glad to see a more modern language than C in the database,
> but
> > > I’m afraid that it will be like with C++,
> > > that we will get a couple of daemons and utilities and that’s all.
> > >
> >
> > These are all good questions that need good answers, though necessarily
> to
> > get started.
> >
> > But the other question that occured to me after my last posting was "What
> > about build integration?"
> > How much of the rust automation do we take in vs how much do we drive
> from
> > a future bsd.rust.mk.
> > I can sketch out bsd.rust.mk (to pick an arbitrary name, we'd likely
> need
> > one for what we traditionally
> > think of as libraries (which may or may not map 1:1 onto crates: we could
> > have c callable libraries
> > written in rust in the future, for example) and one for binaries.
> > Initially, though, if we go with the
> > 'make rust tests possible' then we'd likely need the appropriate packages
> > installed for whatever
> > dependencies we'd have in the tests. This would give us a taste for what
> > we'd need to do for
> > base, I'd think. Once we had that notion, I can easily see there needing
> to
> > be some sort of
> > rust bindings for ATF/kyua as one of the first libraries / crates that
> > would test that aspect of
> > the build system. That all would be up to the people writing the tests in
> > rust, I'd imagine.
> >
> > While I could jot out the basics of this integration (so one could just
> add
> > the rust
> > tools to a subdir or subdirs, include the bsd.rust.mk or whatever and
> then
> > it would build
> > if rust is enabled, and would emit a warning it was skipped because rust
> > was disabled).
> > We'd find out if this is workable or not and iterate from there. But that
> > would also require
> > active participation from the rust advocates to make it a reality: I can
> > put together the
> > build infrastructure for the disabled case, but likely can't on my own do
> > the rust enabled
> > case. I'd be happy to work with someone to do that, but I'm not going to
> be
> > able to do
> > that myself: my need for rust is slight, my knowledge of rust is weak,
> etc.
> > Working with
> > someone (or ideally several someones), though it could become reality. So
> > please contact
> > me if you'd like to work on this.
> >
> > Warner
>
> One way to go could be moving programs rewritten with rust to ports.
> There are some programs (not in rust, though) moved to ports, like rcs.
>

Yes. And a number of programs have been made available as ports, but not
too many replace c programs.

Currently, it would not be so realistic, but once we completely switch
> to pkgbase, IIUC, programs in base can sanely depemd on ports programs,
> excluding kernel and fundamental libraries.


Possible, but I'd wager not for quite some time for fundamental things. I
could be wrong on this timeline...

As non-rust consumers of graphics/librsvg2-rust can sanely link with
> it, I assume kmods in ports written in rust can kldload'ed sanely.
> This could be a good starting point.
>

Kernel modules are far off om the horizon at this point. There was a proof
of concept done years ago, but much has changed so an effort is needed to
get back to that basic point.

And would be not all, but test for rust libraries could be implemented
> with C/C++ or any other language suitable, if the rust libraries can
> sanely linked with test codes.
>

That's not a given... ABIs are hard enough without a C to rust layer
sneaking in.

Warner


Am I wrong?
>
>
> > > 20.01.2024, 19:51, "Alan Somers" <asomers@freebsd.org>:
> > >
> > > In a recent thread on src-committers, we discussed the costs and
> > > benefits of including Rust code in the FreeBSD base system. To
> > > summarize, the cost is that it would double our build times. imp
> > > suggested adding an additional step after buildworld for stuff that
> > > requires an external toolchain. That would ease the build time pain.
> > > The benefit is that some tools would become easier to write, or even
> > > become possible. Here is a list of actual and potential Rust projects
> > > that could benefit from being in-tree. If anybody else has items to
> > > add, I suggest moving this into the project wiki:
> > >
> > > Stuff that could only be written in Rust if it were in base
> > > ===========================================================
> > >
> > > * ctl-exporter (I started this, but discovered that the CTL stats API
> is
> > >   unstable, so it can't live in ports. Instead, I had to do it in C).
> > >
> > >
> https://github.com/freebsd/freebsd-src/commit/1a7f22d9c211f504f6c48a86401469181a67ec34
> > >
> > > * fusefs tests. Absolutely impossible to do in C. I considered Rust,
> but
> > > went
> > >   with C++ so they could live in base. They are too closely coupled to
> > >   fusefs(5) to live out-of-tree.
> > >   https://github.com/freebsd/freebsd-src/tree/main/tests/sys/fs/fusefs
> > >
> > > * devd. Currently C++, but imp suggested a rewrite.
> > >   https://github.com/freebsd/freebsd-src/tree/main/sbin/devd
> > >
> > > * zfsd. Currently C++, but I've long pondered a rewrite. Using Rust
> would
> > >   make it more testable.
> > >   https://github.com/freebsd/freebsd-src/tree/main/cddl/usr.sbin/zfsd
> > >
> > > * nscd. Currently C, but confusing and with no test coverage. I've
> > >   contemplated a rewrite myself, but I don't want to do it in C.
> > >   https://github.com/freebsd/freebsd-src/tree/main/usr.sbin/nscd
> > >
> > > * The userland portion of the 802.11ac and Lightning stacks. scottl
> > > suggested
> > >   that these were good candidates for Rust.
> > >
> > > * freebsd-kpi-r14-0 . https://crates.io/crates/freebsd-kpi-r14-0
> > >
> > > Stuff that can live in ports, but would be nicer in base
> > > ========================================================
> > >
> > > * gstat-rs https://crates.io/crates/gstat
> > >
> > > * geom-exporter (I've started this, but haven't published it)
> > >
> > > * nfs-exporter https://crates.io/crates/freebsd-nfs-exporter
> > >
> > > * virtiofsd-rs . Nobody has yet tried to port it to FreeBSD. But if the
> > >   connection to bhyve(8) is too intimate, it might be hard to do in
> ports.
> > >   https://gitlab.com/virtio-fs/virtiofsd
> > >
> > > * jail-exporter https://crates.io/crates/jail_exporter
> > >
> > > * Various jail managers have been attempted in Rust. I think these are
> > > fine in
> > >   ports, but others like Goran Mekic have opined that they should be
> moved
> > > to
> > >   base instead.
> > >
> > > * musikid's pjdfstest rewrite. I think it would be great to start using
> > > this
> > >   to test the base system's file systems. If the tests themselves
> lived in
> > >   base, they would be easier to sync with file system development.
> > >   https://github.com/musikid/pjdfstest
> > >
> > > * pf-rs. I suspect that the API isn't very stable.
> > >   https://crates.io/crates/pf-rs
> > >
> > > * benchpmc. The pmc counter names changes between releases.
> > >   https://crates.io/crates/benchpmc
> > >
> > > FreeBSD-related applications that are just fine in ports
> > > =========================================================
> > >
> > > * fsx-rs. Unlike pjdfstest, this only tests datapath APIs. Those are
> > > usually
> > >   more stable than control path APIs, so I think there's little to be
> > > gained by
> > >   moving this into base. https://crates.io/crates/fsx
> > >
> > > * ztop. It uses ZFS's kstats sysctl interface, which is pretty stable.
> > >   https://crates.io/crates/ztop
> > >
> > > * iocage-provision https://crates.io/crates/iocage-provision
> > >
> > > * rsblk https://crates.io/crates/rsblk
> > >
> > > * xfuse https://github.com/KhaledEmaraDev/xfuse
> > >
> > > Other FreeBSD-related libraries in Rust
> > > =======================================
> > > Just see the list at https://crates.io/keywords/freebsd
>
>
> --
> Tomoaki AOKI    <junchoon@dec.sakura.ne.jp>
>