Re: A Demo of rust-in-base

Hey Alan et al,

I apologize for the silence on my end. It has been a busier two months
than anticipated. In that time, I've been entertaining some thoughts
on Rust in base support. ${LIFE} is starting to calm down again, and I
do believe I'll be able to start the research in time in September. I
will be splitting my free time between this and studying for my OSCP
cert.

So, to those thoughts, in list form (in no particular order):

1. Use of Rust compiler toolchain support will be for userland
   components in an opt-in fashion. Meaning, all userland components
   written in Rust will be optional.
2. It does not make sense to perform a vendor import of the Rust
   compiler toolchain and standard libraries. All Rust code in the src
   tree must be built from an external toolchain.
3. I believe the notion of an external toolchain could be abstracted
   such that we can support any optional userland component written in
   a language supported by that external toolchain. This would imply
   that other alternative languages could be supported with minimal
   work (Zig, TypeScript, Python, Java, etc.)
4. We could provide auto-detection mechanisms for determining which
   external toolchains are available, their language support, etc. The
   initial proof-of-concept would likely be limited to Rust to save on
   time and complexity.
5. As the work matures, and perhaps as a requisite for eventual
   inclusion, we could land support for more than Rust. This might be
   a step too far, but hey, it's one of the thoughts I had.
6. So all of this wrapped up means that:
   A. This is NOT a call to rewrite everything in Rust. This work will
      only permit NEW, OPTIONAL components to be written.
   B. Other languages/toolchains/ecosystems could be supported, not
      just Rust.
   C. Initial focus is on userland components. Rust in the kernel is
      out of scope for this initial proof-of-concept.
   D. I would like to see Rust in the kernel. That would be a good
      next area of focus once userland support reaches some level of
      maturity.

My first goal will be to get a better understanding of
src.git/Makefile and src.git/Makefile.inc1. As I study that, I'll also
study your work, Alan. I really appreciate the time you have taken. I
might reach out to you and Warner directly for further questions.

Thanks,

-- 
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50
https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc

On Sun, Aug 04, 2024 at 11:55:26AM UTC, Alan Somers wrote:
> Due to all of the recent discussion of using Rust for code in the
> FreeBSD base, I've put together a demo of what it might look like.  It
> demonstrates:
> 
> * Interspersing Rust crates through the tree (usr.bin/nfs-exporter,
> cddl/usr.bin/ztop, etc) rather than in some special directory.
> * Build integration for all Rust crates.  You can build them all with
> a single "cargo build" command from the top level, and test them all
> with a single "cargo test".
> * Wholly new programs written from scratch in Rust (ztop plus three
> Prometheus exporters)
> * Old programs rewritten in Rust with substantial new features (gstat and fsx)
> * Libs (freebsd-libgeom and freebsd-libgeom-sys)
> * Commits that reconcile the dependencies of multiple crates, so as to
> minimize duplicate dependency versions (5764fb383d4 and 1edf2e19e50)
> * Vendoring all dependencies, direct and transitive, to ensure
> internet-independent and reproducible builds (37ef9ffb6a6).  This
> process is automated and requires almost no manual effort.  Note:
> don't panic if you look in the "vendor" directory and see a bunch of
> crates with "windows" in the name.  They're all just empty stubs.
> * All Rust object files get stored in the "target" directory rather
> than /usr/obj.  Today, if you want them to be stored in /usr/obj the
> best way is to use a symlink, though there's WIP to add
> MAKEOBJDIRPREFIX-like functionality to Cargo.
> 
> It does NOT demonstrate:
> 
> * Integrating the Rust build system with Make.  Warner has some ideas
> about how to do that.
> * Pulling rustc into contrib.  This tree requires an external Rust toolchain.
> * Building any cdylib libraries with Rust.  That's useful if you want
> a C program to call a Rust library, but I don't have any good examples
> for it.
> * kernel modules.  As already discussed, those are hard.
> * Any Rust crates that involve private APIs, like CTL stuff.  Those
> are among the most tantalizing programs to move from ports to base,
> but nobody's written any yet, because Rust-in-base doesn't exist yet.
> 
> Also, I want to address a question that's popped up a few times:
> backwards-compatibility.  There is a fear that Rust code needs to be
> updated for each new toolchain release.  But that's not true.  It
> hasn't been true for most crates since Rust 1.0 was released about a
> decade ago.  A few exotic crates required "nightly" features after
> that, but they are very few in number these days, and none of them are
> included in this branch's vendored sources.  What Rust _does_ do is it
> releases a new toolchain about every six weeks.  Each new release
> typically includes a few new features in the standard library and they
> often add more compiler warnings, too.  Sometimes they include wholly
> new compiler features, but they are _always_ backwards compatible with
> existing syntax.  Roughly every three years, Rust publishes a new
> "Edition".  Rust Editions are very similar to C++ versions.  i.e. Rust
> 2018 is to Rust 2021 as C++14 is to C++17.  New editions can include
> backwards-incompatible syntax changes, but each crate always knows
> which Edition it uses.  Crates of different Editions can be linked
> together in the same build.  This branch, for example, contains crates
> using Editions 2015, 2018, and 2021.
> 
> If you have any questions about what Rust in Base would look like,
> please examine this branch.  And if you've never used Rust before, I
> highly encourage you to try it.  It really is the best new
> systems-program language for decades.  IMHO, it's the only one that's
> a compelling replacement for C++ in all new applications, and C in
> most.
> 
> https://github.com/asomers/freebsd-src/tree/rust-in-base-demo
>