Re: orch(1) for tty testing

On Tue, Jan 16, 2024 at 6:27 PM Kyle Evans <kevans@freebsd.org> wrote:

> On 1/16/24 13:22, Mark Johnston wrote:
> > On Mon, Jan 15, 2024 at 01:54:54PM -0600, Kyle Evans wrote:
> >> Hi,
> >>
> >> I wrote a tool for tty testing w/ lua scripts, inspired by expect(1) and
> >> with some similarities, but explicitly without the goal of trying to be
> >> compatible:
> >>
> >> https://git.kevans.dev/kevans/orch
> >>
> >> An example of its use is here:
> >>
> >> https://git.kevans.dev/kevans/tty-tests/src/branch/main/test_canon.orch
> >>
> >> Each spawn() creates a new pts(4) and allows us to drive I/O to the
> process
> >> with write/match statements.  I have one or two tests written with it
> so far
> >> and the end result is, IMO, much less painful looking and less error
> prone
> >> than the equivalent in C.
> >>
> >> It still needs a bit of polish (e.g., improved diagnostics when a match
> >> fails), but I'd like to consider importing this somewhere for use in the
> >> near future for tty testing given how much it simplifies.  I can push
> it to
> >> ports, but we have a private liblua in base so it could be built and
> >> banished off into, e.g., /usr/tests/sys/kern/tty/, to make my life just
> a
> >> tad easier re: getting these to run in CI.  It's pretty lightweight; the
> >> orch binary on my amd64 system clocks in at ~27K, and orch.lua at 13K.
> >>
> >> Any objections to importing this new tool into /usr/tests or maybe even
> >> /usr/libexec?
> >
> > This looks quite handy.  I have some lua scripts which I use to checkout
> > a src tree, build a VM image, boot the image and run tests, then
> > download the resulting kyua DB, all with a single CLI command.  I
> > currently run the hypervisor under expect(1) and send commands over a
> > pipe, but that was an expedient hack and I'd rather use something
> > implemented natively.  There are some existing lua expect ports, but
> > orch looks simpler and I'll try it out to see if it's missing anything
> > for my use-case.  More generally, we lack a test framework which lets
> > one boot and interact with VMs, and orch might be a useful component of
> > that.
>
> That seems like a good exercise for it- feel free to get in touch. I
> wouldn't mind porting whatever you've written in expect(1) over and take
> the opportunity to fill in the feature gaps, regardless of whether you
> use the end result or not.
>

Last summer's gsoc student wrote a bunch of Lua to do similar things with
testing boot loaders against a kernel and minimal userland... Would be nice
to have those in-tree and maybe using this.

Warner