From nobody Fri Sep 06 14:48:59 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0fJ84wpsz5VDwq for ; Fri, 06 Sep 2024 14:49:12 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0fJ83S7Sz4bw7 for ; Fri, 6 Sep 2024 14:49:12 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-2da55ea8163so1594877a91.1 for ; Fri, 06 Sep 2024 07:49:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1725634151; x=1726238951; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=UnQoLlCQVcdA3krTyWMnqt89ADwgqV5U25SGKWmv7Bs=; b=WWkRv/+hV6Bup9UzZjrMVE+hl9vCkOr9A3d32pk+PCba0Ka73arw2yaDViWSIeIS7A QLLuRY8ZdDl299N9USZgPulVwV4ki2QvLrV3x8DaWSONhiFeAgeSsi1YfXIiLWv7xrPj DXU8wFuxAE3OsSEJ++aBVtAkPOJjmJME9G6A9SeITDp96mJiV1uR0jT3w1A+dN3rIIA4 coNNZsXkHqRc6hWZamjplOWI4P+1PJo2y2C8XktPMhSXR+GU1XKDax8RNcHRoANZ92nu wh/Zt3hUa9pJub/LeR7RvQ4YMkT/ksg9G2UAuMgQCMxjNH8zEbCpEDW8E42tlo/lKaDN 3BwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725634151; x=1726238951; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=UnQoLlCQVcdA3krTyWMnqt89ADwgqV5U25SGKWmv7Bs=; b=qUwhRaMPKMQ1yebFVgglDJfOP3J6JpB8affSqINAlxdYSp4FlDNts8X8/h+XbCx/hJ mauKJo0vMKDCzGW39IWlBDFhn+csty0yfWIjDlvBkyv1kvHC9OGh7fAf00OSDVEYlDtA jdTun9pUzUp5+6rW2+q4wWg1iunImNa053IflHjK6o8K0yICmCGyuMoCXseMoMNnAmWH 0+Pojn00owZyhU+mjmL5tpHJA1NreCxq5QUJYxqm5VAphc80W7svHU/OZLKrsY0QAtGv OBXs+Ah1LGuhxW3ggEy4ktDfDBmSTRA3J3u36UAng5jbxu+258ukSIweOB0vs3hVWgZd bMsA== X-Forwarded-Encrypted: i=1; AJvYcCXbV3eI9PtcwlApcu32qpB799N2BC1SINkLUss9yfS1Em1GDSU0qUK3/xXq0XyIh5xQPYRljxCeF3dboNqnfAo=@freebsd.org X-Gm-Message-State: AOJu0YyFr5e1cEgPyXt9dMapkW69rBfMPpr5wLy0/0hXZRGUcBxpEUMw nTOsxeraY9JKtZfn2YJYpBTwh+iU/Fk8kv+adoAFxnZMDBo4QeXvKNscQ7lYgs8QUxqFqOKTZXO v8s7lbcuMAsQ649xLUKWhbh9ydb3zwaGV0ucp7NY3L/fSAmxx X-Google-Smtp-Source: AGHT+IH+yQ2SL51hMow0n76Df03dmT3UADOiqscixwj3dcj6JSZx9kRWDY+s104DieB8hDpwmgvGyneLmVwqG0ccZC8= X-Received: by 2002:a17:90a:304a:b0:2d8:82a2:b093 with SMTP id 98e67ed59e1d1-2dad50d297cmr3386458a91.13.1725634150916; Fri, 06 Sep 2024 07:49:10 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 References: <2f3039bc-883c-4b11-ab20-d1873c6cc555@FreeBSD.org> In-Reply-To: <2f3039bc-883c-4b11-ab20-d1873c6cc555@FreeBSD.org> From: Warner Losh Date: Fri, 6 Sep 2024 08:48:59 -0600 Message-ID: Subject: Re: adding new flua libraries To: Kyle Evans Cc: Baptiste Daroussin , Mark Johnston , freebsd-hackers@freebsd.org, imp@freebsd.org Content-Type: multipart/alternative; boundary="00000000000094008c0621748294" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4X0fJ83S7Sz4bw7 --00000000000094008c0621748294 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Sep 6, 2024 at 8:45=E2=80=AFAM Kyle Evans wrot= e: > On 9/6/24 04:29, Baptiste Daroussin wrote: > > On Thu 05 Sep 18:51, Mark Johnston wrote: > >> FreeBSD ships a Lua 5.4 implementation, flua, in the base system. It'= s > >> intended for use by the base system, and so far has a few consumers, b= ut > >> not many so far. (As an aside, flua's intended scope is not totally > >> clear to me. Is it only for use by the base system? What compatibili= ty > >> guarantees does it provide, if any?) > >> > >> A few flua modules wrapping FreeBSD libraries (e.g., libjail) are > >> available, but they don't provide enough to make flua useful as a > >> general purpose programming tool. It lacks interfaces for interacting > >> with the system (e.g., libc/libsys/libutil/etc wrappers) as well as > >> standard programming facilities (e.g., classes, higher-order functions= , > >> etc.). Here I'm mostly interested in discussing the former. > >> > >> I think flua could be a very useful alternative to shell scripts and C > >> code where performance is not critical. It's very good at wrapping C > >> interfaces and thus could be used to make FreeBSD features (jails, > >> bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who > >> don't want to interact with C. > >> > >> It's a lot of work to build up a set of flua modules that provide enou= gh > >> functionality to be generally useful. My feeling is that the easiest > >> way to get there is to wrap C interfaces directly (to the extent that > >> that's possible, but it's usually easy) and expose them in flua as a > >> stable interface. Libraries written in pure Lua (or other languages > >> that interoperate well with Lua) could be built on top of that. > >> > >> I'm inclined to start by wrapping libc and libsys interfaces in a mann= er > >> similar to luaposix. There, the namespace is partitioned by C headers= , > >> so posix.unistd contains identifiers from unistd.h, posix.sys.stat > >> contains identifiers from sys/stat.h, and so on. In fact, flua alread= y > >> has a small subset of luaposix's functionality. Wrapping C interfaces > >> isn't much fun, but it's easy, and it saves us having to come up with > >> names and interfaces (naming things is hard), and helps ensure that th= e > >> glue code is relatively small and doesn't impose a large testing burde= n. > >> Moreover, C interfaces don't change much and are subject to > >> well-understood compatibility constraints, which should mean that Lua > >> interfaces built on top of them are subject to the same constraints. > >> > >> Assuming there's general agreement on that approach, the question I'd > >> really like to answer is, what should the namespace look like? It wou= ld > >> be useful to provide a posix module compatible with luaposix, but that > >> obviously won't contain FreeBSD-specific functionality. > >> > >> I propose having a top-level "freebsd" namespace for all modules > >> implemented in the base system, excluding posix and contrib libraries > >> which already define a Lua interface (libucl is the one example I see = so > >> far; we could import sqlite bindings as well). Then, if we follow > >> luaposix's convention, we could have freebsd.sys.capsicum.* for > >> Capsicum-related syscalls and constants, freebsd.sys.event.* for keven= t > >> wrappers, and so on. The posix module could simply provide a subset o= f > >> freebsd.*. > >> > >> Maybe it's better to separate C wrappers from native Lua modules thoug= h, > >> so it could be better to have freebsd.c.sys.capsicum.*, etc., and > >> provide higher-level interfaces for FreeBSD features under freebsd.pf, > >> freebsd.zfs, freebsd.jail, etc.. I'm not really sure. > >> > >> In the interest of prompting discussion a bit, I posted some patches t= o > >> add some example wrappers to flua: > >> https://reviews.freebsd.org/D46553 > >> https://reviews.freebsd.org/D46554 > >> https://reviews.freebsd.org/D46556 > >> > >> Does anyone have opinions on anything I've brought up above? I'm pret= ty > >> happy to write a lot of this glue code or find ways to automate it, as > >> I've already done a fair bit of it, but it's hard to make progress > >> without having some rigourous conventions for the flua namespace (agai= n, > >> naming things is hard). > > > > I like all of what I read and I am fully aligned. > > > > What I don't see here, is I think we should have dynamic modules libucl= , > fbsd & > > friends are not dynamic and the reviews I see for the freebsd module > reviews, > > this is still bundled. > > > > Right, this is an artifact of how flua's evolved that we need to move > away from. We didn't have module support at the beginning- that came > later, but we didn't really move things out into modules. We still > can't move some things because of the bootstrap flua problem I mentioned > elsewhere (doing so would block work to do `make sysent` type work as > part of the build as it would break cross-builds), but at least > libucl/fbsd would be fine. > Yea, if the sysent.lua code doesn't use modules, would we still have the same bootstrapping issues? Today, I don't think here's anything but vanilla lua code in there. Warner > > my proposal for unbundling, I need kldload for a project of mine, so I > did: > > https://reviews.freebsd.org/D46558 > > > > Best regards, > > Bapt > > --00000000000094008c0621748294 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Fri, Sep 6, 2024 at 8:45=E2=80=AFA= M Kyle Evans <kevans@freebsd.org> wrote:
On= 9/6/24 04:29, Baptiste Daroussin wrote:
> On Thu 05 Sep 18:51, Mark Johnston wrote:
>> FreeBSD ships a Lua 5.4 implementation, flua, in the base system.= =C2=A0 It's
>> intended for use by the base system, and so far has a few consumer= s, but
>> not many so far.=C2=A0 (As an aside, flua's intended scope is = not totally
>> clear to me.=C2=A0 Is it only for use by the base system?=C2=A0 Wh= at compatibility
>> guarantees does it provide, if any?)
>>
>> A few flua modules wrapping FreeBSD libraries (e.g., libjail) are<= br> >> available, but they don't provide enough to make flua useful a= s a
>> general purpose programming tool.=C2=A0 It lacks interfaces for in= teracting
>> with the system (e.g., libc/libsys/libutil/etc wrappers) as well a= s
>> standard programming facilities (e.g., classes, higher-order funct= ions,
>> etc.).=C2=A0 Here I'm mostly interested in discussing the form= er.
>>
>> I think flua could be a very useful alternative to shell scripts a= nd C
>> code where performance is not critical.=C2=A0 It's very good a= t wrapping C
>> interfaces and thus could be used to make FreeBSD features (jails,=
>> bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers wh= o
>> don't want to interact with C.
>>
>> It's a lot of work to build up a set of flua modules that prov= ide enough
>> functionality to be generally useful.=C2=A0 My feeling is that the= easiest
>> way to get there is to wrap C interfaces directly (to the extent t= hat
>> that's possible, but it's usually easy) and expose them in= flua as a
>> stable interface.=C2=A0 Libraries written in pure Lua (or other la= nguages
>> that interoperate well with Lua) could be built on top of that. >>
>> I'm inclined to start by wrapping libc and libsys interfaces i= n a manner
>> similar to luaposix.=C2=A0 There, the namespace is partitioned by = C headers,
>> so posix.unistd contains identifiers from unistd.h, posix.sys.stat=
>> contains identifiers from sys/stat.h, and so on.=C2=A0 In fact, fl= ua already
>> has a small subset of luaposix's functionality.=C2=A0 Wrapping= C interfaces
>> isn't much fun, but it's easy, and it saves us having to c= ome up with
>> names and interfaces (naming things is hard), and helps ensure tha= t the
>> glue code is relatively small and doesn't impose a large testi= ng burden.
>> Moreover, C interfaces don't change much and are subject to >> well-understood compatibility constraints, which should mean that = Lua
>> interfaces built on top of them are subject to the same constraint= s.
>>
>> Assuming there's general agreement on that approach, the quest= ion I'd
>> really like to answer is, what should the namespace look like?=C2= =A0 It would
>> be useful to provide a posix module compatible with luaposix, but = that
>> obviously won't contain FreeBSD-specific functionality.
>>
>> I propose having a top-level "freebsd" namespace for all= modules
>> implemented in the base system, excluding posix and contrib librar= ies
>> which already define a Lua interface (libucl is the one example I = see so
>> far; we could import sqlite bindings as well).=C2=A0 Then, if we f= ollow
>> luaposix's convention, we could have freebsd.sys.capsicum.* fo= r
>> Capsicum-related syscalls and constants, freebsd.sys.event.* for k= event
>> wrappers, and so on.=C2=A0 The posix module could simply provide a= subset of
>> freebsd.*.
>>
>> Maybe it's better to separate C wrappers from native Lua modul= es though,
>> so it could be better to have freebsd.c.sys.capsicum.*, etc., and<= br> >> provide higher-level interfaces for FreeBSD features under freebsd.pf,=
>> freebsd.zfs, freebsd.jail, etc..=C2=A0 I'm not really sure. >>
>> In the interest of prompting discussion a bit, I posted some patch= es to
>> add some example wrappers to flua:
>> https://reviews.freebsd.org/D46553
>> https://reviews.freebsd.org/D46554
>> https://reviews.freebsd.org/D46556
>>
>> Does anyone have opinions on anything I've brought up above?= =C2=A0 I'm pretty
>> happy to write a lot of this glue code or find ways to automate it= , as
>> I've already done a fair bit of it, but it's hard to make = progress
>> without having some rigourous conventions for the flua namespace (= again,
>> naming things is hard).
>
> I like all of what I read and I am fully aligned.
>
> What I don't see here, is I think we should have dynamic modules l= ibucl, fbsd &
> friends are not dynamic and the reviews I see for the freebsd module r= eviews,
> this is still bundled.
>

Right, this is an artifact of how flua's evolved that we need to move <= br> away from.=C2=A0 We didn't have module support at the beginning- that c= ame
later, but we didn't really move things out into modules.=C2=A0 We stil= l
can't move some things because of the bootstrap flua problem I mentione= d
elsewhere (doing so would block work to do `make sysent` type work as
part of the build as it would break cross-builds), but at least
libucl/fbsd would be fine.

Yea, if the = sysent.lua code doesn't use modules, would we still have the same
=
bootstrapping issues? Today, I don't think here's anything but= vanilla lua code
in there.

Warner
=
=C2=A0
> my proposal for unbundling, I need kldload for a project of mine, so I= did:
> https://reviews.freebsd.org/D46558
>
> Best regards,
> Bapt

--00000000000094008c0621748294--