From nobody Fri Sep 06 14:51:45 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0fM656BRz5VF0l for ; Fri, 06 Sep 2024 14:51:46 +0000 (UTC) (envelope-from kevans@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0fM64Pz6z4ckJ; Fri, 6 Sep 2024 14:51:46 +0000 (UTC) (envelope-from kevans@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725634306; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ff9BiinzVZJnuGwZLJJqw6VsoNsirSxOWy+IEHIsOSk=; b=MjA2PeWdFBPHeFTay/bhVo9OCsN7No+d+hHl1cGgv2RzMo/qGCSDe3W1+6yP7egQmgpqel Dkk9nqYZJpksNqJlpZmHGD8Ih4u3KC9TDNAkbQkQdYAFEBpDn4yj5ZvAj3yHYuOT/iT06B ttrbWJDfofaOM4VgjsNEKINDDl/4+q9xaYrqzH7KDg3FvKo/pyc2j1PJK3wtYC5NVhM11Z W118EyqRmo9WGQrSqxmJvpvEwCPlM96YAIQ+xnSDFESb7/H6sLM9+7daFu7vZrYntFcJc+ uCBVIelZirKF0dTt/M2/8MuzL59juzSm6fcU8w+aI1v7cTk2Z0iDFr5LUJvbdg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725634306; a=rsa-sha256; cv=none; b=rO/E/zMupzZajAxoFJTz7UV3Lz/gqNFQDX5Lm1EDrLXunrQHxX1J2Qn4taEMhbnxj3WGaq ha3mCiYqFhU17E6QujbIizZ7c0LXRk5QM2DZMD/JAxfxikRbenVzv6WLNt4hjgndNB+Qdn s6aZxyXumV/KaHXcVgtt/YDdsV6ScBfk//u0TXZHeCso62WbxFUTZnMfW+avzb9wSm5r2N kYrr8Ov32n1jtEvkbhpGvmJkKgxJOctMydTBIXUyjqs0k01QvemYk0Fp0UitpbgbAwKRdV QjKU5R5sFNW/4rMFPkCkNbFNTAXiealgsOilxJUv0TU/A1wuXclvYcE6Q2ibQQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725634306; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ff9BiinzVZJnuGwZLJJqw6VsoNsirSxOWy+IEHIsOSk=; b=wm2ZLcmiiW6wB6c3QgLvZ96T2yesiKIWU4AcxQKMbBFLAAlMxeRyn+nKtM+3NCv7gPQNYn bRt+O7p95UtLUHY//rctT7LXRKkl0Kdu8QxZp3vd1h/z+laM4rgZjb98T++0C/EprREE9Y mYglWUwReozZ5mSV4iP2WCp4fVYVVy5MVOy9qKIOgRD2WQThnGXGGb7eqpfdl549LbEzh6 1w4HRYv9BuVrIqq50I49ojgNoqk6twp6DI/0L2LA+bbnWaB7WSPCTJ5CHn4s1g/fZ9sn4H YaagRWnotAoWnD0y75uJmBjSZqMbJPEDi8wAGfoW83bKQ7eq2hemoh8Rruu9lA== Received: from [10.9.4.95] (unknown [209.182.120.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) (Authenticated sender: kevans/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4X0fM60hgczQkP; Fri, 6 Sep 2024 14:51:46 +0000 (UTC) (envelope-from kevans@FreeBSD.org) Message-ID: <4b06cc12-104a-4ee5-bc92-d9ccec18fe3f@FreeBSD.org> Date: Fri, 6 Sep 2024 09:51:45 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: adding new flua libraries To: Warner Losh Cc: Baptiste Daroussin , Mark Johnston , freebsd-hackers@freebsd.org, imp@freebsd.org References: <2f3039bc-883c-4b11-ab20-d1873c6cc555@FreeBSD.org> Content-Language: en-US From: Kyle Evans In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 9/6/24 09:48, Warner Losh wrote: > > > On Fri, Sep 6, 2024 at 8:45 AM Kyle Evans > wrote: > > On 9/6/24 04:29, Baptiste Daroussin wrote: > > On Thu 05 Sep 18:51, Mark Johnston wrote: > >> FreeBSD ships a Lua 5.4 implementation, flua, in the base > system.  It's > >> intended for use by the base system, and so far has a few > consumers, but > >> not many so far.  (As an aside, flua's intended scope is not totally > >> clear to me.  Is it only for use by the base system?  What > compatibility > >> guarantees does it provide, if any?) > >> > >> A few flua modules wrapping FreeBSD libraries (e.g., libjail) are > >> available, but they don't provide enough to make flua useful as a > >> general purpose programming tool.  It lacks interfaces for > interacting > >> with the system (e.g., libc/libsys/libutil/etc wrappers) as well as > >> standard programming facilities (e.g., classes, higher-order > functions, > >> etc.).  Here I'm mostly interested in discussing the former. > >> > >> I think flua could be a very useful alternative to shell scripts > and C > >> code where performance is not critical.  It's very good at > wrapping C > >> interfaces and thus could be used to make FreeBSD features (jails, > >> bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who > >> don't want to interact with C. > >> > >> It's a lot of work to build up a set of flua modules that > provide enough > >> functionality to be generally useful.  My feeling is that the > easiest > >> way to get there is to wrap C interfaces directly (to the extent > that > >> that's possible, but it's usually easy) and expose them in flua as a > >> stable interface.  Libraries written in pure Lua (or other languages > >> that interoperate well with Lua) could be built on top of that. > >> > >> I'm inclined to start by wrapping libc and libsys interfaces in > a manner > >> similar to luaposix.  There, the namespace is partitioned by C > headers, > >> so posix.unistd contains identifiers from unistd.h, posix.sys.stat > >> contains identifiers from sys/stat.h, and so on.  In fact, flua > already > >> has a small subset of luaposix's functionality.  Wrapping C > interfaces > >> isn't much fun, but it's easy, and it saves us having to come up > with > >> names and interfaces (naming things is hard), and helps ensure > that the > >> glue code is relatively small and doesn't impose a large testing > burden. > >> Moreover, C interfaces don't change much and are subject to > >> well-understood compatibility constraints, which should mean > that Lua > >> interfaces built on top of them are subject to the same constraints. > >> > >> Assuming there's general agreement on that approach, the > question I'd > >> really like to answer is, what should the namespace look like? > It would > >> be useful to provide a posix module compatible with luaposix, > but that > >> obviously won't contain FreeBSD-specific functionality. > >> > >> I propose having a top-level "freebsd" namespace for all modules > >> implemented in the base system, excluding posix and contrib > libraries > >> which already define a Lua interface (libucl is the one example > I see so > >> far; we could import sqlite bindings as well).  Then, if we follow > >> luaposix's convention, we could have freebsd.sys.capsicum.* for > >> Capsicum-related syscalls and constants, freebsd.sys.event.* for > kevent > >> wrappers, and so on.  The posix module could simply provide a > subset of > >> freebsd.*. > >> > >> Maybe it's better to separate C wrappers from native Lua modules > though, > >> so it could be better to have freebsd.c.sys.capsicum.*, etc., and > >> provide higher-level interfaces for FreeBSD features under > freebsd.pf , > >> freebsd.zfs, freebsd.jail, etc..  I'm not really sure. > >> > >> In the interest of prompting discussion a bit, I posted some > patches to > >> add some example wrappers to flua: > >> https://reviews.freebsd.org/D46553 > > >> https://reviews.freebsd.org/D46554 > > >> https://reviews.freebsd.org/D46556 > > >> > >> Does anyone have opinions on anything I've brought up above? > I'm pretty > >> happy to write a lot of this glue code or find ways to automate > it, as > >> I've already done a fair bit of it, but it's hard to make progress > >> without having some rigourous conventions for the flua namespace > (again, > >> naming things is hard). > > > > I like all of what I read and I am fully aligned. > > > > What I don't see here, is I think we should have dynamic modules > libucl, fbsd & > > friends are not dynamic and the reviews I see for the freebsd > module reviews, > > this is still bundled. > > > > Right, this is an artifact of how flua's evolved that we need to move > away from.  We didn't have module support at the beginning- that came > later, but we didn't really move things out into modules.  We still > can't move some things because of the bootstrap flua problem I > mentioned > elsewhere (doing so would block work to do `make sysent` type work as > part of the build as it would break cross-builds), but at least > libucl/fbsd would be fine. > > > Yea, if the sysent.lua code doesn't use modules, would we still have the > same > bootstrapping issues? Today, I don't think here's anything but vanilla > lua code > in there. > Right- if it didn't we'd have no problem, but the in-tree version uses lposix to construct a tmpdir to throw the intermediate files in while we're building them. I suppose the new model would more likely generate them in-place in .OBJDIR instead and could avoid that.