Re: adding new flua libraries

On Thu, Sep 5, 2024 at 4:52 PM Mark Johnston <markj@freebsd.org> wrote:

> FreeBSD ships a Lua 5.4 implementation, flua, in the base system.  It's
> intended for use by the base system, and so far has a few consumers, but
> not many so far.  (As an aside, flua's intended scope is not totally
> clear to me.  Is it only for use by the base system?  What compatibility
> guarantees does it provide, if any?)
>

To date, it's only provided for FreeBSD provided things. That means there's
close coordination between what we provide and it. In general, it should
remain
compatible, but I think we've taken some liberties when it wasn't widely
used
to, for example, go from 5.3 to 5.4 w/o providing a fully compatible 5.3
env.


> A few flua modules wrapping FreeBSD libraries (e.g., libjail) are
> available, but they don't provide enough to make flua useful as a
> general purpose programming tool.  It lacks interfaces for interacting
> with the system (e.g., libc/libsys/libutil/etc wrappers) as well as
> standard programming facilities (e.g., classes, higher-order functions,
> etc.).  Here I'm mostly interested in discussing the former.
>

Yea, LUA is a language you can build OO concepts on, but doesn't
directly provide more than the building blocks. Most librairies I've seen
either pick a set of wrappers, or expand the wrappers in-line. We should
pick one: I'm partial to the provide a common set of wrappers model
when the time comes.


> I think flua could be a very useful alternative to shell scripts and C
> code where performance is not critical.  It's very good at wrapping C
> interfaces and thus could be used to make FreeBSD features (jails,
> bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who
> don't want to interact with C.
>

Yes. Last summer's GSoC code to test all the boot loader operations
went great in lua until the very end when the desire to test all 50
combinations in parallel rather than serially (to reduce the run time
from sneaking up on an hour to about a minute) couldn't do it in Lua,
so things were rewritten in Python...


> It's a lot of work to build up a set of flua modules that provide enough
> functionality to be generally useful.  My feeling is that the easiest
> way to get there is to wrap C interfaces directly (to the extent that
> that's possible, but it's usually easy) and expose them in flua as a
> stable interface.  Libraries written in pure Lua (or other languages
> that interoperate well with Lua) could be built on top of that.
>

Yes. I tend to agree, though I think we may want to find a way to
provide some things written in Lua and some in C for these
wrapping.


> I'm inclined to start by wrapping libc and libsys interfaces in a manner
> similar to luaposix.  There, the namespace is partitioned by C headers,
> so posix.unistd contains identifiers from unistd.h, posix.sys.stat
> contains identifiers from sys/stat.h, and so on.  In fact, flua already
> has a small subset of luaposix's functionality.  Wrapping C interfaces
> isn't much fun, but it's easy, and it saves us having to come up with
> names and interfaces (naming things is hard), and helps ensure that the
> glue code is relatively small and doesn't impose a large testing burden.
> Moreover, C interfaces don't change much and are subject to
> well-understood compatibility constraints, which should mean that Lua
> interfaces built on top of them are subject to the same constraints.
>

Generally I agree, though with the caveat that I think posix module
tends to have too-deep a namespace, and I'm hesitant about using
that for FreeBSD, given how much clutter it would add to the lua code.


> Assuming there's general agreement on that approach, the question I'd
> really like to answer is, what should the namespace look like?  It would
> be useful to provide a posix module compatible with luaposix, but that
> obviously won't contain FreeBSD-specific functionality.
>
> I propose having a top-level "freebsd" namespace for all modules
> implemented in the base system, excluding posix and contrib libraries
> which already define a Lua interface (libucl is the one example I see so
> far; we could import sqlite bindings as well).  Then, if we follow
> luaposix's convention, we could have freebsd.sys.capsicum.* for
> Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent
> wrappers, and so on.  The posix module could simply provide a subset of
> freebsd.*.
>

I think that may be too hierarchical and provide little benefit. What would
we have in freebsd.* that would conflict with freebsd.sys.*? I barely see
the
benefit for the posix module, but I think it was a mistake to do it
there... but
we're stuck with the design.

And depending on how compatible we want to be with the posix module,
this may tie our hands a bit in providing more info, or info in a better
format if we try to make it strictly speaking, a subset with the implication
that the behavior is identical.


> Maybe it's better to separate C wrappers from native Lua modules though,
> so it could be better to have freebsd.c.sys.capsicum.*, etc., and
> provide higher-level interfaces for FreeBSD features under freebsd.pf,
> freebsd.zfs, freebsd.jail, etc..  I'm not really sure.
>

I think that's going even further into the weeds. There's no reason
that freebsd.zfs couldn't provide the zfs system calls. freebsd.jail the
jail
ones, etc, in addition to the higher levels. We're not likely going to nest
multiple modules that do jail things on top of a base jail system call-only
module. I'm having trouble seeing the benefit for all the extra clutter
this would bring to the lua code we write.


> In the interest of prompting discussion a bit, I posted some patches to
> add some example wrappers to flua:
> https://reviews.freebsd.org/D46553
> https://reviews.freebsd.org/D46554
> https://reviews.freebsd.org/D46556
>
> Does anyone have opinions on anything I've brought up above?  I'm pretty
> happy to write a lot of this glue code or find ways to automate it, as
> I've already done a fair bit of it, but it's hard to make progress
> without having some rigourous conventions for the flua namespace (again,
> naming things is hard).
>

We should agree on the namespace stuff, then put the conversation in our
rearview mirror. :).

But generally, I love the idea and any comments about how to design "the
crinkly bits on the fjords"
shouldn't be taken to my not liking the fjords.

Warner

with apologies to douglas adams