adding new flua libraries

From: Mark Johnston <markj_at_freebsd.org>
Date: Thu, 05 Sep 2024 22:51:53 UTC
FreeBSD ships a Lua 5.4 implementation, flua, in the base system.  It's
intended for use by the base system, and so far has a few consumers, but
not many so far.  (As an aside, flua's intended scope is not totally
clear to me.  Is it only for use by the base system?  What compatibility
guarantees does it provide, if any?)

A few flua modules wrapping FreeBSD libraries (e.g., libjail) are
available, but they don't provide enough to make flua useful as a
general purpose programming tool.  It lacks interfaces for interacting
with the system (e.g., libc/libsys/libutil/etc wrappers) as well as
standard programming facilities (e.g., classes, higher-order functions,
etc.).  Here I'm mostly interested in discussing the former.

I think flua could be a very useful alternative to shell scripts and C
code where performance is not critical.  It's very good at wrapping C
interfaces and thus could be used to make FreeBSD features (jails,
bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who
don't want to interact with C.

It's a lot of work to build up a set of flua modules that provide enough
functionality to be generally useful.  My feeling is that the easiest
way to get there is to wrap C interfaces directly (to the extent that
that's possible, but it's usually easy) and expose them in flua as a
stable interface.  Libraries written in pure Lua (or other languages
that interoperate well with Lua) could be built on top of that.

I'm inclined to start by wrapping libc and libsys interfaces in a manner
similar to luaposix.  There, the namespace is partitioned by C headers,
so posix.unistd contains identifiers from unistd.h, posix.sys.stat
contains identifiers from sys/stat.h, and so on.  In fact, flua already
has a small subset of luaposix's functionality.  Wrapping C interfaces
isn't much fun, but it's easy, and it saves us having to come up with
names and interfaces (naming things is hard), and helps ensure that the
glue code is relatively small and doesn't impose a large testing burden.
Moreover, C interfaces don't change much and are subject to
well-understood compatibility constraints, which should mean that Lua
interfaces built on top of them are subject to the same constraints.

Assuming there's general agreement on that approach, the question I'd
really like to answer is, what should the namespace look like?  It would
be useful to provide a posix module compatible with luaposix, but that
obviously won't contain FreeBSD-specific functionality.

I propose having a top-level "freebsd" namespace for all modules
implemented in the base system, excluding posix and contrib libraries
which already define a Lua interface (libucl is the one example I see so
far; we could import sqlite bindings as well).  Then, if we follow
luaposix's convention, we could have freebsd.sys.capsicum.* for
Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent
wrappers, and so on.  The posix module could simply provide a subset of
freebsd.*.

Maybe it's better to separate C wrappers from native Lua modules though,
so it could be better to have freebsd.c.sys.capsicum.*, etc., and
provide higher-level interfaces for FreeBSD features under freebsd.pf,
freebsd.zfs, freebsd.jail, etc..  I'm not really sure.

In the interest of prompting discussion a bit, I posted some patches to
add some example wrappers to flua:
https://reviews.freebsd.org/D46553
https://reviews.freebsd.org/D46554
https://reviews.freebsd.org/D46556

Does anyone have opinions on anything I've brought up above?  I'm pretty
happy to write a lot of this glue code or find ways to automate it, as
I've already done a fair bit of it, but it's hard to make progress
without having some rigourous conventions for the flua namespace (again,
naming things is hard).