Re: adding new flua libraries

On 9/5/24 17:51, Mark Johnston wrote:
> FreeBSD ships a Lua 5.4 implementation, flua, in the base system.  It's
> intended for use by the base system, and so far has a few consumers, but
> not many so far.  (As an aside, flua's intended scope is not totally
> clear to me.  Is it only for use by the base system?  What compatibility
> guarantees does it provide, if any?)
> 

Only by the base system (hence the rename and hiding it in 
/usr/libexec), no guarantees except for modules we've re-implemented 
subsets of-  if a conflicting name exists in ports, its use must be 
compatible with that.

> A few flua modules wrapping FreeBSD libraries (e.g., libjail) are
> available, but they don't provide enough to make flua useful as a
> general purpose programming tool.  It lacks interfaces for interacting
> with the system (e.g., libc/libsys/libutil/etc wrappers) as well as
> standard programming facilities (e.g., classes, higher-order functions,
> etc.).  Here I'm mostly interested in discussing the former.
> 
> I think flua could be a very useful alternative to shell scripts and C
> code where performance is not critical.  It's very good at wrapping C
> interfaces and thus could be used to make FreeBSD features (jails,
> bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who
> don't want to interact with C.
> 
> It's a lot of work to build up a set of flua modules that provide enough
> functionality to be generally useful.  My feeling is that the easiest
> way to get there is to wrap C interfaces directly (to the extent that
> that's possible, but it's usually easy) and expose them in flua as a
> stable interface.  Libraries written in pure Lua (or other languages
> that interoperate well with Lua) could be built on top of that.
> 
> I'm inclined to start by wrapping libc and libsys interfaces in a manner
> similar to luaposix.  There, the namespace is partitioned by C headers,
> so posix.unistd contains identifiers from unistd.h, posix.sys.stat
> contains identifiers from sys/stat.h, and so on.  In fact, flua already
> has a small subset of luaposix's functionality.  Wrapping C interfaces
> isn't much fun, but it's easy, and it saves us having to come up with
> names and interfaces (naming things is hard), and helps ensure that the
> glue code is relatively small and doesn't impose a large testing burden.
> Moreover, C interfaces don't change much and are subject to
> well-understood compatibility constraints, which should mean that Lua
> interfaces built on top of them are subject to the same constraints.
> 
> Assuming there's general agreement on that approach, the question I'd
> really like to answer is, what should the namespace look like?  It would
> be useful to provide a posix module compatible with luaposix, but that
> obviously won't contain FreeBSD-specific functionality.
> 
> I propose having a top-level "freebsd" namespace for all modules
> implemented in the base system, excluding posix and contrib libraries
> which already define a Lua interface (libucl is the one example I see so
> far; we could import sqlite bindings as well).  Then, if we follow
> luaposix's convention, we could have freebsd.sys.capsicum.* for
> Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent
> wrappers, and so on.  The posix module could simply provide a subset of
> freebsd.*.
> 

I think that's fine.  Ideally, we probably just import luaposix 
wholesale now once we get the bootstrap flua module situation sorted and 
we can just re-export those implementations into the freebsd.* namespace 
similar to how we discussed luaposix was doing things today, unless 
those interfaces are weird (I don't remember them being problematic, 
though).

> Maybe it's better to separate C wrappers from native Lua modules though,
> so it could be better to have freebsd.c.sys.capsicum.*, etc., and
> provide higher-level interfaces for FreeBSD features under freebsd.pf,
> freebsd.zfs, freebsd.jail, etc..  I'm not really sure.
> 
> In the interest of prompting discussion a bit, I posted some patches to
> add some example wrappers to flua:
> https://reviews.freebsd.org/D46553
> https://reviews.freebsd.org/D46554
> https://reviews.freebsd.org/D46556
> 

Will take a look when I get a bit of time.

> Does anyone have opinions on anything I've brought up above?  I'm pretty
> happy to write a lot of this glue code or find ways to automate it, as
> I've already done a fair bit of it, but it's hard to make progress
> without having some rigourous conventions for the flua namespace (again,
> naming things is hard).
> 

We should also figure out where to document this stuff (policy, mostly- 
we have a 3lua section for libraries themselves).

Thanks,

Kyle Evans