Re: adding new flua libraries

On 9/5/24 19:34, Mark Johnston wrote:
> On Thu, Sep 05, 2024 at 06:11:47PM -0500, Kyle Evans wrote:
>> On 9/5/24 17:51, Mark Johnston wrote:
>>> FreeBSD ships a Lua 5.4 implementation, flua, in the base system.  It's
>>> intended for use by the base system, and so far has a few consumers, but
>>> not many so far.  (As an aside, flua's intended scope is not totally
>>> clear to me.  Is it only for use by the base system?  What compatibility
>>> guarantees does it provide, if any?)
>>>
>>
>> Only by the base system (hence the rename and hiding it in /usr/libexec), no
>> guarantees except for modules we've re-implemented subsets of-  if a
>> conflicting name exists in ports, its use must be compatible with that.
>>
>>> A few flua modules wrapping FreeBSD libraries (e.g., libjail) are
>>> available, but they don't provide enough to make flua useful as a
>>> general purpose programming tool.  It lacks interfaces for interacting
>>> with the system (e.g., libc/libsys/libutil/etc wrappers) as well as
>>> standard programming facilities (e.g., classes, higher-order functions,
>>> etc.).  Here I'm mostly interested in discussing the former.
>>>
>>> I think flua could be a very useful alternative to shell scripts and C
>>> code where performance is not critical.  It's very good at wrapping C
>>> interfaces and thus could be used to make FreeBSD features (jails,
>>> bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who
>>> don't want to interact with C.
>>>
>>> It's a lot of work to build up a set of flua modules that provide enough
>>> functionality to be generally useful.  My feeling is that the easiest
>>> way to get there is to wrap C interfaces directly (to the extent that
>>> that's possible, but it's usually easy) and expose them in flua as a
>>> stable interface.  Libraries written in pure Lua (or other languages
>>> that interoperate well with Lua) could be built on top of that.
>>>
>>> I'm inclined to start by wrapping libc and libsys interfaces in a manner
>>> similar to luaposix.  There, the namespace is partitioned by C headers,
>>> so posix.unistd contains identifiers from unistd.h, posix.sys.stat
>>> contains identifiers from sys/stat.h, and so on.  In fact, flua already
>>> has a small subset of luaposix's functionality.  Wrapping C interfaces
>>> isn't much fun, but it's easy, and it saves us having to come up with
>>> names and interfaces (naming things is hard), and helps ensure that the
>>> glue code is relatively small and doesn't impose a large testing burden.
>>> Moreover, C interfaces don't change much and are subject to
>>> well-understood compatibility constraints, which should mean that Lua
>>> interfaces built on top of them are subject to the same constraints.
>>>
>>> Assuming there's general agreement on that approach, the question I'd
>>> really like to answer is, what should the namespace look like?  It would
>>> be useful to provide a posix module compatible with luaposix, but that
>>> obviously won't contain FreeBSD-specific functionality.
>>>
>>> I propose having a top-level "freebsd" namespace for all modules
>>> implemented in the base system, excluding posix and contrib libraries
>>> which already define a Lua interface (libucl is the one example I see so
>>> far; we could import sqlite bindings as well).  Then, if we follow
>>> luaposix's convention, we could have freebsd.sys.capsicum.* for
>>> Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent
>>> wrappers, and so on.  The posix module could simply provide a subset of
>>> freebsd.*.
>>>
>>
>> I think that's fine.  Ideally, we probably just import luaposix wholesale
>> now once we get the bootstrap flua module situation sorted and we can just
>> re-export those implementations into the freebsd.* namespace similar to how
>> we discussed luaposix was doing things today, unless those interfaces are
>> weird (I don't remember them being problematic, though).
> 
> Could you please elaborate on the bootstrap module problem?
> 

We build a bootstrap version of flua that was initially used to 
accommodate, e.g., `make sysent` after the conversion of makesyscalls. 
That'll become more important for the Linux/macOS builds if (when) some 
or all sysent artifacts are generated as part of the build as well.

The problem is that we don't setup the bootstrap flua to be able to do 
loadable modules (and we don't bootstrap any modules, for that matter), 
so we need to fix that before we further adopt it.

> Regarding importing luaposix, one potential problem there is with
> interopability of userdata types, mainly the treatment of file
> descriptors (which I would want to use in, e.g., freebsd.sys.capsicum).
> For example, in the reviews linked below I went with a userdata type for
> file descriptors so that they can be garbage-collected automatically,
> but luaposix doesn't do that AFAICS.  To me that's a bit unsavoury but
> I'm not sure how important it really is.  I do think that we could quite
> quickly and easily reimplement most of luaposix, for what it's worth.
> 

Ahh, that makes sense.  I don't feel that strongly about importing the 
rest of luaposix wholesale, so I'm perfectly happy to drop that idea; as 
you note, it's easy enough to reimplement it.

>>> Maybe it's better to separate C wrappers from native Lua modules though,
>>> so it could be better to have freebsd.c.sys.capsicum.*, etc., and
>>> provide higher-level interfaces for FreeBSD features under freebsd.pf,
>>> freebsd.zfs, freebsd.jail, etc..  I'm not really sure.
>>>
>>> In the interest of prompting discussion a bit, I posted some patches to
>>> add some example wrappers to flua:
>>> https://reviews.freebsd.org/D46553
>>> https://reviews.freebsd.org/D46554
>>> https://reviews.freebsd.org/D46556
>>>
>>
>> Will take a look when I get a bit of time.
> 
> Thank you kindly.
> 
>>> Does anyone have opinions on anything I've brought up above?  I'm pretty
>>> happy to write a lot of this glue code or find ways to automate it, as
>>> I've already done a fair bit of it, but it's hard to make progress
>>> without having some rigourous conventions for the flua namespace (again,
>>> naming things is hard).
>>>
>>
>> We should also figure out where to document this stuff (policy, mostly- we
>> have a 3lua section for libraries themselves).
> 
> Yep, I added a hacky posix.3lua in one of the reviews linked above.  A
> man page for flua itself is probably the next step.

Excellent!