Re: adding new flua libraries

On Fri, Sep 6, 2024 at 8:45 AM Kyle Evans <kevans@freebsd.org> wrote:

> On 9/6/24 04:29, Baptiste Daroussin wrote:
> > On Thu 05 Sep 18:51, Mark Johnston wrote:
> >> FreeBSD ships a Lua 5.4 implementation, flua, in the base system.  It's
> >> intended for use by the base system, and so far has a few consumers, but
> >> not many so far.  (As an aside, flua's intended scope is not totally
> >> clear to me.  Is it only for use by the base system?  What compatibility
> >> guarantees does it provide, if any?)
> >>
> >> A few flua modules wrapping FreeBSD libraries (e.g., libjail) are
> >> available, but they don't provide enough to make flua useful as a
> >> general purpose programming tool.  It lacks interfaces for interacting
> >> with the system (e.g., libc/libsys/libutil/etc wrappers) as well as
> >> standard programming facilities (e.g., classes, higher-order functions,
> >> etc.).  Here I'm mostly interested in discussing the former.
> >>
> >> I think flua could be a very useful alternative to shell scripts and C
> >> code where performance is not critical.  It's very good at wrapping C
> >> interfaces and thus could be used to make FreeBSD features (jails,
> >> bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who
> >> don't want to interact with C.
> >>
> >> It's a lot of work to build up a set of flua modules that provide enough
> >> functionality to be generally useful.  My feeling is that the easiest
> >> way to get there is to wrap C interfaces directly (to the extent that
> >> that's possible, but it's usually easy) and expose them in flua as a
> >> stable interface.  Libraries written in pure Lua (or other languages
> >> that interoperate well with Lua) could be built on top of that.
> >>
> >> I'm inclined to start by wrapping libc and libsys interfaces in a manner
> >> similar to luaposix.  There, the namespace is partitioned by C headers,
> >> so posix.unistd contains identifiers from unistd.h, posix.sys.stat
> >> contains identifiers from sys/stat.h, and so on.  In fact, flua already
> >> has a small subset of luaposix's functionality.  Wrapping C interfaces
> >> isn't much fun, but it's easy, and it saves us having to come up with
> >> names and interfaces (naming things is hard), and helps ensure that the
> >> glue code is relatively small and doesn't impose a large testing burden.
> >> Moreover, C interfaces don't change much and are subject to
> >> well-understood compatibility constraints, which should mean that Lua
> >> interfaces built on top of them are subject to the same constraints.
> >>
> >> Assuming there's general agreement on that approach, the question I'd
> >> really like to answer is, what should the namespace look like?  It would
> >> be useful to provide a posix module compatible with luaposix, but that
> >> obviously won't contain FreeBSD-specific functionality.
> >>
> >> I propose having a top-level "freebsd" namespace for all modules
> >> implemented in the base system, excluding posix and contrib libraries
> >> which already define a Lua interface (libucl is the one example I see so
> >> far; we could import sqlite bindings as well).  Then, if we follow
> >> luaposix's convention, we could have freebsd.sys.capsicum.* for
> >> Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent
> >> wrappers, and so on.  The posix module could simply provide a subset of
> >> freebsd.*.
> >>
> >> Maybe it's better to separate C wrappers from native Lua modules though,
> >> so it could be better to have freebsd.c.sys.capsicum.*, etc., and
> >> provide higher-level interfaces for FreeBSD features under freebsd.pf,
> >> freebsd.zfs, freebsd.jail, etc..  I'm not really sure.
> >>
> >> In the interest of prompting discussion a bit, I posted some patches to
> >> add some example wrappers to flua:
> >> https://reviews.freebsd.org/D46553
> >> https://reviews.freebsd.org/D46554
> >> https://reviews.freebsd.org/D46556
> >>
> >> Does anyone have opinions on anything I've brought up above?  I'm pretty
> >> happy to write a lot of this glue code or find ways to automate it, as
> >> I've already done a fair bit of it, but it's hard to make progress
> >> without having some rigourous conventions for the flua namespace (again,
> >> naming things is hard).
> >
> > I like all of what I read and I am fully aligned.
> >
> > What I don't see here, is I think we should have dynamic modules libucl,
> fbsd &
> > friends are not dynamic and the reviews I see for the freebsd module
> reviews,
> > this is still bundled.
> >
>
> Right, this is an artifact of how flua's evolved that we need to move
> away from.  We didn't have module support at the beginning- that came
> later, but we didn't really move things out into modules.  We still
> can't move some things because of the bootstrap flua problem I mentioned
> elsewhere (doing so would block work to do `make sysent` type work as
> part of the build as it would break cross-builds), but at least
> libucl/fbsd would be fine.
>

Yea, if the sysent.lua code doesn't use modules, would we still have the
same
bootstrapping issues? Today, I don't think here's anything but vanilla lua
code
in there.

Warner


> > my proposal for unbundling, I need kldload for a project of mine, so I
> did:
> > https://reviews.freebsd.org/D46558
> >
> > Best regards,
> > Bapt
>
>