Re: adding new flua libraries

On 9/6/24 09:48, Warner Losh wrote:
> 
> 
> On Fri, Sep 6, 2024 at 8:45 AM Kyle Evans <kevans@freebsd.org 
> <mailto:kevans@freebsd.org>> wrote:
> 
>     On 9/6/24 04:29, Baptiste Daroussin wrote:
>      > On Thu 05 Sep 18:51, Mark Johnston wrote:
>      >> FreeBSD ships a Lua 5.4 implementation, flua, in the base
>     system.  It's
>      >> intended for use by the base system, and so far has a few
>     consumers, but
>      >> not many so far.  (As an aside, flua's intended scope is not totally
>      >> clear to me.  Is it only for use by the base system?  What
>     compatibility
>      >> guarantees does it provide, if any?)
>      >>
>      >> A few flua modules wrapping FreeBSD libraries (e.g., libjail) are
>      >> available, but they don't provide enough to make flua useful as a
>      >> general purpose programming tool.  It lacks interfaces for
>     interacting
>      >> with the system (e.g., libc/libsys/libutil/etc wrappers) as well as
>      >> standard programming facilities (e.g., classes, higher-order
>     functions,
>      >> etc.).  Here I'm mostly interested in discussing the former.
>      >>
>      >> I think flua could be a very useful alternative to shell scripts
>     and C
>      >> code where performance is not critical.  It's very good at
>     wrapping C
>      >> interfaces and thus could be used to make FreeBSD features (jails,
>      >> bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who
>      >> don't want to interact with C.
>      >>
>      >> It's a lot of work to build up a set of flua modules that
>     provide enough
>      >> functionality to be generally useful.  My feeling is that the
>     easiest
>      >> way to get there is to wrap C interfaces directly (to the extent
>     that
>      >> that's possible, but it's usually easy) and expose them in flua as a
>      >> stable interface.  Libraries written in pure Lua (or other languages
>      >> that interoperate well with Lua) could be built on top of that.
>      >>
>      >> I'm inclined to start by wrapping libc and libsys interfaces in
>     a manner
>      >> similar to luaposix.  There, the namespace is partitioned by C
>     headers,
>      >> so posix.unistd contains identifiers from unistd.h, posix.sys.stat
>      >> contains identifiers from sys/stat.h, and so on.  In fact, flua
>     already
>      >> has a small subset of luaposix's functionality.  Wrapping C
>     interfaces
>      >> isn't much fun, but it's easy, and it saves us having to come up
>     with
>      >> names and interfaces (naming things is hard), and helps ensure
>     that the
>      >> glue code is relatively small and doesn't impose a large testing
>     burden.
>      >> Moreover, C interfaces don't change much and are subject to
>      >> well-understood compatibility constraints, which should mean
>     that Lua
>      >> interfaces built on top of them are subject to the same constraints.
>      >>
>      >> Assuming there's general agreement on that approach, the
>     question I'd
>      >> really like to answer is, what should the namespace look like? 
>     It would
>      >> be useful to provide a posix module compatible with luaposix,
>     but that
>      >> obviously won't contain FreeBSD-specific functionality.
>      >>
>      >> I propose having a top-level "freebsd" namespace for all modules
>      >> implemented in the base system, excluding posix and contrib
>     libraries
>      >> which already define a Lua interface (libucl is the one example
>     I see so
>      >> far; we could import sqlite bindings as well).  Then, if we follow
>      >> luaposix's convention, we could have freebsd.sys.capsicum.* for
>      >> Capsicum-related syscalls and constants, freebsd.sys.event.* for
>     kevent
>      >> wrappers, and so on.  The posix module could simply provide a
>     subset of
>      >> freebsd.*.
>      >>
>      >> Maybe it's better to separate C wrappers from native Lua modules
>     though,
>      >> so it could be better to have freebsd.c.sys.capsicum.*, etc., and
>      >> provide higher-level interfaces for FreeBSD features under
>     freebsd.pf <http://freebsd.pf>,
>      >> freebsd.zfs, freebsd.jail, etc..  I'm not really sure.
>      >>
>      >> In the interest of prompting discussion a bit, I posted some
>     patches to
>      >> add some example wrappers to flua:
>      >> https://reviews.freebsd.org/D46553
>     <https://reviews.freebsd.org/D46553>
>      >> https://reviews.freebsd.org/D46554
>     <https://reviews.freebsd.org/D46554>
>      >> https://reviews.freebsd.org/D46556
>     <https://reviews.freebsd.org/D46556>
>      >>
>      >> Does anyone have opinions on anything I've brought up above? 
>     I'm pretty
>      >> happy to write a lot of this glue code or find ways to automate
>     it, as
>      >> I've already done a fair bit of it, but it's hard to make progress
>      >> without having some rigourous conventions for the flua namespace
>     (again,
>      >> naming things is hard).
>      >
>      > I like all of what I read and I am fully aligned.
>      >
>      > What I don't see here, is I think we should have dynamic modules
>     libucl, fbsd &
>      > friends are not dynamic and the reviews I see for the freebsd
>     module reviews,
>      > this is still bundled.
>      >
> 
>     Right, this is an artifact of how flua's evolved that we need to move
>     away from.  We didn't have module support at the beginning- that came
>     later, but we didn't really move things out into modules.  We still
>     can't move some things because of the bootstrap flua problem I
>     mentioned
>     elsewhere (doing so would block work to do `make sysent` type work as
>     part of the build as it would break cross-builds), but at least
>     libucl/fbsd would be fine.
> 
> 
> Yea, if the sysent.lua code doesn't use modules, would we still have the 
> same
> bootstrapping issues? Today, I don't think here's anything but vanilla 
> lua code
> in there.
> 

Right- if it didn't we'd have no problem, but the in-tree version uses 
lposix to construct a tmpdir to throw the intermediate files in while 
we're building them.  I suppose the new model would more likely generate 
them in-place in .OBJDIR instead and could avoid that.