From nobody Thu Sep 05 23:11:47 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0FVX4nFqz5VFyd for ; Thu, 05 Sep 2024 23:11:48 +0000 (UTC) (envelope-from kevans@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0FVX4LkRz4Nbs; Thu, 5 Sep 2024 23:11:48 +0000 (UTC) (envelope-from kevans@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725577908; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wpYHRSdhkJePMCYlwNheBDUmemIjnsyWLX1VwLl6v2c=; b=gJG689h66HAvKXNqB9do1ISmJA1Gbmo6cCmSmW2GO2Jb29SbKjycgFl3cytn9ja4GIkAn1 oCsI1RiRMptBxD5Gs6TDRLGPWfB8FoZxaTQvafpSA1wiGLJWdWFmyNWiBhGZR1IPV0D4Wj P3JF91omAx5RjibFHKY+yetZEhmf/ibwgcbiqd/Ie3ydPDN5NXomAMivv4NgULesgiWabb q8NcNGyVxpMBJ3UFreep4EY7uUN9NyIqVZJsPM/9DHdRQvT/kR7S/i1PCQnm8w+oQeCklx gGOHazCN1xbMnuWY75JEtPraWqiJUL7SCqILeZ/FUEimq9IWfvcu102Ofs+CGg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725577908; a=rsa-sha256; cv=none; b=PU798w9B0HaaDQEV6LE+/knU0kVM49jbmBfFuHs1H0Tde2I5bIdE7HstOZWTFGD+BRI5gk yis110FWG2IJdGMtffuw+TGID4S5Q7i0MiOQphx1nDiEmEFIC/hqUfRZldrnXJordEshLz bBn2GWv/k363Tf1eH7OZf5DXMEsHITQ8T/1lhcF79kcqdhveEbuz2w5bOlJNxkZLGL6Vq1 BoRw7SUw8UczA2yw4gzJq6kp5y7CbMChAdc0YadZK2QstrDjYmGuRNeONhFECD8cbQzosa y/B9bDzI21ymLGnkuD9ZaD6f759vC4F5kRQYN3Dbs74U+YazmUPKIzzp7xEwUw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725577908; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wpYHRSdhkJePMCYlwNheBDUmemIjnsyWLX1VwLl6v2c=; b=f4Gc4++6fVNg7N9yNeFdEcex1mOZKMgFlplqdPe6L75M2HDBfwsSx854NZ9Kzb76b0P4ff hx+1oOjJ1jabUfqv9kVnFOFv+liiKDHuzX948Pw7yymE02u2y1BVEPc+pP19tOLe800y0O BM1QAB45jrY9GCuAmcJHrIHxu/ewzfE1d+LfOkfsdAmQqKuVpJFbDh4FPm6yY/Dsequvoh 3WhVFKTFrV+nWBPLsdw8AfmmxbYFH44spqt01ZGhNf2QYtzbvd0cY9sc6Rfs5zblfzYQCy yfBWyulno1LDUK8mIjA3sGuRznGkh7C92oob1JkTifTjUKr+Akbp9ixokox2Vw== Received: from [10.9.4.95] (unknown [209.182.120.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: kevans/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4X0FVX19Ymz1M2p; Thu, 5 Sep 2024 23:11:48 +0000 (UTC) (envelope-from kevans@FreeBSD.org) Message-ID: Date: Thu, 5 Sep 2024 18:11:47 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: adding new flua libraries To: Mark Johnston , freebsd-hackers@freebsd.org Cc: imp@freebsd.org, bapt@freebsd.org References: Content-Language: en-US From: Kyle Evans In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 9/5/24 17:51, Mark Johnston wrote: > FreeBSD ships a Lua 5.4 implementation, flua, in the base system. It's > intended for use by the base system, and so far has a few consumers, but > not many so far. (As an aside, flua's intended scope is not totally > clear to me. Is it only for use by the base system? What compatibility > guarantees does it provide, if any?) > Only by the base system (hence the rename and hiding it in /usr/libexec), no guarantees except for modules we've re-implemented subsets of- if a conflicting name exists in ports, its use must be compatible with that. > A few flua modules wrapping FreeBSD libraries (e.g., libjail) are > available, but they don't provide enough to make flua useful as a > general purpose programming tool. It lacks interfaces for interacting > with the system (e.g., libc/libsys/libutil/etc wrappers) as well as > standard programming facilities (e.g., classes, higher-order functions, > etc.). Here I'm mostly interested in discussing the former. > > I think flua could be a very useful alternative to shell scripts and C > code where performance is not critical. It's very good at wrapping C > interfaces and thus could be used to make FreeBSD features (jails, > bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who > don't want to interact with C. > > It's a lot of work to build up a set of flua modules that provide enough > functionality to be generally useful. My feeling is that the easiest > way to get there is to wrap C interfaces directly (to the extent that > that's possible, but it's usually easy) and expose them in flua as a > stable interface. Libraries written in pure Lua (or other languages > that interoperate well with Lua) could be built on top of that. > > I'm inclined to start by wrapping libc and libsys interfaces in a manner > similar to luaposix. There, the namespace is partitioned by C headers, > so posix.unistd contains identifiers from unistd.h, posix.sys.stat > contains identifiers from sys/stat.h, and so on. In fact, flua already > has a small subset of luaposix's functionality. Wrapping C interfaces > isn't much fun, but it's easy, and it saves us having to come up with > names and interfaces (naming things is hard), and helps ensure that the > glue code is relatively small and doesn't impose a large testing burden. > Moreover, C interfaces don't change much and are subject to > well-understood compatibility constraints, which should mean that Lua > interfaces built on top of them are subject to the same constraints. > > Assuming there's general agreement on that approach, the question I'd > really like to answer is, what should the namespace look like? It would > be useful to provide a posix module compatible with luaposix, but that > obviously won't contain FreeBSD-specific functionality. > > I propose having a top-level "freebsd" namespace for all modules > implemented in the base system, excluding posix and contrib libraries > which already define a Lua interface (libucl is the one example I see so > far; we could import sqlite bindings as well). Then, if we follow > luaposix's convention, we could have freebsd.sys.capsicum.* for > Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent > wrappers, and so on. The posix module could simply provide a subset of > freebsd.*. > I think that's fine. Ideally, we probably just import luaposix wholesale now once we get the bootstrap flua module situation sorted and we can just re-export those implementations into the freebsd.* namespace similar to how we discussed luaposix was doing things today, unless those interfaces are weird (I don't remember them being problematic, though). > Maybe it's better to separate C wrappers from native Lua modules though, > so it could be better to have freebsd.c.sys.capsicum.*, etc., and > provide higher-level interfaces for FreeBSD features under freebsd.pf, > freebsd.zfs, freebsd.jail, etc.. I'm not really sure. > > In the interest of prompting discussion a bit, I posted some patches to > add some example wrappers to flua: > https://reviews.freebsd.org/D46553 > https://reviews.freebsd.org/D46554 > https://reviews.freebsd.org/D46556 > Will take a look when I get a bit of time. > Does anyone have opinions on anything I've brought up above? I'm pretty > happy to write a lot of this glue code or find ways to automate it, as > I've already done a fair bit of it, but it's hard to make progress > without having some rigourous conventions for the flua namespace (again, > naming things is hard). > We should also figure out where to document this stuff (policy, mostly- we have a 3lua section for libraries themselves). Thanks, Kyle Evans