From nobody Thu Sep 05 22:51:53 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X0F3h3xDmz5VCT9 for ; Thu, 05 Sep 2024 22:52:00 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X0F3g2WVQz4J7L; Thu, 5 Sep 2024 22:51:59 +0000 (UTC) (envelope-from markjdb@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=mrVKZH6b; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::831 as permitted sender) smtp.mailfrom=markjdb@gmail.com Received: by mail-qt1-x831.google.com with SMTP id d75a77b69052e-456757d8871so8818291cf.0; Thu, 05 Sep 2024 15:51:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725576718; x=1726181518; darn=freebsd.org; h=content-disposition:mime-version:message-id:subject:cc:to:from:date :sender:from:to:cc:subject:date:message-id:reply-to; bh=iezm2oDnk2EshbwXp6MXvP+p92UYNWcl2ZVPJzJxCSM=; b=mrVKZH6bL8WeYt+6411We3Pr/p3MRUT3QuARxmSWhUf8JKmnViT6wwQ1PVLpdPCGF+ TPu7EK93h2aVqWZlS8zLL74UoMb3n18b626ZRE3ig87ZhRtzxR/8qIjXHkj2dDTGwzuq 2VjRd6WIsf8Pv+lyehmPNNJOUjJd64Zx0QqJsycpkJG/B1bRNuzcaqvLf2FMAcJRq3uE aylXUqw3J/Fs8A1P1z0OlG4FgCbaVl1TEKKHF9PtccxiCtptVLs1oNtZqtDL5mrjdgli r3FuwnDqPmG74bbcGfgZHzakdQ+bVaRM4Mw/qINr1BJxxaK5xMFj9a1PAN1lBlBXaudR rxTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725576718; x=1726181518; h=content-disposition:mime-version:message-id:subject:cc:to:from:date :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=iezm2oDnk2EshbwXp6MXvP+p92UYNWcl2ZVPJzJxCSM=; b=fuSnFRvljBgVuF7tv/t8caZC48+/1aeoIV5P21RD6b4QxkIYqHxrwoNUEqPX/WxW4V 51r2V2HCL+f6ksJ6hbX+jZjtD8tM7ku34dUBXku5fcI7O841MQJ8gyGBT+Pvx+840jhB gIoqlPYH14PPdIH2/4+W7j1izWuzkHujsl20o9TFnwo6hXBJejQGOlH4MCYG1OckmQw/ q5/KMP4xx0yvVjAS8kF17eFE3guRzgm/gRu7DyotQhOODDC960YXXH++/o6h3MAeQ4bJ rnMQdhE1tAYivVgdCoo1PrbbUFg8zrGzIVxlLgoYGC38sOoihZP/O3aXNAF0buuGME03 FU0w== X-Forwarded-Encrypted: i=1; AJvYcCU190HvQ0OMuaifMhCV5Iqsxf9Emztcltcwxt5D76kEK+t5IBoltNCwnpNf5iX9caH5dEeQ@freebsd.org, AJvYcCVPo+gc2zxqV286XClyjqzUk4cmwoWz0/XQvfdCqpxDd+ouXmstgbu0NGVYiAqYRahkGPWM@freebsd.org X-Gm-Message-State: AOJu0Yyl2YrOniOuSZOnfs09uuacl+GNoZv0Zx34j/gg8qrDzvkelF6R PCzl5M3FNDLUq6+qsyvmWcaSrXssEPDDR76/svLcQWU9t8HwYukXLIZChw== X-Google-Smtp-Source: AGHT+IGAKClnPjn6A3uTEPtr+rL+T7gqPURXhB/EaPVuNZO0ZaZOeBFGzZ+67QKty02MB4lR0Lj7Qg== X-Received: by 2002:a05:622a:13d3:b0:43f:edde:7a55 with SMTP id d75a77b69052e-4580c6e1f22mr8189351cf.28.1725576717232; Thu, 05 Sep 2024 15:51:57 -0700 (PDT) Received: from nuc (192-0-220-237.cpe.teksavvy.com. [192.0.220.237]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45801cbb1acsm11233491cf.71.2024.09.05.15.51.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Sep 2024 15:51:56 -0700 (PDT) Date: Thu, 5 Sep 2024 18:51:53 -0400 From: Mark Johnston To: freebsd-hackers@freebsd.org Cc: kevans@freebsd.org, imp@freebsd.org, bapt@freebsd.org Subject: adding new flua libraries Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.57 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.974]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::831:from]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RCPT_COUNT_THREE(0.00)[4]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4X0F3g2WVQz4J7L FreeBSD ships a Lua 5.4 implementation, flua, in the base system. It's intended for use by the base system, and so far has a few consumers, but not many so far. (As an aside, flua's intended scope is not totally clear to me. Is it only for use by the base system? What compatibility guarantees does it provide, if any?) A few flua modules wrapping FreeBSD libraries (e.g., libjail) are available, but they don't provide enough to make flua useful as a general purpose programming tool. It lacks interfaces for interacting with the system (e.g., libc/libsys/libutil/etc wrappers) as well as standard programming facilities (e.g., classes, higher-order functions, etc.). Here I'm mostly interested in discussing the former. I think flua could be a very useful alternative to shell scripts and C code where performance is not critical. It's very good at wrapping C interfaces and thus could be used to make FreeBSD features (jails, bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who don't want to interact with C. It's a lot of work to build up a set of flua modules that provide enough functionality to be generally useful. My feeling is that the easiest way to get there is to wrap C interfaces directly (to the extent that that's possible, but it's usually easy) and expose them in flua as a stable interface. Libraries written in pure Lua (or other languages that interoperate well with Lua) could be built on top of that. I'm inclined to start by wrapping libc and libsys interfaces in a manner similar to luaposix. There, the namespace is partitioned by C headers, so posix.unistd contains identifiers from unistd.h, posix.sys.stat contains identifiers from sys/stat.h, and so on. In fact, flua already has a small subset of luaposix's functionality. Wrapping C interfaces isn't much fun, but it's easy, and it saves us having to come up with names and interfaces (naming things is hard), and helps ensure that the glue code is relatively small and doesn't impose a large testing burden. Moreover, C interfaces don't change much and are subject to well-understood compatibility constraints, which should mean that Lua interfaces built on top of them are subject to the same constraints. Assuming there's general agreement on that approach, the question I'd really like to answer is, what should the namespace look like? It would be useful to provide a posix module compatible with luaposix, but that obviously won't contain FreeBSD-specific functionality. I propose having a top-level "freebsd" namespace for all modules implemented in the base system, excluding posix and contrib libraries which already define a Lua interface (libucl is the one example I see so far; we could import sqlite bindings as well). Then, if we follow luaposix's convention, we could have freebsd.sys.capsicum.* for Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent wrappers, and so on. The posix module could simply provide a subset of freebsd.*. Maybe it's better to separate C wrappers from native Lua modules though, so it could be better to have freebsd.c.sys.capsicum.*, etc., and provide higher-level interfaces for FreeBSD features under freebsd.pf, freebsd.zfs, freebsd.jail, etc.. I'm not really sure. In the interest of prompting discussion a bit, I posted some patches to add some example wrappers to flua: https://reviews.freebsd.org/D46553 https://reviews.freebsd.org/D46554 https://reviews.freebsd.org/D46556 Does anyone have opinions on anything I've brought up above? I'm pretty happy to write a lot of this glue code or find ways to automate it, as I've already done a fair bit of it, but it's hard to make progress without having some rigourous conventions for the flua namespace (again, naming things is hard).