libc/libsys split coming soon

From: Brooks Davis <brooks_at_freebsd.org>
Date: Fri, 02 Feb 2024 22:31:43 UTC
TL;DR: The implementation of system calls is moving to a seperate
library (libsys).  No changes are required to existing software (except
to ensure that libsys is present when building custom disk images).

Code: https://github.com/freebsd/freebsd-src/pull/908

After nearly a decade of intermittent work, I'm about to land a series
of patches which moves system calls, vdso support, and libc's parsing of
the ELF auxiliary argument vector into a separate library (libsys).  I
plan to do this early next week (February 5th).

This change serves three primary purposes:
  1. It's easier to completely replace system call implementations for
     tracing or compartmentalization purposes.
  2. It simplifies the implementation of restrictions on system calls such
     as those implemented by OpenBSD's msyscall(2)
     (https://man.openbsd.org/msyscall.2).
  3. It allows language runtimes to link with libsys for system call
     implementations without requiring libc.

libsys is an auxiliary filter for libc.  This means that for any symbol
defined by both, the libsys version takes precedence at runtime.  For
system call implementations, libc contains empty stubs.  For others it
contains copies of the functions (this could be further refined at a
later date).  The statically linked libc contains the full
implementations so linking libsys is not required.

Additionally, libthr is now linked with libsys to provide
_umtx_op_err().

The overall implementation follows https://reviews.freebsd.org/D14609,
but is redone from scratch as multiple commits to facilitate review and
assist git's rename detection.

Testing:
  - Boot testing on amd64, aarch64, and riscv
  - make tinderbox (prior version, final run in progress)
  - exp-run: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276391
  - Kyua tests in poudriere amd64 jails: same 359 failures as with the
    latest freebsdci build

Thanks to Ali Mashtizadeh and Tal Garfinkel for D14609 and many
apologies for not landing this in a timely manner.  Additional thanks to
kib@ for many rounds of review, markj@ and kib@ for debugging rtld
issues exposed by this patch, and antoine@ for exp-runs.

Future work:
  - Purely functional interfaces to system calls (no errorno).
    Unfortunately there isn't an obvious way to do this without
    significant (possibly generated) assembly code.
  - Investigate msyscall(2) and pinsyscalls(2).
  - Reduce the size of stubs in libc.  I’ve errored on the
    side of not touching the copies that end up in libc to keep diff
    size down.  We might want to generate empty stubs instead.

See also:
  - Solaris Linker and Libraries Guide:
    https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter4-4.html

-- Brooks