mtree "language" enhancements

[Simon J. Gerraty]

Warner Losh <imp at bsdimp.com> wrote:
> As part of making NanoBSD buildable by non-root, I've found a need to have
> a richer mtree language than we currently have.

No fundamental objection there.
Indeed I'd really like the ability to provide default uid/gid
for the case that a uname/gname cannot be looked up.
Or even just a flag to say if lookup fails use 0:0
This would avoid the need to post-process BSD.var.dist to replace all
uname/gname with uid=0/gid=0 during various bootstrap situations.

> I'd like a new type called 'action' (so type=action in the records). This
> type is defined loosely to manipulate and earlier entry (or maybe entries,
> still unsure) in the file.
> 
> Each action entry would have an 'action' keyword. The keywords I've defined

would or could?

> so far are as follows:
> 1. "unlink" which throws away the previous entry. That entry has been
> removed. It may apply to files or directories, but it is an error not to
> remove all entries in a directory when removing the directory.
> 2. "move" which relocates a previous entry. An additional targetpath
> keyword specifies the ultimate destination for this entry.
> 3. "copy" which duplicates a previous entry. It too takes targetpath.
> 4. "meta" which changes the meta data of the previous entry. All keywords
> on this are merged with the previous entry.

Probably need to know a bit more about how NanoBSD is built/packaged to
comment more usefully.  Any useful references?

> The one other thing that my merging tool does is to remove all size
> keywords. In the NanoBSD environment, size is irrelevant. Files are
..
> replaced and appended to all the time in the build process, and it doesn't
> make sense to track the size. makefs fails if the size is different, so

Agreed.

Where do these size keywords come from?
We (Juniper) do not have them in any of our mtree based manifests.
Which we use directly with makefs.

On the off chance it is of interest...
I wonder if this style of manifest would simplify your problem?
I believe all the code needed (other than makefiles) is in head at least.

There are two styles supported, classic mtree:

#mtree
#
# Group IDs used:
#       0       wheel
#
# User IDs used:
#       0       root
#
/set uid=0 gid=0 mode=555 type=file

bin type=dir
  cat contents="${STAGE_OBJTOP}/bin/cat"
  cp contents="${STAGE_OBJTOP}/bin/cp"
      
  ..

which is good for manually maintained manifests,
and for autogenerated (eg via find) an full path format:

usr/tests/bin/cat/d_align.in mode=0644 contents="/b/sjg/work/stable10/obj/stage/i386/usr/tests/bin/cat/d_align.in"
usr/tests/bin/cat/d_align.out mode=0644 contents="/b/sjg/work/stable10/obj/stage/i386/usr/tests/bin/cat/d_align.out"

the two can be combined - an mtree style header with autogenerated
info appended.

> If things go well, we could eventually move these extensions into mtree so
> that the post-processing stage is no longer necessary. I'm content to
> maintain the hundred or two lines of awk I've written to implement it. I
> chose awk because it does the job well enough, though python might do it
> better. But I don't want to talk about that choice since right now it is
> purely internal to NanoBSD (though I hope that other build orchestration
> systems like src/release and crochet look to adopt).

FWIW we use python when awk/sed etc prove insufficient or cumbersome
but awk/sed are usually adequate.