nosh version 1.9

[Jonathan de Boyne Pollard]

Jonathan de Boyne Pollard:
> nosh version 1.9 is out. [...]
> It's worth noting that OpenBSD 5.6 now specifies that 
> /etc/rc.conf{,.local} doesn't have shell expansions and isn't 
> necessarily sourced by a shell, [...]

Alfred Perlstein:
> Very cool.
>
> Wondering about the idea of /etc/rc.conf *not* being a shell script... 
> this is sort of bad imo as I can't see any other way to provide the 
> settings dynamically for the startup scripts at a glance.
>
> I'll give you an example... FreeNAS (and by extension the appliance we 
> are building at Norse) has /etc/rc.conf.local as a shell script that 
> pulls data from an sqlite database, this allows us to set various 
> services on/off based on the contents of that sqlite database file. 
> This in turn allows us to leverage most of the existing /etc/rc.d and 
> by extension the /usr/local/etc/rc.d files provided by ports.
>
> I'm wondering how one could still do that if /etc/rc.conf and 
> /etc/rc.conf.local were no longer scripts? 

Adrian Chadd:
> The same way /etc/rc.conf and /etc/rc.conf.local is pulled in - via 
> the little snippet of stuff at the end of /etc/defaults/rc.conf , and 
> this bit of config in that file:
>
> local_startup="/usr/local/etc/rc.d" # startup script dirs.
> script_name_sep=" " # Change if your startup scripts' names contain 
> spaces
> rc_conf_files="/etc/rc.conf /etc/rc.conf.local"
>
> So, we just need some method of pulling in environment variables in 
> whatever order we need from whatever place we need. (God, why do I 
> know this stuff? Then I remembered - 
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=17595 . God damnit.) 
> The tricky bit is trying to make it so we don't call sqlite like a 
> thousand times to pull out all of the environment variables for each 
> invocation of an rc script.

The Introduction to the nosh Guide (q.v.) states the maxim that 
bootstrap time is not build again and again time, and this is a good 
example of where that maxim should be put into action.  The time to run 
pwd_mkdb is not again and again whenever the password database is 
consulted, or at system bootstrap; it is after vipw has run the 
editor.   And if one is generating /etc/rc.conf{,.local} from some 
source, such as a database, then the same principle applies. 
/etc/rc.conf{,.local} should not contain shell code that is parsed and 
executed again and again.  The code to pull things out of an sqlite 
database and write out /etc/rc.conf{,.local} belongs at the end of the 
content update process, whatever that is (be it some TUI administrator 
settings tool or a daemon that provides an RPC interface for managing 
system settings to flashy GUI desktop widgets), for the sqlite database.

There are many good things that fall out of this approach.  Here are 
just some.

/etc/rc.conf{,.local} don't need to be on a writable filesystem until 
the sqlite database itself is modifiable.  The approach of making 
/etc/rc.freenas or /etc/rc.conf{,.local} or some other part of the 
bootstrap auto-update /etc/rc.conf{,.local} whenever it detects a 
database that is newer: does, however.  (And as M. Chadd points out, the 
alternative approach of not writing out /etc/rc.conf{,.local} and simply 
pulling shell variables straight from the database has the disadvantages 
of running the database client again and again, and of needing it in 
rescue and emergency modes.)

/etc/rc.conf{,.local} don't have to contain anything that requires a 
shell, just plain name=value assignments (with a small number of lexing 
rules for quoting) that can be parsed in perl, Python, C++, or a 
language of one's choice.

One can see this design used in nosh system management itself from nosh 
1.17 onwards.  /etc/system-control/convert/ contains a suite of redo 
scripts that do a similar form of settings import.  As you can see, it 
doesn't run over and over at bootstrap.  It's designed to be run at 
various points *when the system administrator is changing settings*, 
using redo to check whether source files have changed and update the 
relevant target files, and service bundle settings, when they have.

So running

 > rcctl get mdmfs at -tmp

which is  another way of saying

 > system-control print-service-env mdmfs at -tmp

one obtains (on one of my systems)

 > size="20m"
 > flags="-S"

These settings for the mdmfs at -tmp service, contained in the service 
bundle for that service, come directly from tmpsize and tmpmfs_flags in 
/etc/defaults/rc.conf (on that particular system).  They are not 
regenerated at bootstrap.  They aren't stored in shell script that is 
parsed and executed repeatedly.  They aren't surrounded by a large 
number of additional variables unrelated to that service bundle, that 
have to be parsed again and again only to be then thrown away.  They are 
in a daemontools-style envdir directory that is specific to the service 
bundle; and they are the only settings that that service bundle needs, 
and thus the only ones that get imported to that service bundle.

Interestingly, the nosh configuration import system could be extended to 
replacing that FreeNAS mechanism.  Notice that it already builds a 
hosts.conf from nsswitch.conf.  This is a similar sort of exercise.  I 
expect that the configuration import system could easily be extended, or 
copied, to build an rc.conf{,.local} from an sqlite database.  One would 
need little more than an rc.conf.do and an rc.conf.local.do that 
executed some sqlite SELECTs and massaged the results appropriately; 
code to do which, you already have from FreeNAS.  One wouldn't even need 
the current FreeNAS's hairy shell logic in rc.conf.local for repeatedly 
taking checksums of the database file(s) to see whether it has been 
modified since last time (even though the containing filesystems might 
be mounted read-only and there is no possibility of such a change).  
That comes free with the redo system.

Put another way: All of that stuff comes out of FreeNAS rc.conf.local, 
which goes back to being a plain data file containing nothing but 
variable=value settings, and (minus the parts that come for free with 
redo) goes into (say) /etc/system-control/convert/rc.conf.local.do .  
You lose the need for /var/tmp/rc.conf.freenas entirely.  You thus lose 
the need for a writable /var .  And the data import process doesn't run 
at bootstrap, again and again.  It runs from the system installer, from 
the flashy GUI tool that an administrator uses for adjusting the stuff 
on the sqlite database, from the humble TUI tools that do the same, or 
from (even) whatever DBus-speaking freenas-settings-d daemons you might 
care to dream up for speaking to the KDE and GNOME desktops and their ilk.