Openssl, nss, ca_root_nss, and certificate stores - SOLUTION PROVIDED near end

[Da Rock]

I'm sure this will turn into a very lively discussion :)

I very nearly resurrected this old thread 
https://lists.freebsd.org/pipermail/freebsd-hackers/2016-March/049162.html, 
but decided against it, although the subject matter is very close.

It seems that this is a wide issue for all OS types - none is immune :D 
(not even winblow$) But the solutions are similar between linux and BSD. 
The general consensus is that a system wide certificate store should be 
available for all applications (there is some contention as well I 
believe, with a minority? disagreeing with this stance), and some means 
of achieving this are similar as well. What should and shouldn't work 
OOB is in question, and given the secure nature of the problem this is 
understandably so.

The main problem lies in openssl and nss contention and configuration, 
and the need for "minimal" solutions. Currently, openssl is configured 
(in base) to prioritise a certificate bundle over a path, and is 
prioritised to use /usr/local/etc/ssl over /etc/ssl (as of 11.1). As 
noted, this is rather counterintuitive, as openssl in base (intuitively) 
should look in base.

Further, /usr/local/etc/ssl/cert.pem (ca bundle) is not installed by 
default (fair enough IMHO), but overrides any local trusted 
certificates. So if this is updated, as it is by ca_root_nss, sysadmin 
has to update the certificates - which presents some security issues as 
well. Given the number of ports that depend on this port (hardcoded and 
optional) - this is a relatively big issue for a relative minority who 
require local trusts. But it actually is not quite as simple as that...

Why its not that simple is that if the updates to root certs is not 
atomic, then it stands to reason that there is a possibility of breakage 
(particularly in large deployment/remote installations). At the least, 
there is a small window where security policy may be broken - 
organisationally or otherwise (this may be highly debatable).

Ironically, ca_root_nss gets the certificate bundle from nss, which has 
to be downloaded and parsed. This solution is the same as curl has used, 
and a few other linux solutions do similar. The nss src contains a text 
file to be parsed and then hardcoded into the nss libnssckbi.so library, 
but this is bypassed and parsed directly by ca_root_nss scripts (curl 
and other project use this method as well). And while ca_root_nss offers 
an option to symlink, only a symlink from /usr/local/etc/ssl to /etc/ssl 
is completed; the cert bundle is actually installed in 
/usr/local/share/certs AND /usr/local/etc/ssl.

There are arguments for and against this, with whatever policy wrt 
mozilla/nss, symlinks to /etc, even store locations. As a sysadmin 
though, I think that a solution that gives control to the sysadmin makes 
more sense as whatever organisational policy is going to be adhered to - 
not whatever Freebsd devs decide. Does that sound reasonable?

I personally "dumpster dived" into this issue for several days (this 
began with pkg using libfetch and a local certificate) and was 
thoroughly overwhelmed, so I can understand the confusion and contention 
in this area. 2 competing engines with 2 very different configurations - 
trying to bridge the 2 is insane. I may have discovered a reasonable 
solution though.

POSSIBLE SOLUTION:

We already use the nss root certificates anyway - most ports are 
dependent on it. nss compatibility exists in many apps as well - except 
for base (which is fine - keep it minimal). So what seems the reasonable 
thing to do is more overtly use this system, rather than covertly as we 
are atm. Instead of parsing the src, have nss port setup a store 
(optionally, if needed), add the libnssckbi.so module, and extract the 
certificates into a cert.pem.

If certificates need to be added, add them to nss and run the update 
script to extract the "trusted" certificates again into 
/usr/local/etc/ssl/cert.pem (symlinked to /etc of course).

This has an added benefit in that the sysadmin can decide not to trust 
the mozilla root certificates, and not include the libnssckbi.so module, 
can even decide not trust any one certificate in the store (including 
the mozilla roots), and updates automatically through the lib module.

The certificates in the trust store are merged, not overwritten, and 
base applications and libraries as well as port/pkg installed are all 
kept happy.

A further benefit is that even user apps like firefox,chromium, and 
thunderbird can then be configured to use a shared store (using a sqlite 
db - not the old dbm format), which means sysadmin can ensure that only 
trusted certficates are installed. It should be noted that db's can be 
separate and merged - handy so that the system store and user stores can 
remain separate if desired.

This is only a step in the right direction, not necessarily the end 
solution, as there is work on a bridging solution as well. I believe 
using p11-kit one can use a "drop-in" replacement library for 
libnssckbi.so, and maybe a capath style trust store backend. Of course 
this requires a lot more configuration.

For those who prefer openssl in the ported apps (such as curl) most of 
those actually have nss support. It s mainly base that doesn't.

This would require some patching though of many ports, and some extra 
install scripts in nss (or maybe ca_root_nss might need hijacking?). 
First of which would be the checking of 
/usr/local/share/certs/ca-root-certs.pem and the triggered dependency. 
nss install would need to check the existence of the system wide root 
store (and like openssl, it has preferences but not hardcoded as such) 
to see if creation is required, and then a quick update script. Given 
current preference policy, I'd say this should be a sqlite db located in 
/usr/local/etc/pki/nssdb.

Note that AFAICT the nssdb updates the root certs in the library, but 
the db details if the certs are actually trusted. So when the update 
script occurs, the trusted certs will only be added/updated within the 
library, and the db will still detail if they are actually trusted - 
only trusted certs will allow any connections. This has added benefits 
to those with strict organisational policies.

This is not perfect, but it would be reasonably elegant for now. It also 
gives ultimate control to the system's administrator, not necessarily 
devs in any particular project. Ultimately something will inevitably be 
resolved to bridge the 2 engines (even if libressl were used instead of 
openssl), but neither are going away (or winning either), so we have to 
learn to deal with both as best as possible. p11-kit appears to be 
working towards a bridge, but I believe its still rather experimental in 
a lot of ways.

I think that ultimately a distributed store solution is more inclined to 
breakage or policy issues, or at the very least may become inconsistent. 
I'd personally prefer a single store that can be maintained easily. But 
that's me - and there may even be room to allow for both with this 
solution as well.

Any thoughts? Suggestions? I figured discussion might be better here 
than in a PR. If this goes well, I can write the scripts and submit a PR.