Re: FreeBSD, Illumos ZFS, and OpenZFS

On 2/18/24 20:10, David Christensen wrote:
> freebsd-questions:
> 
> 
> I have been running FreeBSD 12 RELEASE amd64 with ZFS on root and a ZFS 
> data pool 24x7 on a SOHO storage server for the past four years.  The 
> learning curve has been tough at times, but I have been very pleased 
> with the results.
> 
> 
> AIUI FreeBSD 12 used Illumos ZFS code, while FreeBSD 13 uses OpenZFS code.
> 
> 
> Given the OpenZFS data corruption bug in December 2023, I was hesitant 
> to upgrade to FreeBSD 13 and OpenZFS:
> 
> https://www.phoronix.com/news/OpenZFS-Data-Corruption-Battle

https://www.freebsd.org/security/advisories/FreeBSD-EN-23:16.openzfs.asc 
has a patch also including 12.4 for this was a ZFS issue from 2006 and 
not just an OpenZFS issue. The way the corruption would occur would lead 
to the corruption being what ZFS stored on disk and therefore checksums 
protect the corruption instead of detect it; I wish ZFS could chechsum 
the file's data and not ZFS's data but it doesn't seem to be that way. 
Performance needs the current way while reliability would benefit from 
the file data checksums and could lead to simple ways through issues 
like compression/encryption comparative differences at the cost of 
performance. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275308 
had details from those tracking + fixing the bug. Using an unsupported 
release means you are no longer likely to receive fixes for bugs that 
are found+fixed after the support date, though being opensource does 
leave the option for them to still be fixed by yourself and others.

> Now another OpenZFS bug has been found:
> 
> https://www.phoronix.com/news/OpenZFS-Encrypt-Corrupt

If I recall, native encryption was not present in the non-OpenZFS copy 
so this would be an issue that only applies to the OpenZFS copy only. 
Originally ZFS got native encryption after Oracle bought Sun and closed 
it off; opensource projects did not import that work. I haven't followed 
if Illumos's native encryption is any different in terms of 
stability/bugs but though it was pulled in from OpenZFS. FreeBSD also 
has encryption available at other levels using things like geli; ZFS 
encryption bugs then don't apply.

> My avoidance of OpenZFS has been reaffirmed.  I do not want my data on 
> OpenZFS.
> 
> 
> But FreeBSD 12 is now EOL and unsupported.  I cannot access the final 
> set of packages.

Catch 22 you put yourself into. Easiest way out of this corner on 
FreeBSD without upgrading to 13+ is going to be building from ports but 
over time you will slowly find more and more things break due to not 
supporting 12. Though easy, there is the cost of electricity and 
computing resources to account for. You could dedicate a cheap computer 
or virtual machine to build the ports into packages for install/upgrade 
if you do not want to do that on the original hardware.

> It is possible to run Illumos ZFS code on FreeBSD 13 RELEASE amd64?  I 
> do not see a port:

You could try to manually bring back the illumos branch of the code. As 
OpenZFS was available through the ports tree before it also replaced 
what is in base, you could look at that and the FreeBSD base code prior 
to OpenZFS being imported as a sample of how to try to bring up an 
Illumos branch of the work either as a port or just built in natively. 
You would then be attempting to maintain the code that others were 
trying to do before the migration happened.

> https://ports.freebsd.org/cgi/ports.cgi?query=zfs&stype=name&sektion=all
> 
> 
> Should I just switch to an Illumos based distribution?  Recommendations?

Illumos does sometimes pull in code from OpenZFS and can still be prone 
to bugs there in addition to its own. If you want an Illumos ZFS 
experience without work on your part, I'd expect switching to Illumos is 
the most reliable way to get there. I do not know if they ever plan to 
migrate their work to pull off of OpenZFS (now based on ZFS on Linux) 
directly as FreeBSD did.

> https://illumos.org/docs/about/distro/
> 
> 
> Any other comments or suggestions?

'Some' bugs can be avoided by creating pools without certain features 
enabled. This applies to any ZFS codebase.

> David
>