Revisiting the ZFS test suite

[Allan Jude]

On 2019-10-06 18:29, Alan Somers wrote:
> On Sun, Oct 6, 2019 at 10:20 AM Allan Jude <allanjude at freebsd.org
> <mailto:allanjude at freebsd.org>> wrote:
> 
>     On 2019-10-04 12:23, Alan Somers wrote:
>     > My employment status changed last week, and I find myself once again
>     > working on ZFS.  It's hard to have confidence in any changes without a
>     > solid test suite, so my main task right now is to fix the problems in
>     > FreeBSD's ZFS test suite.  Our most recent run had 64 failed tests
>     and 99
>     > skipped ones; let's get those down to 0 and something small,
>     respectively.
>     >
>     >
>     https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/3880/testReport/
>     >
>     > Would anybody be interested to review my changes to the test
>     suite?  It's
>     > not terrible if I have to commit without review; I can't possibly
>     break
>     > anything but the test suite itself.
>     >
>     > -Alan
>     > _______________________________________________
>     > freebsd-testing at freebsd.org <mailto:freebsd-testing at freebsd.org>
>     mailing list
>     > https://lists.freebsd.org/mailman/listinfo/freebsd-testing
>     > To unsubscribe, send any mail to
>     "freebsd-testing-unsubscribe at freebsd.org
>     <mailto:freebsd-testing-unsubscribe at freebsd.org>"
>     >
> 
>     Have you followed the work here:
> 
>     https://github.com/zfsonfreebsd/ZoF
> 
>     That has ported the improved ZFS test suite from ZFSonLinux back to
>     FreeBSD (as part of the effort to upstream FreeBSD support into that
>     newer version of ZFS)
> 
>     -- 
>     Allan Jude
> 
> 
> Oh, boy, I didn't know about that fork of the test suite.  When I made
> FreeBSD's, I didn't try for portability.  Merging the two will not be
> easy.  There are > 60,000 lines of diffs between the two, plus  > 1,000
> files that don't exist on both sides.  Worse, there are some major
> structural differences:
> 1) FreeBSD's test suite runs as root and drops privileges selectively,
> but ZoL's runs as an unprivileged user and uses sudo selectively to gain
> privileges
> 2) FreeBSD's test suite uses ATF, just like everything else, but ZoL's
> test suite uses a bespoke python program
> 3) FreeBSD's test suite runs setup and cleanup once for every test file,
> but ZoL does it once for every directory of test files.
> 4) FreeBSD's test suite uses FreeBSD make, but ZoL's uses autotools and
> gmake
> 
> Of those item 3 is the most troubling.  I wonder if ZoL could be
> convinced to switch to ATF and Kyua?  They're pretty portable.
> -Alan

Yeah, so I think the ZoF project is just porting the ZoL test suite to
FreeBSD, not necessarily pushing anything from ours back. It'd be good
to get to keep more of our tests.

The goal was really to be able to take advantage of the existing CI that
ZFSonLinux has, and to require all future pull-requests against the new
OpenZFS repo require passing the ZTS on both OSes to be able to be merged.

So the more FreeBSD stuff we can get in there the better.


-- 
Allan Jude

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-testing/attachments/20191006/00216b67/attachment.sig>