Looking at replacing ATF/Kyua (in a limited fashion) with Google Test/shunit2

[Alan Somers]

On Tue, Jan 17, 2017 at 4:01 PM, Ngie Cooper (yaneurabeya)
<yaneurabeya at gmail.com> wrote:
> Hello all,
>
>         I had an initial discussion with a handful of other test stakeholders and due to the number of caveats with ATF/Kyua and a variety of issues in contributing back to the ATF/Kyua projects (time on maintainer’s end, legal issues, technical issues, etc), I'm seriously considering replacing parts of ATF/Kyua with GoogleTest and shunit2. In particular, I want to do these things [better]:
>
>         - Detecting kernels features and modules. It is ad hoc and done on a per-testcase basis or in ad hoc common routines, which bloats tests unnecessarily.
>         - Detecting network interfaces. My employer desires that — we had something in use internally for detecting interfaces in tests, but it was really messy; it would be nice if there was a clean, generalized way to do this.
>         - Adding test fixtures to ease testing on a per-suite basis. For example, setting up a service like bsnmpd for testing, running pjdfstest in different configurations, (a temporarily setup UFS or ZFS filesystem, etc), collecting core files after tests for analysis purposes, etc.
>         Why these frameworks?
>
>         GoogleTest:
>         - It is easier to extend upon.
>         - It is better documented than ATF/Kyua.
>         - It has more brevity in terms of writing testcases.
>         - It has features that I (and others who write tests) really want.
>         - It is very similar to atf-c-api in core ways, in terms of macros.
>         - It has been accepted by multiple open source projects, like clang/llvm, so including them in the FreeBSD Test Suite is more trivial after the porting and integrating work.
>         - It is lightweight and a good candidate for inclusion in the base system.
>         - Google and other groups are actively maintaining the project.
>
>         shunit2:
>         - It has more brevity in terms of writing testcases.
>         - It has features that I (and others who write tests) really want.
>         - It has syntactic sugar/API construction patterned after JUnit/XUnit in other ways, so it’s familiar to testers who have written JUnit/XUnit tests.
>
>         Some concerns that have been brought up:
>
>         Q: Why those frameworks? Have you considered other options?
>         A: Yes. A number of users have provided a matrix on Wikipedia and I’ve looked at the list to try and determine :
>                 a. C: https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C
>                 b. C++: https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C.2B.2B
>                 c. sh: https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#Shell
>
>         Q: Google sometimes tinkers with projects, then abandons them later. Is GoogleTest one of those projects?
>         A: As far as I can tell, no. This is widely used internally and developed on by their engineers. It is used external to Google by larger projects like LLVM, so even if Google disappeared tomorrow I believe there’s enough inertia and interest to keep things going here.
>
>         Q: What about C test integration?
>         A: I’m trying to see how far I can get with GoogleTest integration on the C API side, and if replacing atf-c-api is possible/doable using C integration in libc++ (the syntax/API construction for atf-c-api and GoogleTest is similar in some regards).
>
>         I’m really early on in the evaluation phase, so if anyone has any input to provide with the direction I’m going, I would really love the input. Writing tests for bsnmp will be my first proof of concept (I hope to use both GoogleTest and shunit2).
>
> Thank you,
> -Ngie

Before you get to far, you should ask yourself how much test case
isolation you desire.  Googletest is great, but I think it'll be hard
to get the strong isolation between testcases provided by ATF.
Perhaps it'll be find if GoogleTest tests provide no per-testcase
isolation; most C/C++ programmers don't expect it anyway.  But many of
FreeBSD's atf-sh tests depend on it.  Also, how are you planning to
hook up GoogleTest and shunit2 to Kyua?

-Alan