NVMe unit tests

[Jim Harris]

On Thu, Jul 30, 2015 at 5:20 AM, Craig Rodrigues <rodrigc at crodrigues.org>
wrote:

>
>
> On Wed, Jul 29, 2015 at 3:02 PM, Alan Somers <asomers at freebsd.org> wrote:
>
>> On Wed, Jul 29, 2015 at 11:59 AM, Jim Harris <jimharris at freebsd.org>
>> wrote:
>>
>> Kyua doesn't natively understand CUnit.  So you have two options: run the
>> tests with the plain test adapter, or convert them all to ATF-C tests.
>> The
>> main disadvantage to the plain test adapter is that Kyua won't be aware of
>> the detailed test results.  For example, if you have 500 CUnit test cases,
>> Kyua will only report whether the whole suite passed or failed rather than
>> whether each test case passed or failed.
>>
>
> In the May/June 2015 issue of FreeBSD Journal ( http://freebsdjournal.com
> ), there is
> a good interview with Julio Merino (jmmv).  In the interview, Julio says:
>
> "If you happen to be an atf-sh user, I'd like ot suggest you have a look
> at shtk's unittest module, which is my latest attempt at implementing a
> more modern testing library for the shell that follows the common xUnit
> idioms.  And if you use atf-c++, consider googletest as an alternative, a
> much more mature C++ testing library".
>
> On a similar note, I have used CUnit, and find it to be a perfectly fine
> library
> for writing unit tests in C.  CUnit and googletest can both generate xUnit
> XML test result
> output which can be used as input to continuous integration systems such
> as Jenkins.
>
> Based on the comments from Julio, I have a very hard time encouraging
> people
> to write new tests using ATF.  I have a harder time encouraging people to
> migrate
> existing tests from other test libraries to ATF, if they have stuff
> already working with things like
> googletest or CUnit.
>
> At my work, I originally tried to steer people to write new tests in ATF
> and migrate tests from CUnit to ATF, but then
> gave up, because of pushback from developers.  The combination we have that
> seems to work for us is:  CUnit for unit tests in C, pyunit tests in
> Python for tests which can run in a scripting language, and all tests
> results reported in Jenkins.  This combination may not work for everyone,
> but it is what seems to be OK for us.
>
> If kyua could be extended to natively run CUnit and googletest, that would
> be
> very useful.  That would allow us to run more types of test libraries and
> have the test results reported natively in kyua.  I don't have the time to
> work on extending kyua in this way.
> I'm not sure if Julio does either.  In that interview, he states that he
> is quite busy.
>
> The alternative is to run CUnit or googletests directly (even if they are
> in the FreeBSD src tree), and report the results in the Jenkins test report
> viewer.  That option may not be attractive, but it would work.
>

Alan and Craig,

Thanks for the feedback.  I will integrate these into HEAD soonish, under
tests/sys/dev/nvme as Alan suggested.

I do not plan to switch these tests from CUnit to something else, but would
be willing to add code to spit out XML files with per-test results if that
would be helpful.  Otherwise the tests currently are organized per nvme
source file, so we would at least get per-file granularity on the test
results.

-Jim



> --
> Craig
>