Incorporating the Capsicum test suite

[Garrett Cooper]

> On Jun 3, 2015, at 08:30, Craig Rodrigues <rodrigc at FreeBSD.org> wrote:
> 
>> On Tue, Jun 2, 2015 at 1:40 PM, Ed Maste <emaste at freebsd.org> wrote:
>> 
>> 
>> I have a couple of questions. First, do we have an existing pattern
>> for connecting googletest tests to Kyua?
> 
> No we do not.  Googletest looks like it is basically trying to do
> the same thing as ATF or CUnit.
> 
> Googletest has a way to generate JUnit XML test output:
> https://code.google.com/p/googletest/wiki/AdvancedGuide#Generating_an_XML_Report
> 
> The easiest thing to do would be to configure the Capsicum tests to
> generate this XML test output.  This can be imported natively into
> Jenkins, which can then parse the test report.
> This would require no changes to the Capsicum tests.
> 
> If you want to integrate googletest with kyua, then things get tricky.
> On the googletest page: https://code.google.com/p/googletest/
> they list a third party plugin which can take googletest and generate TAP
> output.
> I don't know how well that works.  It would be nice if that functionality
> was
> integrated in googletest natively.
> 
> Another option would be to modify kyua, so that it can parse the output of
> googletest
> natively.  This would be quite nice.  I haven't found extending kyua to be
> very
> easy....it requires good knowledge of C++ and Lua.  Also, whoever
> contributes
> code to kyua must sign a Contributor License Agreement (CLA) with Google.
> That's something to keep in mind.
> 
> Another option would be to rewrite the Capsicum tests in ATF.
> I don't know how many tests there are, but this might be doable.
> ATF and Googletest look similar.
> 
> If you wanted to import the Capsicum tests into FreeBSD as they are right
> now, then it looks
> like you would need to import googletest into FreeBSD as well.  googletest
> is BSD licensed,
> so that is not a problem, but we already have ATF in the tree, so I don't
> know how
> useful it would be to have multiple test frameworks in the base system.

Hi Ed,
    I'll take a look at the test infrastructure and see if I can write a simple shim for ATF to interact with it.
    I agree with a lot of what Craig said. The Google corporate CLA has impeded engineers in EMC / Isilon and some other groups from contributing back to the ATF / Kyua projects ;( (this is why I've just been filing bugs against the projects and have been focusing my efforts on integrating tools/regression tests into Kyua), so anything that needs to be done development wise on these projects will need to be done internal to the FreeBSD project for the time being.
Thanks!
-NGie