A new way to test systems in multiple machine scenarios...

[George Neville-Neil]

On 8 Jul 2014, at 4:56, Garrett Cooper wrote:

> On Jul 6, 2014, at 9:06 PM, Craig Rodrigues <rodrigc at FreeBSD.org> 
> wrote:
>
>> On Sat, Jul 5, 2014 at 8:04 PM, George Neville-Neil 
>> <gnn at neville-neil.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I've coded up a system to allow you to control multiple other 
>>> systems for
>>> use in testing.
>>>
>>> https://github.com/gvnn3/conductor
>>>
>>>
>> Cool!  The architecture you have is similar to that of the SPECsfs
>> benchmark test (  http://www.spec.org/sfs2008/ )
>> which involves a "coordinator node" and multiple "client nodes" which
>> direct NFS network
>> traffic towards a System Under Test (SUT).  Garrett Cooper actually 
>> set up
>> the original testbed
>> that I am using now at iXsystems. :)
>>
>> It would be cool to put together tools like Jenkins, Kyua, and 
>> conductor to
>> do more advanced testing
>> of FreeBSD before the project puts out releases.
>
> Agreed. The only thing that I have some concern about is the 
> reinventing of the wheel in python. multiprocessing Managers are one 
> viable option that’s existed since python 2.6; there’s a learning 
> curve though, and you’ll run into problems with pickling some 
> objects because the pickle protocol is far from complete (example: 
> http://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-pythons-multiprocessing-pool-ma 
> ); you might run into this problems regardless because you’re 
> serializing objects using pickle instead of using dill (or using a 
> simpler serialization method like JSON). Fabric has a framework 
> that’s nice to use if you have ssh capability. There are other 
> frameworks that use twisted conch I think too (another library that 
> implements ssh access).

Yes, I learned quite a bit about pickling in writing this.  Conductor 
aims to be quite simple so I am
hoping to avoid any crazy corner cases to do with pickling.

>
> Isilon has a framework they use, but it’s very customized to their 
> infrastructure and product assumptions and it’s in need of an 
> overhaul :(.

This, actually, is the problem I found.  Lots of folks have partial 
solutions that are either proprietary,
internal, not read for prime time, not quite what we want, etc. etc.  I 
did get one private
response of another system to look at as well.

I basically did this as a stake in the ground around which to build 
something we could possibly move forwards
with.  It's not a 100% solution but it's 80% of the solution to the 
problem I run into 80% of the time.

Best,
George