FCP 20190401-ci_policy: CI policy

[Cy Schubert]

In message <DF8CB241-54AE-49DE-88F2-F28189486ED2 at FreeBSD.org>, "Kristof 
Provost
" writes:
> On 29 Aug 2019, at 17:01, Marcelo Araujo wrote:
> > Em qui, 29 de ago de 2019 Ã s 22:54, Kristof Provost <kp at freebsd.org>
> > escreveu:
> >
> >> On 29 Aug 2019, at 16:42, Ian Lepore wrote:
> >>> (And I don't think breaking a test counts as
> >>> breaking the build.)
> >>>
> >> I fundamentally disagree on this point. A test failure is, just like 
> >> a
> >> compiler warning, a precious gift that should not be ignored.
> >> The more distance (both in terms of time, and in terms of the people
> >> involved) there is between a bug being introduced and it being 
> >> detected
> >> the harder it is to fix it. Test accelerate detection of bugs. If we 
> >> do
> >> not take test failures seriously (i.e. as an indication something is
> >> wrong and should be fixed) the tests will inevitable become useless 
> >> in
> >> one of two ways: we’ll either disable failing tests (which is what 
> >> we
> >> tend to do now) reducing test coverage or we’ll have a test suite 
> >> with
> >> many failures in it, which makes it useless as well. (As with 
> >> compiler
> >> warnings, the best way to keep them under control is to consider them 
> >> to
> >> be fatal errors.)
> >>
> >
> > Could you elaborate where is the "fundamentally" you disagree? Where 
> > is the
> > fundament? You guys are introducing something new, yes everybody knows
> > about test, it is year 2019, but nobody can come with new rules tha in
> > hours we gonna revert if you "dare to don't fix it". Sorry, this is 
> > not how
> > people test software and fix it.
> >
> I do think that breaking a test breaks the build. Something used to work 
> and now it doesn’t. That’s breakage, even if it’s not as total as 
> it not compiling any more.

I agree. A broken test is an indication of a regression. If a test is not a 
regression, then it is not a valid test.

Having said that, 48 hours is a little draconian. Maybe 48 hours if no one 
steps up to the plate but if it's actively being worked on with end in 
sight I'd give the person working on it a little space.

What about weekends? I notice commits tend to slow down during weekends. I 
suspect people who work on FreeBSD at $COMPANY for a living treat it as a 
job, which it is. However this should be considered.

OTOH (arguing against myself to point out another consideration), it is 
foolhardy to commit a new feature just before a weekend and go camping (or 
be unreachable for whatever reason). A committer should be reachable N 
hours after a significant commit. Maybe 48 hours is justifiable then. (At 
$JOB we don't allow significant changes prior to vacation or even a flex 
day unless there is someone to cover for the person having done the work.)

My point is, we pick a number like 48. What's the rationale? What is the 
desired result? Can the desired result be accomplished without a 48 hour 
auto-revert rule? Should it be 48 hours during the week and 72  over 
weekends?


-- 
Cheers,
Cy Schubert <Cy.Schubert at cschubert.com>
FreeBSD UNIX:  <cy at FreeBSD.org>   Web:  http://www.FreeBSD.org

	The need of the many outweighs the greed of the few.