[python-committers] PEP 462: Workflow automation for CPython

[Antoine Pitrou]

On sam., 2014-01-25 at 06:35 -0800, Eli Bendersky wrote:
> On Sat, Jan 25, 2014 at 6:14 AM, R. David Murray
> <rdmurray at bitdance.com> wrote:
>         On Sat, 25 Jan 2014 05:49:56 -0800, Eli Bendersky
>         <eliben at gmail.com> wrote:
>         > do the latter in Python, which carries a problem we'll
>         probably need to
>         > resolve first - how to know that the bots are green enough.
>         That really
>         > needs human attention.
>         
>         
>         By "that needs human attention", do you mean: dealing with the
>         remaining
>         flaky tests, so that "stable buildbots are green" is a binary
>         decision?
>         We strive for that now, but Nick's proposal would mean we'd
>         have to
>         finally buckle down and complete the work.  I'm sure we'd make
>         some new
>         flaky tests at some point, but in this future they'd become
>         show-stoppers
>         until they were fixed.  I think this would be a good thing,
>         overall :)
> 
> 
> Non-flakiness of bots is a holy grail few projects attain. If your
> bots are consistently green with no flakes, it just means you're not
> testing enough :-)

There are certainly statistical ways to workaround the "necessary
flakiness", but that would require someone to sit with a pen and paper a
bit and figure out what the right metrics should be :-)

Regards

Antoine.