Report on Pull Request when buildbots fail


(Pablo Galindo Salgado) #1

To help to raise awareness when a buildbot fails post-merge we have implemented (https://github.com/python/buildmaster-config/pull/57) a new reporter in the buildbot master configuration that, in case of a build failure, will issue a comment in the Pull Request of the original commit.

To reduce the noise, the following conditions must be satisfied:

  • The build is a failure
  • The build has sourcestamps.
  • The build does not have more than 1 change. (This will avoid the problem of builds that build multiple commits at the same time).
  • The Pull Request can be detected from the commit message. We extract the PR number
    using the regular expression \((?:GH-|#)(\d+)\) . In case of multiple matches we use the last one added (to account for backports).

The message will look like:

Hi! The buildbot runtests has failed when building commit 6d32e43.

You can take a look here:

https://buildbot.python.org/#builders/1/builds/2

As some of the buildbots are very slow (for example, the ones that check for reference leaks) this has to be done post-merge but we are also working on a way to trigger some set of buildbots when the tag “awaiting-merge” is applied (or similar strategy) so we can know if the Pull Request is going to break some of the buildbots so we can make more informed decisions.

Please, report any errors, bugs or improvements that you can find interacting with this new reporter.

Note

There are “false positives” and race conditions in the test suite, so is possible that the buildbot reports an error that is not related to the Pull Request. I predict a low level of noise due to this fact, but I still think this has value as this will help raising awareness of the status of the buildbots.

Notice that some of us are monitoring the buildbot status mailing list to triage errors in the buildbots and open issues/try to fix them so this will help with this work and will increase visibility of buildbot failures.

Thanks to @vstinner, @yselivanov and @zware for helping with this idea :sparkles: :partying_face:


(Victor Stinner) #2

I suggest to create a page to explain these messages and explain how to deal with them:

  • How to read logs of a buildbot build
  • How to get help on these issues
  • Explain that some issues are random and so a fail doesn’t always mean a regression. For example, mention the common case of a network issue
  • etc.

Maybe put it in a wiki page.


(Victor Stinner) #3

By the way, I love the idea :slight_smile:


(Brett Cannon) #4

Wouldn’t a page in the devguide be better? I thought we had a page already covering buildbot stuff which would be the perfect place to keep such information.


(Victor Stinner) #5

Whatever works, I don’t really care where it’s hosted. I only care about a single point where we explain how to deal with such comment :slight_smile:


(Carol Willing) #6

If someone would like to open a PR on the devguide, I would be happy to review and iterate as needed.