To help to raise awareness when a buildbot fails post-merge we have implemented (Add Pull Request GitHub reporter by pablogsal · Pull Request #57 · python/buildmaster-config · GitHub) a new reporter in the buildbot master configuration that, in case of a build failure, will issue a comment in the Pull Request of the original commit.
To reduce the noise, the following conditions must be satisfied:
- The build is a failure
- The build has sourcestamps.
- The build does not have more than 1 change. (This will avoid the problem of builds that build multiple commits at the same time).
- The Pull Request can be detected from the commit message. We extract the PR number
using the regular expression\((?:GH-|#)(\d+)\). In case of multiple matches we use the last one added (to account for backports).
The message will look like:
Hi! The buildbot runtests has failed when building commit 6d32e43.
You can take a look here:
As some of the buildbots are very slow (for example, the ones that check for reference leaks) this has to be done post-merge but we are also working on a way to trigger some set of buildbots when the tag “awaiting-merge” is applied (or similar strategy) so we can know if the Pull Request is going to break some of the buildbots so we can make more informed decisions.
Please, report any errors, bugs or improvements that you can find interacting with this new reporter.
Note
There are “false positives” and race conditions in the test suite, so is possible that the buildbot reports an error that is not related to the Pull Request. I predict a low level of noise due to this fact, but I still think this has value as this will help raising awareness of the status of the buildbots.
Notice that some of us are monitoring the buildbot status mailing list to triage errors in the buildbots and open issues/try to fix them so this will help with this work and will increase visibility of buildbot failures.
Thanks to @vstinner, @yselivanov and @zware for helping with this idea 
