Testing PRs with buildbots
After some work (and updating the buildbot server to Python3.8), I am happy to announce that is now possible to test a Pull Request with the buildbot fleet on demand by just by adding a label ( test-with-buildbots):
When you add this label to an existing Pull Request:
- A build will be immediately triggered in a selected subset of the stable buildbots. Every buildbot will report to the PR the build status (like Travis, Azure Pipelines…etc). This is what you will see:
As long as the label is in the Pull Request every new commit to the pr will be tested with the buildbots in the same way that Travis or Azure Pipelines tests every commit that is pushed to the PR.As requested, the label will be removed once a build is scheduled. If you want to test again, you should add the label once more.
Clicking the “Details” link in the GitHub status will bring you to the builder page if you want to read logs or see the build live.
The checks are not required to merge the PR. This means that if you decide so, you can proceed and merge the PR even if a buildbot is failing (for example, if the failure is unrelated to the PR).
In the buildbot page you can distinguish all the builders that are building PRs by a new
Pull Request label:
When to use the new label
Using the buildbots before merging will be especially useful in the following situations:
You are reviewing a big C PR and you want to make sure that there are no refleaks by using the refleak-detection builders. These builders take more time but will tell you if the PR has some sneaky reference leaks.
You want to make sure that the PR builds on all platforms. We have a big collection of platforms (various versions of Windows, macOS, linux distros) and configurations (debug, non-debug, PGO/LTO, LTO, clang sanitizers…) and if you want to make sure that a PR that you suspect may behave differently by platform (like adding new POSIX functions) will not break later, is a good practice to test with the buildbots first.
Because you don’t want us to suffer afterwards if your PR fails on some obscure platform
Each individual builder has a limited amount of PRs that can be tested simultaneously (depends on the builder) so it may take a while to start building your PR. So please, be patient! You can check the status of all builders in the buildbot page
Take into account that the buildbots are donated machines, so remember to be a good citizen and double-check the code of the PR before adding the label.
This new addition should allow you to gain confidence when merging a PR with minimum impact in your existing workflow and effort. If you have any feedback or feature request, go ahead and tell us