Which buildbots does `test-with-buildbots` run?

Committers
1

The test-with-buildbots label on PRs is useful to get more CI coverage, but it’s not obvious to me which buildbots exactly are triggered on this label.

For example, I’ve added the label on GH-110829: Ensure Thread.join() joins the OS thread by pitrou · Pull Request #110848 · python/cpython · GitHub but it seems currently that only 4 buildbots have been selected, according to this status page.

I seem to remember that test-with-buildbots selected a much more comprehensive set of buildbots, did that change?

2

It usually takes a while for the full suite of buildbots to start (and longer for them to finish) running on a PR. There’s a bit of a queue. (Agreed that this isn’t always so obvious from the CI!)

1 Like
3

It’s not clear to me neither :frowning: See open discussions at:

IMO test-with-buildbots should be the most trusted buildbots, like stable buildbots, maybe only Tier-1 and Tier-2. But not slow “Refleaks” buildbots. My concern here is to no annoy people with buildbots know to be unstable, but also not to waste resources. A single Refleak build can take 1 to 4 hours! Usually, Refleak builds are run on the same builders than other non-Refleak build, and so the buildbot queue can be blocked for hours because of that.

If some people want to run more buildbots, like running ref leak tests, test-with-refleak-buildbots should be used: but I would prefer that this common only runs Refleaks buildbots.

If some people needs specific buildbots, the !buildbot REGEX can be used.

If we need more buildbots, a new label can be added. The link above are about unstable buildbots like NoGIL.

1 Like
4

+1 from me, that is my preference as PR author as well. But I wouldn’t expect that to boil down to only 4 buildbots :slight_smile:

Edit: ok, finally there are a lot of them :slight_smile:

5

The UI is not obvious. Yeah, builders are added one by one, once the build is scheduled, not before.