# Which buildbots does \`test-with-buildbots\` run? **URL:** \ **Category:** Committers\ **Tags:** buildbots\ **Created:** [October 14, 2023, 8:10am UTC](https://discuss.python.org/t/which-buildbots-does-test-with-buildbots-run/36171 "2023-10-14T08:10:35Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![pitrou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pitrou/32/28_2.png) [@pitrou](https://discuss.python.org/u/pitrou)\ **Post date:** [October 14, 2023, 8:10am UTC](https://discuss.python.org/t/which-buildbots-does-test-with-buildbots-run/36171/1 "2023-10-14T08:10:35Z")
The `test-with-buildbots` label on PRs is useful to get more CI coverage, but it’s not obvious to me which buildbots exactly are triggered on this label. For example, I’ve added the label on [GH-110829: Ensure Thread.join() joins the OS thread by pitrou · Pull Request #110848 · python/cpython · GitHub](https://github.com/python/cpython/pull/110848) but it seems _currently_ that only 4 buildbots have been selected, according to this [status page](https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F110848%2Fmerge). I seem to remember that `test-with-buildbots` selected a much more comprehensive set of buildbots, did that change? ---
**Author:** ![AlexWaygood](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/alexwaygood/32/4722_2.png) [@AlexWaygood](https://discuss.python.org/u/AlexWaygood)\ **Post date:** [October 14, 2023, 8:43am UTC](https://discuss.python.org/t/which-buildbots-does-test-with-buildbots-run/36171/2 "2023-10-14T08:43:34Z")
It usually takes a while for the full suite of buildbots to start (and longer for them to finish) running on a PR. There’s a bit of a queue. (Agreed that this isn’t always so obvious from the CI!) ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [October 14, 2023, 9:07am UTC](https://discuss.python.org/t/which-buildbots-does-test-with-buildbots-run/36171/3 "2023-10-14T09:07:59Z")
> [@pitrou](#): > > The `test-with-buildbots` label on PRs is useful to get more CI coverage, but it’s not obvious to me which buildbots exactly are triggered on this label. It’s not clear to me neither ☹ See open discussions at: - [Allow triggering unstable buildbots with the !buildbot PR comment · Issue #419 · python/buildmaster-config · GitHub](https://github.com/python/buildmaster-config/issues/419) - [gh-419: Allow unstable builders to trigger on PRs by itamaro · Pull Request #420 · python/buildmaster-config · GitHub](https://github.com/python/buildmaster-config/pull/420) IMO `test-with-buildbots` should be the most trusted buildbots, like stable buildbots, maybe only Tier-1 and Tier-2. But not slow “Refleaks” buildbots. My concern here is to no annoy people with buildbots know to be unstable, but also not to waste resources. A single Refleak build can take 1 to 4 hours! Usually, Refleak builds are run on the same builders than other non-Refleak build, and so the buildbot queue can be blocked for hours because of that. If some people want to run more buildbots, like running ref leak tests, `test-with-refleak-buildbots` should be used: but I would prefer that this common _only_ runs Refleaks buildbots. If some people needs specific buildbots, the `!buildbot REGEX` can be used. If we need more buildbots, a new label can be added. The link above are about _unstable_ buildbots like NoGIL. ---
**Author:** ![pitrou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pitrou/32/28_2.png) [@pitrou](https://discuss.python.org/u/pitrou)\ **Post date:** [October 14, 2023, 9:12am UTC](https://discuss.python.org/t/which-buildbots-does-test-with-buildbots-run/36171/4 "2023-10-14T09:12:04Z")
> [@vstinner](#): > > IMO `test-with-buildbots` should be the most trusted buildbots, like stable buildbots, maybe only Tier-1 and Tier-2. But not slow “Refleaks” buildbots. My concern here is to no annoy people with buildbots know to be unstable, but also not to waste resources. +1 from me, that is my preference as PR author as well. But I wouldn’t expect that to boil down to only 4 buildbots 🙂 **Edit:** ok, finally there are a lot of them 🙂 ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [October 14, 2023, 11:15am UTC](https://discuss.python.org/t/which-buildbots-does-test-with-buildbots-run/36171/5 "2023-10-14T11:15:12Z")
The UI is not obvious. Yeah, builders are added one by one, once the build is scheduled, not before. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex002/uploads/python1/original/1X/3aff56846ebe57781a163dfc748fd1417119d8ab.png) [@system](https://discuss.python.org/u/system)\ **Post date:** [October 13, 2024, 8:11am UTC](https://discuss.python.org/t/which-buildbots-does-test-with-buildbots-run/36171/6 "2024-10-13T08:11:07Z")
Closed on October 13, 2024, 8:11am UTC This topic was automatically closed after 365 days. New replies are no longer allowed.