How to prioritize configurations for new buildbot builders

Core Workflow
1

From this PR comment and @Rosuav’s mailing list post, I was wondering if there’s general guidance for prioritizing additional builder configurations.

Since buildbot workers are maintained by volunteers, I understand it’s up to the individual contributing buildbot workers to decide what configurations they want to cover, but I’m sure there are scenarios where the volunteer cares about certain parameters (e.g. “I want to add free-threaded builders”, or “I care about M1 MacOS coverage”) but doesn’t care about other parameters (e.g. “what’s more valuable between refleaks, ASAN, PGO, …”).

So I’m looking for some guidance on how to choose the free parameters.
We can use the two concrete cases linked above, before attempting to make any generalized guidance.

  1. I have two MacOS workers (Intel + M1) that cover the default debug+nogil configuration. I think I can safely add at least one builder on each. I want to add more NoGIL coverage - what would be the most valuable configurations to add? Perhaps one Refleaks and one PGO? Would it be better to have the same configuration for both, or different configurations?
  2. @Rosuav has available capacity on a Debian AMD64 builder. Considering we have Ubuntu coverage of NoGIL debug and NoGIL Refleaks configurations, what might be the next most valuable configuration to add? Perhaps PGO+LTO? or ASAN?
2

As you’ve noticed, buildbot workers are pretty adhoc contributions by people with the resources and interest. Given what you’ve got and that we already have some Linux and Refleaks workers configured for PEP-703, I’d use your macs to add ASAN and PGO+LTO as you suggest.

I don’t think we’ve ever had coverage for those on macOS bots? If that’s the case and there are issues showing up with either, a baseline default/non-703 --enable-gil build of the same to compare against might be useful for comparison.

ASAN is presumably more practically useful from an interesting problems it can reveal perspective than PGO+LTO which is often more of a test that we have our build system all configured to work correctly on the platform than it is a test of our code itself.

2 Likes