Greater sanitizer support in CI?

Ideas
1

In MSVC, the following sanitizer options are available ( /fsanitize (Enable sanitizers) | Microsoft Learn) :

  • /fsanitize=address
  • /fsanitize=kernel-address
  • /fsanitize=fuzzer
  • /fsanitize-address-use-after-return requires set ASAN_OPTIONS=detect_stack_use_after_return=1

With Clang ( AddressSanitizer — Clang 23.0.0git documentation):

  • -fsanitize=realtime, requires marking checked functions with [[clang::nonblocking]]
  • -fsanitize=type
  • -fsanitize=undefined
  • -fsanitize=memory
  • -fsanitize=thread
  • -fsanitize=address, export ASAN_OPTIONS=detect_leaks=1 to enable on all platforms

All of the above-mentions options can be combined with -fsanitize-minimal-runtime to reduce runtime overhead

With GCC ( Instrumentation Options (Using the GNU Compiler Collection (GCC))):

  • -fsanitize=undefined
  • -fsanitize=leak XOR -fsanitize=thread
  • -fsanitize=address XOR -fsanitize=thread
  • -fsanitize=kernel-address (Linux specific)
  • -fsanitize=hwaddress (AArch64 specific)
    • -fsanitize=kernel-hwaddress (AArch64 and Linux only)
  • -fsanitize=memtag-stack (AArch64 specific)
  • -fsanitize=pointer-compare and -fsanitize=pointer-subtract together with either -fsanitize=address or -fsanitize=kernel-address (enabled at runtime with export ASAN_OPTIONS=detect_invalid_pointer_pairs=2)

A quick peruse of CI workflow files show that only ASan, TSan, and UBSan is enabled on Ubuntu 24.04 amd64 images. Should this coverage not be extended to all Tier 1 and Tier 2 platforms listed in PEP 11 – CPython platform support | peps.python.org?

2

CI already takes an annoyingly amount of time and considerable gh resources.

I believe CI only covers Tier 1. Tier 2 is covered by buildbots, which might include some sanitizer runs – you could check. Adding Tier 2 to CI should then double CI time.

Adding sanitizer should then multiply CI time by an additional 4, or 8 total.

3

A far-fetched wish: but perhaps more virtual resources should be allocated to each sanitizer run, given that each sanitizer CI suite takes 12-20 minutes per run.

What’s the default resource limits for GH Action Runner images within cpython itself?

4

Clang Compiler User’s Manual — Clang 23.0.0git documentation
Control Flow Integrity — Clang 23.0.0git documentation
UndefinedBehaviorSanitizer — Clang 23.0.0git documentation

5

If the runs are shorter than the longest run (which is currently Windows free-threading, at ~25 minutes), it won’t take any longer to complete as they run in parallel?

Tier 1 requires CI support, but we also have several other platforms/configurations. E.g. the JIT, docs, lint, (the existing) sanitisers, CI Fuzz, Tier 2/3 platforms (Emscripten, Android, iOS) etc. And yes, we do have quite a few buildbots with sanitizers.

6

You are right, if they run in parallel. However, we have finite resources for CI and human wait times have been up to 2 hours when a particular PR sat queued for 1.5 hours. I believe resources have been increased to make this long be rare. I don’t know how much reserve margin we have currently have.