@guido, that was a good catch! When selecting “Display all in a grid” it’s impossible to spot it.
Anyway I’ve investigated this further and I saw the same behaviour in our internal infrastructure as well. The interesting fact was that the issue happened across all our machines (both aarch64 and x86).
Here for example happening on an Intel Skylake:
I was able to restrict the window of the change to August 21st. Around that day we didn’t change anything in our infrastructure and we have pyperformance pinned to 1.0.8.
When looking at the logs of the build I noticed though a difference on how the coverage package was built. Before the 21st there was something like this:
[2023-08-19T06:38:45.800Z] 2023-08-19 06:38:45,660: Building wheels for collected packages: coverage
[2023-08-19T06:38:45.800Z] 2023-08-19 06:38:45,662: Building wheel for coverage (setup.py): started
[2023-08-19T06:38:48.283Z] 2023-08-19 06:38:47,779: Building wheel for coverage (setup.py): finished with status 'done'
[2023-08-19T06:38:48.292Z] 2023-08-19 06:38:47,780: Created wheel for coverage: filename=coverage-6.4.1-cp313-cp313-linux_aarch64.whl size=216910 sha256=8cd103bb7a9c0017c85fa19b03a3730ef05d45f71383366c1737e08cf91b01e7
After the 21st:
[2023-08-23T00:35:50.189Z] 2023-08-23 00:35:50,087: Building wheels for collected packages: coverage
[2023-08-23T00:35:50.204Z] 2023-08-23 00:35:50,089: Building wheel for coverage (setup.py): started
[2023-08-23T00:35:51.292Z] 2023-08-23 00:35:51,000: Building wheel for coverage (setup.py): finished with status 'done'
[2023-08-23T00:35:51.300Z] 2023-08-23 00:35:51,001: Created wheel for coverage: filename=coverage-6.4.1-py3-none-any.whl size=176625 sha256=81073f3f8773eeb93b90da2d4457bf1303426b5fa73a180fbe055f531c021747
Notice the difference in the wheel size and the platform tag: one is cp313-cp313-linux_aarch64
and the other is py3-none-any
.
The wheel with the cp313-cp313-linux_aarch64
platform tag has a shared object file tracer.cpython-310-aarch64-linux-gnu.so
whilst the other one no. This explains the difference in size and in the platform tag.
Coverage provides two implementations of the tracer: the ctracer (faster) and the python based one (official documentation)
Hence since August 21st we see that coverage switched to using the python implementation of the tracer of coverage.
The next question was to understand why this happens. So I went and built myself with a version of cpython before and after August 21st to replicate the issue. The one before the 21st with with no problems whilst the one after raised an error when trying to build the ctracer. Here is the error:
creating build/temp.linux-aarch64-cpython-313/coverage/ctracer
gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/home/ent-user/ci-scripts/venv/cpython3.13-b51db007c75b-compat-e3aaa88db39d/include -I/home/ent-user/ci-scripts/tmpdir/prefix/include/python3.13 -c coverage/ctracer/datastack.c -o b
uild/temp.linux-aarch64-cpython-313/coverage/ctracer/datastack.o
In file included from coverage/ctracer/util.h:19,
from coverage/ctracer/datastack.c:4:
/home/ent-user/ci-scripts/tmpdir/prefix/include/python3.13/internal/pycore_frame.h:8:4: error: #error "this header requires Py_BUILD_CORE define"
8 | # error "this header requires Py_BUILD_CORE define"
| ^~~~~
In file included from /home/ent-user/ci-scripts/tmpdir/prefix/include/python3.13/internal/pycore_frame.h:13,
from coverage/ctracer/util.h:19,
from coverage/ctracer/datastack.c:4:
/home/ent-user/ci-scripts/tmpdir/prefix/include/python3.13/internal/pycore_code.h:8:4: error: #error "this header requires Py_BUILD_CORE define"
8 | # error "this header requires Py_BUILD_CORE define"
| ^~~~~
**
** Couldn't install with extension module, trying without it...
** BuildFailed: command '/usr/bin/gcc' failed with exit code 1
**
running bdist_wheel
The I looked through the list of commits of August 21st and found out that the commit gh-108220: Internal header files require Py_BUILD_CORE to be defined … · python/cpython@21c0844 · GitHub by @vstinner is the cause of the failure.
The next thins to understand is: why isn’t speed.python.org behaving in the same way? Here I can only speculate on the reason: in our infrastructure (both speed.python.arm.com and our internal one) we blow the environment for every run of pyperformance hence every time we reinstall everything.
This is important because we don’t carry the pip cache across experiments and packages get installed every time and this is true for the coverage package as well. Since August 21st it fails to build the ctracer and reverts back to the python implementation.
If the cache is shared across experiments instead, pip will pick the cached version of coverage (which presumably is coverage-6.4.1-cp313-cp313-linux_aarch64.whl
) until we bump the version of it.
Can anyone on speed.python.org shed some light on how the experiments are executed?
I’ve tested never versions of coverage and it seems that 7.3.2 works with 3.13.
Coverage 7.3.1 is broken with 3.13.
What’s the best way forward here? Do we need to bump coverage version from 6.4.1 to 7.3.2 in pyperformance? Anything else?
Apologies for the long reply but it was a fun an interesting problem to debug