Can you reproduce this locally? On my laptop the shared library in the framework do contain symbols according to both nm(1) and by checking that a LC_SYMTAB load command is present. This is for Python 3.10, 3.11 and 3.12 (all of them installed using the latest release on python.org).
I have used the installer from https://github.com/actions/python-versions/releases/download/3.11.0-3328127706/python-3.11.0-darwin-x64.tar.gz locally and used nm on both the binary (…/bin/python3) and the library (…/Resources/Python.app/Contents/MacOS/Python). All I get is the __mh_execute_header symbol 
I don’t know what that download is and I’m away from a keyboard but that’s likely not the official installer. Go to python.org downloads to find it. We don’t release through GitHub.
The setup-python action maintainers claim they use the official installer now
Please, try to install the official installer from our download page in your machine and then check if the problem is there. If is not there you should then ask the GitHub actions maintainers.
Also, to be able to install on users’ systems without violating macOS Gatekeeper requirements, the official installer binaries are code signed and have specific entitlements. As Pablo suggests, you should try installing on your own macOS system using the official installer download. In a CI environment, you might need to copy the binaries and/or do your own code signing. You should not attempt to change a system’s SIP settings to work around this.
I’ve uninstalled my local installation of Python 3.11 from the setup-python release. Downloaded the official installer (which happens to have the same size as the .pkg contained in the setup-python archive), re-installed Python 3.11 with it, and still got
❯ nm -gU /Library/Frameworks/Python.framework/Versions/3.11/bin/python3
0000000100000000 T __mh_execute_header
❯ nm -gU /Library/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python
0000000100000000 T __mh_execute_header
And Python 3.12 from the official installer
❯ nm -gU /Library/Frameworks/Python.framework/Versions/3.12/bin/python3
0000000100000000 T __mh_execute_header
❯ nm -gU /Library/Frameworks/Python.framework/Versions/3.12/Resources/Python.app/Contents/MacOS/Python
0000000100000000 T __mh_execute_header
I can reproduce this result. This is on macOS Monterey 12.6, on an Intel Mac, with the Python 3.11.0 from python.org.
I get the same result for Python versions all the way back to 3.5.
I suspect the official python.org installer has always behaved this way, and if something has changed, it is the “releases” put out by the GitHub Actions. Which was mentioned in the Original Post. What’s less clear to me is what the OP means by the phrase “In particular, the 3.11 images ship with the official MacOS installer since at least the rc2 release.” Gabriele, can you help us understand what you meant there? It seems everyone has assumed that you were reporting a change in behavior in the rc2 release, but it sounds more likely that you are just stating a fact that you have only verified in rc2 and later.
It seems what they’re saying is that since at least 3.11 rc2, the GitHub Actions container images use the official Python.org macOS installers (and per the linked thread, this started with either 3.11 rc1 or 3.11 rc2, and will be done in the future starting with the first rc), which in turn exposed the stripped symbols in the official installer (that per your testing was an issue back to Python 3.5), that was evidently not a problem with GitHub’s own installers used previously.
@nad It looks like something in the process of building the installers strips the symbols. Can you share the compiler commands and scrips you use to create the installers to see if we can reproduce?
I suspect that maybe when creating the installable thing some apple software is doing some shenanigans to strip symbols for apple reasons.
@pablogsal see Ronald’s reply earlier where he verified that the symbols were there. Perhaps we are talking about different things.
Based on CAM’s message it looks to me that the OP is unhappy that GitHub (not us) has changed the container images used by GitHub Actions to contain a Python installer without symbols. Sure, we might help them by not stripping symbols in our official installers, but why would we incur the extra cost on all our users? I can also understand that GitHub prefers to use official installers where they exist.
So how can we help Austin’s CI jobs recover the symbols?
In case it wasn’t clear (it wasn’t to me until I clicked on their link), the OP is Austin’s author. (Nice job BTW!)
Even if we don’t change anything I still would like to understand in what part of the process the symbols are lost. Also @ronaldoussoren reports that when he installs python the symbols are there for him, which is another mystery.
I checked the framework itself earlier (nm -gU /Library/Frameworks/Python.framework/Versions/3.11/Python) and that does contain a full set of symbols. Both the stub executable and Python.app/…/Python are small binaries linked to the framework library.
What symbols do you expect to see in these executables?
Gabriele, can you help us understand what you meant there?
Sure. I’ll try to add more context to clarify the situation, but I’m afraid this is going to be a bit of a read.
Prelude
Austin is a frame stack sampling tool for Python that works by resolving some of the exported symbols from the Python binaries by reading the interpreter’s remote memory space. In particular, in later releases, Austin relies on the (_)_PyRuntime symbol to get hold of the interpreter state, and from there loop over the thread states and unwind the frame stack for each of them.
If symbols are not available, Austin tries to find the interpreter state from a BSS scan. If that fails too, on some occasions Austin might try to scan (a sensible portion of) the heap, as a final desperate attempt before giving up.
The way Austin finds out which is the “interesting” binary is based on a few heuristics, which don’t make this discovery step super reliable. This is to cope with situations where the Python binary might be embedded in an executable that has no [Pp]ython in its path. Things we look for are the file size (an interesting Python binary is a few MB at least) and then symbols. There is always the chance that Austin picks the wrong binary, making all the efforts of looking up the interpreter state pointless. I haven’t noticed this behaviour with any of the most popular Python distributions though.
Python 3.11 support
Lately I’ve been working on adding Python 3.11 support to Austin. During the early development stages, while the beta releases were out, I was able to successfully test Austin on MacOS using the setup-python action on GitHub. At that stage, the action was providing custom builds of Python. However, with the rc2 release, the setup-action maintainers decided to move to the official installer (plus a script to automate the installation process on the GH workflow runners). I discovered this because the CI for Python 3.11 on MacOS started failing with permission issues. I have opened this issue with setup-python, and the maintainers confirmed that they had moved to the official installer. Thanks to this comment by @ronaldoussoren on a previous discussion, I was able to make the CI job work again by removing the signature from the binaries. Evidence of that is in the CI job for the commit used to produce the latest 3.4.1 release of Austin. There you can also see the extra step for removing the signature from the binaries. I have also done some manual testing on my machine too and Austin seemed to work just fine.
The CI issue
A few days ago, while doing some more work on Austin, I discovered that the CI jobs for Python 3.11 and 3.10 started failing badly. Such big failures are generally an indication that either support for that particular Python version is broken, or something has changed in the binaries and Austin can’t handle that. In the former case, I’d expect jobs to fail across platforms, which was not the case. So I started investigating with Python 3.11 from the official installer on my machine and discovered that Austin was picking the wrong binary. This prompted me to check for symbols in the binaries, as failing to find those would cause Austin to potentially pick the wrong binary.
Now I’ve started using the official installer because of the need to investigate these issues. I’m more familiar with the Pythons installed via pyenv, from which I expect something like
❯ nm -gU `python3.9 -c "import sys; print(sys.executable)"` | grep "_PyRuntime$"
00000001002e6780 S __PyRuntime
The same command for Python 3.11 from the official installer returned nothing. This is “fine”, as we know we should also look for any potential shared library. For some reasons I was convinced that
otool -L /Library/Frameworks/Python.framework/Versions/3.11/bin/python3
had given me /Library/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python in the past, but I have just double-checked, and I can see that it actually links to /Library/Frameworks/Python.framework/Versions/3.11/Python. Indeed, as reported by @ronaldoussoren, I can also confirm that the symbols are there. Furthermore, it seems that the behaviour of the installer hasn’t changed, contrary to what my initial investigation led me to conclude (whence this discussion), as confirmed by @guido.
Where we stand now
To summarise, we now know that there doesn’t seem to be changes in the MacOS installers when it comes to symbols. Is the CI failure an issue with Austin then? I re-downloaded the installer used in the CI job that was used to verify (successfully) the 3.4.1 release commit, used it to re-install Python 3.11 on my machine, checked out the v3.4.1 tag, built Austin from it, removed the signatures from the Python binaries, and I can reproduce the current CI failure!
At this point, my next step for me is to understand why, all of a sudden, Austin is failing to find the shared library, that it probably used to find before, with the same exact target binaries. I hope this also shows why my thought after my initial investigation was that something had changed in the installer, whence this messy discussion.
I hope this is, as usual, something super-silly that I’ve overlooked, rather than a peculiar MacOS time-bomb that I don’t know of, because then I can hope to find the problem and fix it! If anybody has any thoughts they’d be more than welcome, and I hope this all clarifies the situation now.
1 Like
Update
Increasing the timeout before Austin gives up trying to find a binary with symbols seems to “cure” this issue, at least locally. It still remains a mystery to me why all of a sudden there is the need to wait a bit longer for the binary maps to be laid out in memory. The CI job went from “reliably passing” to “reliably failing” with no changes from Austin. This is a rerun of a job that passed, and that is now failing. The only thing that could have changed here is the result of the setup-python action.
1 Like
Thanks for the explanation! All I can do is wish you luck in understanding the timing issue.
2 Likes
Ah, I think I now know what’s happening! The CI started failing when the macos-latest runners have been updated from macos-11 to macos-12. I suspect this means that Austin is now being compiled with a more recent version of gcc which makes Austin slightly faster as a result of the O3 optimisations. This means that Austin gives up looking for the right binary slightly earlier (the loop is based on number of iterations rather than a timer). I think this might also/alternatively mean that 3.10 and 3.11 take slightly longer than older versions to initialise on MacOS 12.