I am the maintainer of a project called Austin, an observability tool that samples the frame stack of a running Python process for profiling purposes. Lately the CI job on MacOS for Python 3.11 started failing, and I discovered that this is due to the fact that the GitHub Action is making use of the official installer for MacOS starting with Python 3.11. The reason why this is a problem is because the official installer puts the binaries in an area of the file system that is protected by the System Integrity Protection policy. Observability tools, like the one that I maintain, are in general not capable of working around the SIP, even using sudo. As a consequence, those that rely on the official installer to run their Python applications, cannot make use of these tools, and have to install Python through other means (brew, pyenv, conda, …).
Hence, I was wondering if it would be possible to customise the installer to allow for a different installation prefix. If not, would it be possible to install Python somewhere else that is outside of the SIP protection?
I don’t think the installer is the type of packaging this group manages. That said, this seems to be an issue with the setup-python action, which is maintained by GitHub. Have you opened an issue with them?
The Python.org (“official”) macOS installers currently only allow installation to a fixed location. Allowing relocatable installations is a desirable feature that is something we’d like to provide. I’m not sure there is an open issue for it at the moment. The place to track that would be here. As David notes, you could also bring up the change with GitHub.
While the Packaging category (and the group of people participating under it) does not manage the Python installer, this particular problem has actually been discussed quite recently since the fixed prefix nature of the macOS installer (and the build system for POSIX systems in generall) is problematic on multiple levels for Python packaging: Creating a standalone CPython distribution
Is the problem due to SIP or due to the binaries being signed and using the hardened runtime on macOS? You can test if the latter is the issue by removing signatures (e.g. codesign --remove-signature ...), if that fixes your problem the issue is likely the hardened runtime.
System Integrity Protection includes protection for these parts of the system:
/System
/usr
/bin
/sbin
/var
Apps that are pre-installed with OS X
These paths become read-only and… well, I doubt any Python install would install to those paths since, well, you can’t install things to a read-only filesystem.
Ah, I didn’t know about hardened runtime! I thought it was all SIP. That would make sense and I’ll give it a test soon. But this would still leave the permission issue, i.e. one cannot opt out of signing I believe?
It depends… There’s two parts of signing: For getting a nice user experience with the installers we need to sign them using a Developer Certificate, otherwise users will have to jump through hoops to even start the installation. To sign the installer you also have to sign the binaries installed by the installer, and that requires using the hardened runtime which makes some runtime introspection (like debugging) harder.
After installation the story gets a bit muddier. For arm64 binaries must be signed, but that can be done using an ad-hoc signature and doesn’t require using the hardened runtime. For x86_64 binaries signing isn’t required at all.
When I want to use Xcode’s lldb debugger with a Python.org installer I generally remove the signatures from {sys.prefix}/bin/python3 and {sys.prefix}/Resources/Python.app/Contents/MacOS/Python and then resign these files using codesign -s - ... on my M1 laptop (the latter stap isn’t necessary on an x86_64 system).
If stripping signatures fixes the issue for you the nicest solution for your users would be to look into the entitlements needed in your program to allow inspecting binaries, possibly the debugging tool entitlment. That’s not something I’ve seriously looked into for my own projects.
I was totally oblivious to all these details as my development experience on macOS is very limited. Based on what you’re suggesting, I think it should work for me:
Apps with the Debugging Tool Entitlement can call task_for_pid()
I can confirm that removing the signatures from both binaries works (it seems that removing the signature on the Python object file is enough). I couldn’t get the self-signing to work though. I have followed the steps in Steps to get GDB actually working in April 2021 on macOS (Intel x86-64 only) · GitHub, but that didn’t seem to work . Perhaps I’m missing some steps that an Xcode project would take?
There’s a scripted version of the cert creation process in one of the last comments, which may be an easier path for command-line use.
Unfortunately, that process contains a typo (or the instructions are for an outdated version of the recommended script), so even if you used those steps things might go wonky.
Where the instructions say,
# download the script
https://github.com/conda-forge/gdb-feedstock/blob/main/recipe/macos-codesign/macos-setup-codesign.sh
# replace the certificate name
sed -i 's/gdb-codesign/gdb-cert/g' macos-setup-codesign.sh
The actual cert name in the file is gdb_codesign, so that sed command won’t do anything as-is (or if you only edit the replacement strring), which means it’s not actually replacing the cert name in the script, so running it will still name the cert gdb_codesign and you might be trying to sign with a nonexistent cert by some other name.
Hi there
. Perhaps I’m missing some steps that an Xcode project would take?