TLDR: We want to make the default build of CPython more relocatable, as a 3.16 feature, with the changes landing in the default build and not behind a build flag. A fully relocatable install may still need some post-install massaging, but less of it. Any concerns?
We’ve been working towards prebuilt relocatable CPython, and the upstream changes so far have been plain bugfixes. The next ones in the pipeline change what make install produces, so before opening PRs we wanted to check that the team is OK with these landing in the default build (as opposed to behind an optional configure flag).
Two concrete examples:
Making the installed PEP 739 build-details.json use paths relative to the file instead of absolute (draft impl). PEP 739 already allows both forms, and consumers are expected to handle both.
Writing prefix=${pcfiledir}/../.. instead of the absolute prefix in the installed python-3.16.pc and python-3.16-embed.pc (draft impl). Both pkg-config and pkgconf support pcfiledir, and this is how Conda and Homebrew ship their .pc files.
These changes are visible to anyone who looks at the installed files, but nothing should break for users of the default build. We’d rather avoid adding a new dimension (relocatability on/off) to the already large CI and buildbots test matrix. A flag would also mean two layouts for consumers to deal with, and the relocatable one would be the less tested one.
I’d say JFDI too, and I’m +1 but I think it needs a (hopefully short) PEP. I’d expect it to be largely uncontroversial, but I think it’s worth capturing the details, decisions, and possible visible effects in a PEP.
Writing prefix=${pcfiledir}/../.. instead of the absolute prefix in the installed python-3.16.pc and python-3.16-embed.pc (draft impl). Both pkg-config and pkgconf support pcfiledir, and this is how Conda and Homebrew ship their .pc files.
These changes are visible to anyone who looks at the installed files, but nothing should break for users of the default build. We’d rather avoid adding a new dimension (relocatability on/off) to the already large CI and buildbots test matrix. A flag would also mean two layouts for consumers to deal with, and the relocatable one would be the less tested one.
I would like to caution that ${pcfiledir}/../../ for cases where it
canonicalizes to /usr, is an actual issue. The pkg-config tool will
get confused and stop deduplicating system directories, and you can get
some fairly weird results.
e.g. Meson has a global option for this (it is automatically available
for all packages that ship pkg-config files):
Users can ask for a relocatable build (and perhaps you’d only test that
mode in CI), but users who don’t need them and aren’t planning to
relocate them, for example Linux distros packaging the software for
installation into /usr, can still choose to get artifacts which play
nice with installation into the system directories.
A flag to change the pkgconfig file does not necessarily imply a flag to
change the way the interpreter is compiled and installed.
I think that’s premature for relocatable builds in general and overkill in this case. We aren’t introducing some new mechanism or semantics here, we just want to take advantage of stuff that already exists that we happen to not be using. I view this more as us saying “we plan to do this unless there’s a specific compatibility issue we aren’t aware”. That feels like it falls under core dev EAFP discretion instead of LBYL SC approval via a PEP.
Well, that’s annoying. Thanks for the info on that!
Not directly, but it would still be a part of ./configure which collects all the flags that control the final output of the files. In this instance we would still need to add a flag or do post-processing on the .pc files to make things work.
Sure, that’s fine if there aren’t any significant user-visible changes that have to be debated, or a transition planned for. I heard that there might be, but if that’s not the case, then cool.
I personally haven’t really noticed any downside to things like python-build-standalone, or any functional differences between it and say homebrew or built-from-source Python, so it’s probably fine for the vast majority of users.
The people it catches out is going to be the Linux distros (or individuals who build packages meant for a distro), I expect. A PEP is a convenient way to document it in a way that says it was considered and not merely decided.
Making it the default seems fine to me (even good, it makes multi-stage CI work better), but hopefully the build option to match the current build is still there and fairly obvious (presumably setting --prefix to any non-default value would do it?). Or maybe all the necessary post-processing can go into make install?
If possible, it would be nice to have a configure option to decide if the built Python uses relative paths (relocatable) or absolute paths (what we have currently). For a Linux distribution package, I think that it would feel more comfortable with absolute paths (no change).
If you use relative paths, how do you deal with virtual environments? Paths are relative to which Python executable?
This would be fine,[1] there’s actually very little in Python itself that relies on the paths being fixed. getpath.py has basically the same logic for Windows, which has always had relocatable binaries. The greatest risk here is that there may be new ways for the “search for argv[0] in $PATH” logic to break, and the $PREFIX/bin/$PROGNAME fallback stops making sense, but once we’re able to get the full path to the running executable, everything else is fine (which on Linux is more complicated than other platforms, but I suspect there are easier options here than 20-30 years ago and so trying some of those may be a good move).
The real problem is locating OS dependencies.
I think you get to choose whether the PEP is “here’s the planned outcome and we’ll do whatever is needed to achieve it” (like my PyManager proposal) or if the PEP is a full plan with every detail and no scope to change (like most packaging specifications). Personally I think the former is preferable here, as it’s far more reasonable to expect the steering council to review and approve a direction rather than all the details, and you’ll want to keep the details flexible so they’re governed by our usual compatibility guarantees rather than specification guarantees.
Paths are always relative to the current process. ↩︎