We’re talking about a JSON file here. Surely if you’re able to parse JSON, you must be able to handle the version_parts object?
1 Like
My usual baseline is whether it’s useful with Bash+jq, so that you can use shell scripts to do useful things with the file. I’m not sure what it’s actually capable of here, but you don’t have to do any better than extract all 8 characters and be able to sort them (case-insensitive ASCII sort of a hex number should match a numeric sort, provided they’re the same length).
(Personally I mostly use Powershell, which can do a rich Version object, though it’d still be more convenient to pull out a ready-crafted hex string.)
1 Like
I don’t think there’s a good definition on the separation between the “Python language” and the CPython implementation. Perhaps it is indeed a mistake to define the fields like this.
Because it’s meant to mimic sys.implementation, just with an escape hatch to allow implementations to remove or add fields as they think it is needed. Ideally, for most implementations this should just be as straight forward as providing sys.implementation as a JSON object.
Regarding the inclusion of site-packages paths. While I understand that it would be nice to include them in certain situations, I think it would be a mistake to do so in this file.
Any usage of these paths will be inherently broken and/or incorrect when not targeting the default environment. While I don’t think this proposal needs to target virtual environments, we should try to avoid designs that can be very easily misused.
I feel strongly that we shouldn’t mix the information regarding the core interpreter and (default) environment in the same file — it should be separate.
Because of this, I am willing to submit a complimentary PEP in parallel, to standardize a static file to describe Python environments. In practice, that would mean that additionally to the interpreter description file we are discussing here, there could be one describing the default environment (eg. environment.json).
1 Like
Well, for example PyPy 7.3.15 provides an interpreter supporting “the syntax and features” of Python 3.10. You probably want to be able to state both the PyPy release number (7.3.15) and the Python compatibility level (3.10) in your description file.
As for CPython, all releases in the 3.10.x line support the Python 3.10 language and standard library features.
2 Likes
I agree, but I’d argue this only applies to virtual environments being separate. The core interpreter will only have a single two site-package directories (including the user one, which can’t easily be specified here) and they are indeed static. Virtual environments are the special, under-specified case.
Personally I wouldn’t waste effort on trying to standardise virtual environments [in this PEP]. Getting all the various tools that create them these days to align is way too much work - let Brett Cannon work on it 
But I really do think we should be able to infer the static parts of the default sys.path from the static file. I actually have a case right now where an x64 tool is installing ARM64 packages into a different site-packages directory that will eventually be deployed to an ARM64 machine. “Guessing” that {prefix}\Lib\site-packages is the directory is fine, but being able to read that out of a configuration file just makes things more robust.
I also have another case where I need to “guess” the executable name and patch some dynamic libraries before launching Python. Right now, I parse patchlevel.h with a regex and guess, but would much prefer to just pull the (relative) locations out of a static file. These involve stdlib/platstdlib rather than site-packages, to be fair, but it’s only a matter of time before I have to patch pre-installed packages as well and then I need the third path.
1 Like
In PEP 514 I deliberately separated Version from SysVersion, where the latter is x.y[.z] of the language rather than the interpreter release.
We’re close to that in the current PEP 739 text (I lost track if it’s been changed already), but it probably makes sense to formalise language.version as being “what sys.version would say” and implementation.version as “what your download/installer would say”. Hex version then goes under language.version, though for CPython at least the versions will match.
(Again, I have an actual scenario where I’m repackaging a modified CPython under a different version number, and so users/tools need to be able to check these two versions independently.)
3 Likes
Huh? The stdlib venv module is the standard here. Virtualenv uses the same layout as that (as far as I know, it uses venv under the hood). What other tools are creating virtual environments that cause problems here? I’d go as far as to say that venv is the standard, and anything wanting to support virtual environments should support the venv layout and treat anything else as out of scope. I’d be happy to accept that the layout of venv is currently under-specified, but that’s fixable.
Having said that, I agree that it’s perfectly fair to ignore virtual environments for the purposes of this PEP.
2 Likes
I meant “standardise” for the purposes of this PEP, not in general. Not going anywhere near that proposal! 
I’m still not 100% clear what you’re saying. As far as I can see:
- We’re in agreement that this proposal shouldn’t be concerned with the layout of virtual environments.
- It’s simple to determine if an environment is virtual - does the root directory of the environment contain a
pyvenv.cfgdirectory? - While not relevant for this proposal, if the environment is virtual, the layout can be assumed to be as per the stdlib
venvmodule. There’s not currently a static description of that layout, but it’s defined by the return value ofEnvBuilder.ensure_directories, and thevenvscheme insysconfig.
Do you agree with that, or do you have some reservation that I’m not understanding?
I think what’s happening is the following:
- people are saying it would be nice to have the
site-packagesdirectory in the static interpreter description file addressed by this PEP - I feel strongly against it, so I have proposed writing a parallel PEP for an
environment.jsonfile if it gets people to drop that point in this discussion - from the feedback I get the impression that people don’t want to open that can of worms
2 Likes
I don’t think it’s that simple 
. The interpreter also looks for a pyvenv.cfg in the upper directory, then there’s the whole PYTHONHOME situation, and I’d expect a few other quirks in reality.
Regardless, this is out of scope for this PEP.
1 Like
My understanding was that your objection (to including site-packages) was the complexities around how we handle virtual environments. I’m saying that I think it’s fine to just ignore them.
I still want to argue for site-packages, although I understand it’s complicated (particularly because people will immediately say “what about the include and scripts directories?”). If you insist on making that a separate PEP, I don’t want it to include virtual environments, or at least I think that virtual environments can be handled trivially[1] by just writing a JSON file containing context.__dict__ alongside the pyvenv.cfg file, and I don’t want to over-engineer a solution for the virtual environment case.
From pip’s point of view virtual environments simply aren’t an issue[2], and I don’t want to end up debating a standard that might make them an issue.
In part, yes, but not fully. I don’t think this file should include anything regarding environments, being it the default one, or virtual environments. The complexities that handling virtual environments bring are just an example of why, it is a different problem, with a totally different set of requirements and complications around it.
If people are so pressed on having the site-packages path in a static description file, then just put them in another file (eg. environment.json). As I said, I am happy to write and see to conclusion a parallel PEP for that.
I guess, essentially, what I am saying is that I don’t think it is okay to simply ignore virtual environments. I think is okay, even the best option, to ignore environments.
Sorry if this is sounding too adamant, it’s hard to portrait tone over text. I feel strongly about this, as I have said before.
2 Likes
Yes, apart from “if the environment is virtual, the layout can be assumed …”. I don’t think you can assume anything about the layout in this case, unless you know that it was created by venv (in which case, it’s no longer an assumption).
To me, “virtual environment” just means “context-specific search paths”, and I use venv to specifically mean when it’s been set up by our module. And “context-specific” I think clearly implies out of scope for this proposal.
I would argue that what we’d be ignoring here is static information set by the site module. It’s not an environment, it’s a property of site, which is as much in the core runtime as sysconfig or sys. It doesn’t have to be “environments” at all, it can just be “a property of a standard module” like the rest. (Edit: More specifically, the result of site.getsitepackages() with default prefixes argument should be entirely static to a particular runtime.)
(I’m less trying to argue with you and more trying to feed you justifications that you would be comfortable using when people try and scope-creep into a full environment definition.)
If it doesn’t get into the file, I’m sure we’ll just keep hard-coding {prefix}\Lib\site-packages or {stdlib}\site-package where needed. I don’t have a need for more information about the environment than that, and would probably hesitate before reaching for a file that might be intended for more dynamic information.
1 Like
Ah. I’d claim that’s a very unusual definition to use, which is why we’re failing to communicate. I think the norm is for “virtual environment” to mean the environments defined by PEP 405, with the stdlib venv module being the official implementation of these.
This is my view, as well. I don’t think of the debate here as being about “including environment definitions”, but rather “being able to get at static interpreter information without running the interpreter”. Things like site-packages are very much “static information” in that sense. It’s incidental for the purposes of this PEP that it’s some sort of “environment path”. Yes, from pip’s point of view I can see ways that we might use that information when inferring the structure of an environment, but that’s entirely separate. That’s why I say “ignore virtual environments” - the consumer can sort out that sort of thing, as long as the static information is available.
To be fair, though, I don’t feel that strongly about this. It’s mostly just that I’d be disappointed at missing an opportunity to do some things in pip more cleanly. So I don’t want to pit my weak preference for this against your strong dislike of it. As @steve.dower says, we’ll probably just keep hard-coding stuff if needed[1].
we’d have to do that for backward compatibility for quite some time, anyway ↩︎
4 Likes
FWIW, one use case I’d love to see enabled by this is being able to point pip at this file with a --target-like option and have it install the right version of the right platform into the right directory for a runtime “install” that won’t run on the current machine. I’d probably even contribute it, that’s how much I’d like it
2 Likes
Yep, that’s the precise use case I see for this information.
4 Likes
That is what I desire the most and I would be willing to contribute as well.
I share the same sentiment, but doing that with the file proposed here would not fix the issue properly. You would either have to special-case default vs non-default environments, or simply not support non-default environments at all. While that would be nice for the use-cases that only concern the default environment, it would create yet another “it works, but not quite so” situation, which, as you probably know, is quite frustrating both for us, as maintainers, and for users.
Anyway, essentially, what I am asking here is just: let’s put the build information on a build-details.json file, and the environment details in a environment-details.json file. I am not fighting against providing the environment information, I am just asking for that to be separate.
I realize that perhaps the PEP title perhaps expressed the wrong vision for the file, and that’s on me, so I am gonna change it to “Static description file for build details of Python installations”.
We can then work on a PEP to standardize a file format for describing Python environments. This file could even have a key pointing to the “build details” file, as that is one aspect of the environment.
1 Like