Allow for overriding the default environment

Would folks here generally be open to changes in packaging/src/packaging/markers.py at 4d8534061364e3cbfee582192ab81a095ec2db51 · pypa/packaging · GitHub such that an environment variable of JSON could determine the default environment? We have a great need for this at work to allow for cross compiling/resolution. Specifically, we are able to successfully cross compile using Zig but we are unable to resolve dependencies because of Various package-index filtering flags do not affect the environment markers · Issue #11664 · pypa/pip · GitHub. I think, however, even if that bug is fixed it would be useful to have a centralized place to explicitly define the environment since packaging is used by everything.

I’m willing to provide the PR of course :slightly_smiling_face:

1 Like

I would think you need a longer name then just JSON that make the meaning clear.
Given that its for default environment then name the variable after its use not its encoding like: PYTHON_PACKAGING_DEFAULT_ENVRONMENT?

Sure, I was just talking about the format of the content not the name of the variable.

1 Like

I’m not massively enthusiastic about the idea of an environment variable containing a JSON value. My main objections are usability and global impact.

The usability issue is that setting such a variable is annoying - getting the quoting of a JSON value correct is really fiddly, especially of you’re working in a shell you’re not familiar with. Also, Windows doesn’t have the “temporary environment variable” feature from Unix of ENV_VAR=xxx cmd, so an environment variable that affects a single command is much harder to use.

As far as global impact is concerned, I don’t like the idea that setting something in your environment can change the behaviour of any tool that uses packaging, even those you didn’t realise used it. I can easily imagine bug reports along the lines of “pip didn’t install my dependency”, which turn out to be because the user set a different environment via a variable they’d forgotten. Such “action at a distance” makes for difficult-to-track bugs.

Having said this, I am in favour of having an API in packaging that can be used to configure the default environment, so that there’s a way for applications like pip to change it without needing to pass an explicit environment to every call that evaluates a marker.

Would you instead be willing to provide a fix for Various package-index filtering flags do not affect the environment markers · Issue #11664 · pypa/pip · GitHub? It seems like that would be less controversial, and from what you say would also address your issue.

4 Likes

Yeah, I’d vote for using the environment variable to point at a JSON file containing the info (and possibly there’s alignment here with PEP 739?).

I would also very much like a way to override the platform tag that packaging chooses for the current system. I’d love to be able to embed a configuration file in a Python environment that says “instead of win32, pretend this is my_special_platform” (and then point at a private package index to get perfectly matched custom builds). If you’re going to go for a big change here, please keep going :smiley:

5 Likes

Yeah, I agree with Steve.

Also, If someone wants to write a PEP for this, I am open to working together.

1 Like

Yes this is a much better idea and I would also prefer it!

I don’t have time for that and I basically know nothing about the pip code base. Also, I would like more feedback here before considering this controversial :wink:

2 Likes

As a maintainer of packaging I don’t love the idea. This feels like a hack to work around other tools that don’t let you specify the environment details explicitly. That function (which we are working on documenting) just gives you the running interpreter’s details, not the way to get arbitrary details of another interpreter.

I’m not quite seeing why packaging is the expected coordination point of passing around data. Isn’t that what function arguments or instance attributes are for in your code? :wink:

1 Like

@brettcannon Are you in favor of this?

Not really because now we are just a data store at some module scope that anyone could have done themselves. Toss in that we would then have to manage threaded access, etc. and it seems like an increase in scope for the package (I don’t think we have persistent state anywhere right now, even when people complain about performance and ask for it to work around pip).

But I also like functional programming, so passing state around doesn’t bother me.

You can also see if @pradyunsg disagrees and/or try to sneak it in while I’m out in April. :wink:

2 Likes

Fyi windows cmd does its done like this:

setlocal
set ENV_VAR=xxx
command
endlocal

This is why an environment variable or config file into packaging makes more sense than pushing it out to every tool - you only need to pick up the user-provided setting once, and it becomes the global constant, rather than the code it currently has to calculate the value from ambient information.

1 Like

Does that work on the command line or only in batch files (I thought it was the latter)? Although it’s only of mild interest as I use Powershell.

Sorry, I know this is offtopic.

Not tried as an interactive commands. Cmd.exe is hard to use outside of batch files… To really see scary stuff have a read of for/? to see what the enterprising nerd can do in cmd.