PEP 582 - Python local packages directory

Seems like it is discussed in the PEP:

Maybe related:

The only point that is discussed in the PEP is that /tmp/__pypackages__ will not be used if running a script from elsewhere.

But this implies it will be used if not running a script, and that’s exactly the usage that is described as vulnerable in the CVE.

3 Likes

Well, I don’t think this PEP opens new vectors for this sort of thing. As you said, sys.path already makes overriding something like encodings to execute arbitrary code an issue if your file system access is already compromised. If it’s truly a concern then I suspect the PEP would mention it doesn’t make anything worse than it already is.

But maybe the PEP just needs an update to the latest PEP template which has a security section anyway.

I will try to update the PEP this weekend (or next) if I manage to get time from all doctor visits.

2 Likes

I agree, it’s within Python’s existing security model, though I think users still need to be cautioned about that model, as Glyph is doing in this blogpost.

I tried IPython 8.0.1 and I was surprised that it arranges for the current working directory to be put towards the end of sys.path instead of the start, so it is harder for the context of the current working directory to break/compromise installed packages. An encodings.py or a sitecustomize.py does not get executed on Python REPL start. An attacker could add a requests.py and hope that somebody runs it in a context that isn’t installed, though it might be better to write a StringIO.py and hope that somewhere in your import graph there’s still a leftover

try:
    from StringIO import StringIO
except ImportError:
    from io import StringIO

So it appears that IPython is trying to improve upon Python’s security model, and maybe the CVE is valid if that’s the case (but this seems like a lost cause; I would dispute the CVE).

Returning to PEP-582, at worst it adds more directories and paths to worry about when considering the security of an application. Maybe pip could audit this a bit, e.g. warn about __pypackages__ existing in a location where it is writing console_scripts/gui_scripts but which is not the directory in which it is writing package contents. I guess a logical extension would also be to warn about or refuse to install console_scripts/gui_scripts that end with .py.

2 Likes

On the security side, as long as the “Isolated Mode” behaviour of local packages support is sensible, and a local packages directory doesn’t affect running scripts outside the directory from a working directory outside the directory when running in a normal mode, then the ability for local packages to override system ones will be a feature rather than a bug.

The better argument for putting the script (or working) directory later in sys.path in general is a beginner-friendliness one: if you call your “experimenting with the socket module” script “socket.py”, your “import socket” line no longer does what you expect. Ditto for “experimenting with numpy”, etc - it’s natural to name experimentation scripts after the library you’re experimenting with, and the default path configuration means doing so not only doesn’t work, but fails in a cryptic fashion.

The “multiple platform specific local environments or only one generic one?” question is a thornier one, as the trade-off changes based on the use cases you’re attempting to support. A fully general solution would have three tiers of environment in the local packages directory:

  • a common platform and Python version independent environment with only pure Python and stable C ABI based extension modules in it
  • a Python version specific environment with pure Python and C extension modules in it (extension modules have their own mechanisms for parallel installation of extension modules for more than one platform)
  • a platform compatibility tag specific environment that aligns with the way wheel distribution files get named and installed

PEP 582 intentionally simplifies this topic by offering only the middle Python version specific tier, as it best reflects the way that system level and user level package installations work in the absence of virtual environments. What it doesn’t point out (and probably should make explicit) is that starting with only that version dependent but platform neutral environment doesn’t lock us in to only supporting that tier forever - the version independent environment and the platform specific environments could be proposed by later PEPs, citing the specific use cases that the simpler starting point didn’t address.

1 Like

I was wondering why is this not approved yet? what can we (the community) do to push this to be approved?

Most of the popular languages uses this approach it also can help to stop using virtual environments so that efforts can go to other projects that will provide more value to the community. So I don’t see any downsides here only benefits to everyone

1 Like

The discussions (AFAICT) have petered out but I don’t know where things were left. OTOH, PDM supports PEP 582 and I’ve enjoyed using it when developing my packages.

What can we do to make it gain traction and make it to the next level ??

I see that you are a core dev, probably you saw similar PEPs or you know better what can be done ??

There were enough concerns raised (and push back) on this (and other channels) that the original proposer (@steve.dower ) decided to step away from driving it ahead. For this to become a thing a new champion needs to step up, go through the threads, address the concerns (start a new round of discussion) and submit it to the council for approval. This is going to be a long and heavy lift and no guarantee that it will be accepted. However, we eagerly await for someone to fight the righteous fight for the rest of us.

2 Likes

None of the authors have sent it to the SC to consideration.

There will be some costs to the packaging community (e.g. pip) in order to support it.

@kushaldas ?

2 Likes

I am again working on this, I will push an update to the PEP next week. Need to update the examples before that. Hopefully by next month I will be able to pass this to SC.

FWIW I still think PEP 582 is a bad approach, for all the reasons we talked about before. Do you think there new developments that change that calculus?

I think the general community (tooling) support for the PEP even though it isn’t accepted and people continuously asking for it suggests it’s pragmatically good enough for folks. Otherwise we will have to see what the PEP update contains.

5 Likes

I wonder if @frostming can provide some insights given that PDM has supported it for a while now. FWIW, I find it quite handy for development.

I think there’s still a need, and the fact that no-one has come up with an alternative approach in the time PEP 582 has been on hold suggests that we should at least accept that the choice is more likely to be between PEP 582 and nothing, and not between PEP 582 and “something better”.

Personally, I’ve become more supportive of PEP 582 for exactly that reason. I want to look at the details again, but I’m inclined not to let the perfect be the enemy of the good.

I can’t say I’ve noticed a bunch of tools that let me use a local packages directory. Did I miss them?

3 Likes

I know of PDM and PyFlow off the top of my head. My (potentially faulty) memory may also be stemming from the fact that every tool I am involved with that has anything to do with environments has been asked to support __pypackages__. So either PDM is way more popular than we all realize or something else is supporting it that I can’t remember off the top of my head that makes folks ask for its support.

1 Like

I think it’s just familiarity with other ecosystems like JS.

If this gets accepted, people will start committing the directory to version control systems like Git on day 1, which scares me greatly.

3 Likes

Is that somehow worse than people committing virtual environments? You’ll find plenty of site-packages and pyvenv.cfg files as well on any major public version control hosting provider.

4 Likes

No, but now standardized rather than discouraged so will be far more common and not (mostly) accidental.

2 Likes