One has a concrete proposal, the other doesn’t (yet?).
Like I said before, I’m not interested in going in that direction, because I still think the declarative approach outlined in 829 is the right way to go. But I also won’t discourage someone else from proposing an alternative and seeing where the chips land.
Fine, I’ll turn my proposal into a PEP.
But honestly, a problem here is that I don’t see a way to advocate for “do nothing, all of this simply isn’t worth the upheaval”. Which is really what I think[1]. My gut feeling from this thread is that there isn’t a huge amount of support in the community for the proposal - but I’m very aware that who posts in this type of thread is a very unbalanced cross-section of the community. On the other hand, I know of no way of getting a sense of what the SC might think of the proposal, so I’m left in a position where I feel that I need to submit a “damage limitation” PEP that explains how to get most of the benefits of PEP 829 without the disruption of a new file format - particularly if you’re using the argument that PEP 829 is the better option because it has a formal proposal…
With the proviso that I believe that deprecation of “extra code on the import line” in .pth files could have been achieved with nothing more than the normal PR process, if we hadn’t already escalated the debate into PEP territory. ↩︎
If we really need to write a counter PEP just to say no to this one then here you go…
Lets leave .pth files alone.
Breaking stuff without good motivation is unhelpful.
We will make the following changes:
No changes being proposed actually fix anything wrong with .pth files (which in itself, no-one can define) or provide any new functionality that we have any use for.
None. Not even for the use cases we didn’t even know about.
Not doing anything that could be misconstrued as a security feature is a security feature!
Python users who don’t know or care that we’re having this discussion may carry on doing so.
See any semi-recent Python version.
Descoping existing .pth files so they can at most, import another module with side effects to achieve the same thing
Why bother? It achieves the same thing.
Turning .pth files into a structured config file
Same again.
(Sorry, I couldn’t help myself 
)
It seems like this alternative should just be a “rejected idea” in the pep, with an actual explanation why it wasn’t the preferred direction. The SC shouldn’t need a competing proposal just to see that argument.
That’s really the main tension I see, and that’s why I’ve been asking about the intended use cases: We want very different capabilities for external packages vs. the user’s own customization. Not even from a security standpoint, but for package interoperability and because tools need to support this, where .pth files are a problem today (that’s why I’m engaging here, I care the user experience in the interaction of these features with tools)
My experience is that these custom formats cause all sorts of problems (we certainly have them with WHEEL, METADATA and PEP 508), so I’d prefer an existing, structured format.
I agree. If the argument is, “reject this PEP for these reasons”, that should be covered in the Rejected Ideas section. Same goes for a more restrictive .pth file format if people think that doesn’t require a PEP or don’t want to write a PEP unless the SC says they prefer that approach.
@pf_moore and I had an actual conversation (imagine that!). It was quite pleasant and I think we may have some common ground. I’ll work on an updated 829 in the next couple of days.
Yep, exactly. The trick is that the narrowed syntax would be valid on all existing versions, so affected projects would just need to migrate to that narrowed format before the deprecation period ran out. No need to ship two different formats, just stop using the deprecated constructs (replacing them with an importable callable).
In the case of legacy namespace packages, it would potentially be feasible to migrate over to native namespace packages and drop the associated .pth file entirely.
I have published an update to this PEP based on various feedbacks. Big thanks to @pf_moore for hopping on a video call with me to work out details.
The big change is the removal of <name>.site.toml files, a deprecation of import lines in .pth files and the introduction of a separate <name>.start file specifically and exclusively to name entry points to run. Change History.
Nice, I like this version. Allowing arguments to the entry points could be an interesting extension, but there is nothing preventing us from adding that later if we decide it would be a worthwhile enhancement.
Minor technical nit: to allow for the smooth migration of import lines without redundant existence checks, the scan for *.start files has to happen before the *.pth files are processed (so the runtime knows whether to emit warnings or not).
I still think it’s misleading to have any mention of the word security in this PEP outside of a statement along the lines of this is not in any way a security feature.
And given the frequently sited incorrect statements like Recent supply chain attacks have used arbitrary code execution in .pth files as an attack vector, IMO the most counterproductive thing we can possibly do here is mislead people.
The main aspect that makes this version a genuine security enhancement is splitting the file types: once the import line processing is gone, you can tell from the file listing whether a package injects startup code or not. Both the status quo and the initial TOML proposal require looking at the file contents to determine that, making it less likely that scanners will do it.
Creating the opportunity for more fine-grained startup code permissions is also a genuine benefit (even without a new dedicated policy layer, the updated startup code execution mechanism creates patterns that are more readily identifiable in audit logs)
Shadowing the standard library from a package without assistance from some other form of startup code injection isn’t trivial due to the way we order import paths. Deliberate naming collisions between third party packages are also a different flavour of attack that the interpreter itself can’t really help with resolving.
Does this PEP eliminate the security problems associated with startup code? No, those problems are inherent to the functionality. Does it make managing some of those security problems more tractable? Yes (which is enough to justify the stance taken in the PEP)
I don’t see this as any better for security. It is better for auditability. Those two things may be related, but this is still a package can set things up so that arbitrary code runs at interpreter startup, and the security boundary is still installing a package, so being clear on that difference seems like a good idea.
Just a couple thoughts on the timeline and dual-shipping startup code.
I’m curious how many people will prefer to dual-ship *.pth and *.start during the deprecation period, as opposed to simply switching to *.start at the end of it.
With a three-year deprecation period, it will be required to dual-ship, as 3.14 and 3.18 will be live at the same time. Further, 3.18 will warn because of the presence of the lines necessary to support 3.14. With a four-year deprecation period, you can either dual-ship and then remove the import lines after 3.14 is end-of-life, or else transition with a release immediately after 3.19 is out. With a five-year deprecation period, you can simply keep your files as they are, and then transition after 3.14 is end-of-life.
I would prefer a five-year deprecation, which would give package maintainers a bit of breathing room to be active but not have to time a release to match Python 3.19 to avoid bug reports from users. (Many will start getting the bug reports when the alpha or beta releases start getting tested by distros.) I would be fine with a four-year deprecation. Three years seems aggressive, to me.
Speaking for the editables project, I’d probably just switch “at a convenient time”. Which could well be when import lines in .pth files are finally desupported - I see no compelling reason to switch earlier. Given that dual-shipping means duplicating the information in the .start file, I’d prefer to do that for as short a period as possible, all other things being equal.
This is what really bothers me. I don’t think it’s reasonable to put projects that wish to support more than 3 versions of Python at once in a position where they have no way of doing so without warnings. Remember that these warnings will be seen by end users, who are not the people who have the ability to fix the problem. This is a well-known problem with using the warnings system for deprecaions, they target the wrong people.
I’m fine with a 3-year deprecation period, if the warnings are removed (specifically, the warning about import lines in .pth files being ignored) or made opt-in, so that projects can see them in their testing, but end users don’t see them. With the warnings as proposed, a 5-year deprecation (long enough to cover all non-EOL Python versions) is probably sufficient.
I could agree with all of this only if .pth files were the single way to hijack startup but that’s not the case.
I’ve already demonstrated in the other thread that the standard library can be overshadowed by entrypoints and by bundled resources. The common 3rd party library numpy.py[1] trick is very practically exploitable even if it can’t catch every invocation of python. And I’m sure someone with more imagination than me can come up with other ways.
Checking a wheel contains no files matching *.start makes a package no safer to install than it was before. We mustn’t lead people to assume that it does.
which takes precedence over numpy/__init__.py without creating any giveaway file conflicts ↩︎
Well, “we should only fix a security problem if we’re sure it’s the last security problem of the kind” doesn’t sound like a very productive line, IMHO.
That seems like an unfair reading–I don’t see an objection to the proposal itself, just to the framing.
The middle ground is something like “this improves security a tiny bit by removing this one vector, but others remain”.
True, but the situation here is “should we break legitimate users without actually fixing the problem”.
Even if we were fixing every last instance of the “problem”, we’d still have to weigh that up against breaking legitimate users, but the weighing is even easier to calculate when (a) we’re not fixing all cases, and (b) the solution for the remaining cases is identical to the one we’re “fixing”[1].
To be precise, scanning packages for malicious code before you install them. ↩︎
I just couldn’t come up with anything obvious to pass to the entry point functions! Yes, easily possible in the future.
That’s definitely the intent, but I can see it’s not clear from the spec. Thanks for calling that out.