Since I’m one of the people who weighed in from the outside on this, let me just put in my $0.02.
I believe that some attack vectors are inherently more dangerous than others. Certainly, vectors that involve changes in the actual code of a library are less dangerous than ones that can be executed via code that doesn’t touch the core code. Likewise, vectors that execute just on the package being installed are more dangerous than those that require a package being actually imported and run.
All that said, I understand phasing out .pth files outright can be hard. However, I think a good approach is the one that has been adopted by the alternate Node package manager pnpm - by default, it doesn’t execute any postinstall scripts, but instead warns the user that some packages do require postinstall scripts and you can approve them to run. Then, the user can just do pnpm approve-builds and mark the scripts they want to approve. Of course, this is a package-manager level solution, not a language-level solution, but it does seem to me a valid road to take.
I don’t think the remarks here that claim “well, not everyone will be be affected even if we do it in pip since not everyone uses pip” are valid. When it comes to attack vectors, a secondary goal after removing the vector altogether is limiting its reach. If patching a package manager used by, say, 70% of the community stops 70% of the propagation, that is already a good result. An “all or nothing” approach of “well either we secure it completely or do not secure it at all” is IMO a bad idea when it comes to security risk management.
But it won’t protect 70% of users. It will protect 0% of users because of the previously mentioned plethora of other ways to inject startup code. And that’s ignoring the security fatigue aspect and the fact that not many people really install packages without the intention of using them.
Post-install scripts are a different beast - they run at install time (potentially with elevated privileges), and Python’s package management community has already chosen not to support them when installing from binary packages. Switching off source builds by default is still a work in progress, but one we hope will eventually be achieved.
The vector here is instead startup code, which changes an attack from “importing a particular module runs the attack code” to “starting the interpreter runs the attack code”. That splits the design discussion in half:
a packaging discussion about when it makes sense to ship startup files, especially through a public index like PyPI (answer: not often, potentially never, so index level and installation time filtering may be beneficial, modulo security theatre concerns)
an interpreter level discussion about improved auditing and filtering support for startup code execution (which is now primarily in the PEP 829 thread)
and the fact that not many people really install packages without the intention of using them.
I don’t think this is a good argument. On any bigger installation a large portion of packages are installed transitively, and indeed many might not be imported at all in the code paths that user cares about — especially as the lazy keywords becomes available and adopted.
Malicious packages can shadow the standard library, including modules guaranteed to be imported.
Malicious packages can do certain things to place the malicious code where they will be executed on any import of the library, lazy or not, due to how the import behavior is specified even with lazy imports (specifically, stick the malicious code in a meta path finder there’s slightly more required for this, but it’s still possible.)
lazy isn’t a security boundary of any sort. People need to understand that choosing to install a package is giving that package and all of it’s transitive dependencies access to whatever the interpreter has access to.
Saw that you had 3 ways to do things, and picked one of them to focus on and implement the other 2 based on
Namely the __path__/__init__.py strategy was extended with basically execfile()
This eliminates the runtime from editables.redirector import dependency
Symlinks are optional
This strategy could be adjusted so that pip install -e . wouldn’t require .pth files either
As ncoghlan said, this doesn’t affect binary packages. Most of the time, setup.py executes on the author’s PC, not the user’s PC.
I oppose this from a debuggability and performance perspective in addition to the security perspective others have described. I don’t want anything to run until I import it, so that my double-click breakpoints work.
Additionally, time python -c 'import torch' takes real 0m1.019s. If we give library authors this power, they’ll abuse it to be lazy and make Python startup as slow as Internet Explorer, which each ActiveX being a ML framework. It’s precisely synchronous startup code that is slow, and threaded async startup code wouldn’t have many use cases. I also explained why the other startup hooks can be removed.
It depends on the use-case. For meson-python, because the files are not laid correctly on the build directory, I had to write a custom finder. The only alternative I am seeing there is to install a dummy module and do some import trickery like trying to replace the module during execution, or trying to emulate the real module object in the dummy one. None of which is something I’d feel super comfortable shipping to a wide user base.
Per the point above, the import ordering is not really an issue right now, so I’d lean towards just forcing the __startup__ file to have the package name.
Thanks. My comment was more of a response to the suggestion @steve.dower made that it was possible to do what they currently do in another way “without too much trouble” And I’ll admit, I was a little annoyed that he suggested it was just about laziness
Your PR demonstrates pretty clearly to me that it is “too much trouble” to avoid .pth files.
You do understand that pip install -e doesn’t use .pth files, it simply delegates to the build backend? Which in turn either implements its own logic or calls editables to provide the logic.
IMO you really need to do more research before proposing these ideas. You’re not helping your case at all at the moment.
All in all, given there’s no real plan to address any of the other attack vectors right now, I am okay with basically any of the proposed options, as long as they have feature parity, and don’t significantly impact the startup time.
To circle back to this, the approach that @daniel.z.tg proposed of using “self-replacing” modules instead of an import hook does actually offer similar functionality to import hooks. I’m expecting that the next version of editables will offer that approach as an alternative to the import hook - it avoids the runtime dependency and the execution of code at startup time. I won’t make it the default immediately as there may be subtle incompatibilities that could break user workflows.
Having said that, I’m strongly of the opinion that the simpler approach of just using a .pth file to add one or more directories to sys.path is entirely sufficient for nearly every use case, and far superior in terms of compatibility and robustness.
I do still object to the characterisation of using import hooks as “lazy”. They are a core part of Python’s import mechanism, and it’s not at all clear how tools are expected to install them without running code that executes at interpreter startup. If the core team as a whole (as opposed to just @steve.dower) want to discourage people from using on-startup code, they need to provide better advice on how to use mechanisms like import hooks properly.
I’m not sure how any of this ties back into this thread, except to say that editable installs may not be impacted quite as much as I originally claimed. But I can’t speak for setuptools, or other build backends that implement their own editable install code rather than using editables.
I agree, and I totally understand that the arbitrary code is only going to be for cases where the build backend needs to be re-run.
For editables, at least, you know the name of the module that someone is going to want to import. So install that_module_name.py into site-packages and make it install the hook and then (essentially) from that_actual_module.__init__ import *. I implemented the equivalent in pymsbuild for the DLL packing feature.[1]
An arbitrary extension to importlib that’s trying to hook things that aren’t meant to be hooked, just like custom codecs that may be used for source files, really do need to be injected early. But editables is not this case - you can catch the initial import event without a hook and set it up then.
Though half in C, which gave me the advantage that I could just execute the real bytecode “in” the module to emulate the import *. Code is here but be brave, reading importlib is easier! ↩︎
I don’t want to hijack this thread into “how to redirect module imports”, but with this approach, there’s a whole bunch of tricky details:
You need to ensure the hook isn’t installed multiple times.
You need to “clean up” the module namespace before importing the actual code, or you’ll have implementation symbols leak into the module.
In practice, it’s the same as the “self-replacing” code[1], but with added downsides (import hook priority issues, the runtime support requirement).
I’ll accept “didn’t think of it”, but “laziness” suggests that people know better approaches than on-startup code and can’t be bothered using them, which is not a helpful characterisation of the situation (because it ignores the possibility that we need to communicate better how to do the things that people use on-startup code for).
Which was sufficiently non-obvious that I needed @daniel.z.tg to describe the approach to me before I realised it was possible ↩︎
Installing a file like that seems a lot simpler than installing a file which installs a hook and then uses the hook to replace the module in essentially the same way…
One of the proposed restricted formats in the PEP 829 thread is for executable lines in .pth files to be at most:
import pkg.mod; mod.fn(arg1, arg2, ...)
(where argN are literals, and subpackages are permitted). I think we could define this restricted format in a simple PEP, and installers and package indices can opt in to rejecting .pth files which don’t adhere to this format.
I’d be willing to write this PEP and a PR for pip (and maybe Warehouse). I would add an escape hatch environment variable to skip this on a per-package basis.
This would still allow for manually constructed and runtime-generated .pth files to be unrestricted, while achieving the goals of the original post.
Perhaps even dropping the function call would help with auditing.
I also think that this characterization is a bit misleading…
The approval and subsequent implementation of PEP 660 was a notably bumpy process, largely because editable installs are a genuinely hard problem. A key difficulty is that the term “editable install” itself is overloaded, and users employ it conflicting expectations[1].
To my knowledge, there is no single robust or universally “correct” way to implement editable installs, and all existing approaches involve trade offs. (The many treads related to PEP 660 and PEP 662, the many problems discussed, and the fact that we are still discussing them years after the PEP approval are a testament to that).
For that reason, I would say that the major implementations available of editable installs are anything but lazy. A lot of thinking, tinkering and testing was required in order to achieve a somewhat satisfactory implementation of the current generation of editable installs.
Given the constraints of the current import machinery, .pth files remain a fundamental mechanism for achieving that.
With that said, I would like to report the main uses that setuptools make of .pth files:
One is the distutils compatibility hack, used to provide consistent versions of distutils across different versions of Python (this use will be superseded once Python 3.11 reaches EoS, but arguably this technique is useful when offloading parts of the stdlib into separated PyPI projects, so who knows it may be used by other packages in the future).
Another is the editable installation. As far as I know, setuptools has one of the most complete/advanced implementations of the PEP 660 out there (not free of limitations, because as I previously mentioned, editable installs is a hard problem). This usage itself is split into 3:
A static .pth file that allows adding a link farm directory to sys.path for stricter installations.
A dynamic .pth file when the source tree is complex (i.e. when packages have complex directory mappings, or for flat-layouts to avoid accidentally adding unintended files to sys.path entries).
These cases exist because users expect different and sometimes contradictory properties from editable installs, and a single mechanism does not satisfy all of them. Implementing all of them and providing a way for users choosing was a way to get out of the deadlock in the discussions.
Given this scenario, it would be good that any proposal intended to replace or deprecate current .pth files offer an equivalent expressive power. Or alternatively, we may decide to revise the editable installation standard and “shrink” its scope in a more clear cut way, so that we can streamline the implementation in a way that we no longer need dynamic behaviour.
One interesting thing about the setuptools implementation of PEP 660, is that there is some level of “static metadata” that could be extracted and used to parameterize a common dynamic finder behaviour…
In principle, this suggests a path where a shared finder or importer, either in the standard library or as a blessed PyPI package, implements the runtime behaviour[2], and backends only emit static metadata files such as JSON. However, any such approach still needs to account for the full range of layouts and constraints that current PEP 660 compliant implementations handle.
Very briefly, this usually comes down to a tension between two expectations: (a) that an editable install should behave like a regular installation, exposing the same package contents and no more, so that packaging errors can be caught during development; and (b) that the editable install should reflect changes to the source tree immediately. (P.S:. Someone may be able to provide a better summary with the inherit problems and conflicting expectations of editable installs). ↩︎
And as far as I understand that common shared runtime behaviour is a “side-effect” at initialisation time, unless implemented directly on the import machinery itself… ↩︎
This is a very valid point, one of the main criticisms (if not THE MAIN) with the setuptools implementation of editable installs is that static analysis tools cannot cope with the dynamic effects of it.
