Keep in mind that as currently specified, the presence of a <name>.start file explicitly disables warnings about import lines in the matched <name>.pth file during the deprecation period.
I’d say that convenient time would be Python 3.15, because once you’ve added <name>.start files, import lines in the <name>.pth file are ignored without warning, until after the deprecation period expires.
If I were writing a package scanner like you allude to, I would feel more comfortable if all I had was to look for was a certain type of files, rather than if I had to parse their contents and look for a Python import line (what is the grammar for that exactly? do I risk missing some variations? what about whitespace and other shenanigans? can unicode be used to confuse my parser?).
Also, with the PEP I could easily find out which 3rd party packages are involved in the startup sequence, which allows for a whitelist-based approach.
It’s literally .startswith(("import ", "import\t")): https://github.com/python/cpython/blob/main/Lib/site.py#L210 Add a ';' in line to check for the possibility that it’s doing something suspicious.
Now also add to your scanner looking for files that don’t belong to the package being scanned, and you’ve got a much trickier problem with exactly the same outcome. If you can solve the latter part, the first part should be easy for your scanner.
By that argument, you’ll never be able to make any claims about start up security because I don’t see the functionality of pre-start arbitrary execution ever completely going away. This PEP gives us a way to recognize when it’s there, along with a future way to lock down pre-start code execution.
Whatever other vectors exist for hijacking your interpreter aren’t in scope for this PEP. Python is Python so I think there will always be ways[1].
IMHO, this PEP is careful in its very modest security claims. It can’t help people misreading it, but if you think the PEP needs clarity while still addressing the benefits it provides, please do let me know how I can clarify the language.
the ancient past failure of bastians and restricted execution provide a useful lesson ↩︎
Ah, OK. I think I maybe misread that part. Thanks for clarifying.
However, that does require duplicating the information, which is something I’d prefer to avoid. So it looks like there’s a choice - duplicate data or accept warnings that users can’t do anything about.
I’d still prefer a solution that let people delay adoption without users getting warnings. Even if it’s only something like a 3-release deprecation with warnings off by default for the first release, giving projects a full release to add support.
Is that specified somewhere, and set in stone? Or do I have to rely on some implementation details of the stdlib and a Discourse comment, and trust that it’s sufficient and that nobody will add “improvements” to this parsing logic?
It depends on how you define “the problem”. We’re also not breaking legitimate users; completely removing all possible pre-start code would do that, but that’s not what the PEP is proposing.
The PEP does give users a framework for improvement, and if we add the policy mechanism in the future, then we could break all users by disabling pre-start code execution. The advantage then would be that you, the end user whose environment is getting locked down, can make the policy decisions based on your own use cases, without Python imposing something that might or might not work for you.
I agree! I also believe that it will be much easier to scan for vulnerabilities in entry point callables than it ever will be in arbitrary exec()'d code. If you’re scanning third party packages for problems today, then you’re already set up to scan those entry points.
You really have to scan the entire line and figure out what is getting exec()'d. Otherwise, maybe your safe and maybe you’re not.
But the point is that with entry point execution, if you’re already scanning third party packages for insecure code, then you already have the framework in place to scan entry point functions. But scanning import lines in .pth files is a very different type of scan.
Nothing is set in stone, but it’s documented at https://docs.python.org/3/library/site.html (emphasis mine):
A path configuration file is a file whose name has the form
name.pthand exists in one of the four directories mentioned above; its contents are additional items (one per line) to be added tosys.path. Non-existing items are never added tosys.path, and no check is made that the item refers to a directory rather than a file. No item is added tosys.pathmore than once. Blank lines and lines beginning with#are skipped. Lines starting withimport(followed by space or tab) are executed.
We can easily change what “executed” means here to either warn/ignore anything after a semicolon. We could switch to parsing the imported names and using importlib rather than exec without anyone noticing (providing we still exec the rest of the line).
We can also do that without changing anything else. We don’t need a new file format for that, we can set up a policy that relies on the filename of the .pth file to choose whether imports are made and/or code is executed (or even to ignore the added search paths). We could add that whenever we like, and use a deprecation period to change the default behaviour.
The new file format is entirely just work and churn for our users without enabling anything that we can’t already do.
It’s really not - they get passed directly to exec, which means they’re just Python code, which means the same scan as the rest of the package.
I’m not against hiding those warnings behind the -v flag during the deprecation period. We’re already hiding parsing errors behind that flag so that should be pretty straightforward.
Ok, so anyone writing a package scanner is potentially vulnerable to unforeseen changes in the underlying parsing logic. That sounds like a great idea.
Apply that same reasoning to every syntax change we ever make, or API change that’s ever made. It’s fairly ridiculous, sorry.[1] The people who try and scan for this stuff are constantly making changes to adapt to everything - that’s their job. All we have to do is keep it shaped like executable Python code (even if it stops being directly exec’d) and they’ll be just fine.
But realistically, most scanners will do “this package has a .pth file that does more than set search paths - send it for manual analysis”, and PyPI could do the same to flag a new upload. They don’t need to automatically go deep enough to determine if it’s safe or not, they really just need to detect enough of a red flag to get a human to look at it. (As I’ve mentioned before, I am one of those humans, and often get sent packages with “is this malicious?” attached to it.)
In case it’s not clear, I took the “sounds like a great idea” as sarcasm. Apologies if it was genuine, but it’s still not a great idea. ↩︎
“We won’t make things easier and safer because it’s their job to live dangerously” is a strange line of argument (did you ask them?). It’s also an argument against many kinds of progress.
And, of course, if those “people” fail adapting to unforeseen changes in Python’s stdlib, they will not be the ones suffering the consequences. Their users will.
By “realistically”, you mean that manual analysis is the only thing that’s realistically possible with .pth files?
So you agree that this deserves being replaced with a better, more auditable mechanism, right?
Why would I ask them a strawman argument that you just invented? I didn’t suggest that - I suggested that deprecating code in .pth files and moving it elsewhere has zero impact on safety and negative impact on “easier”.
But yes, I do work with these people and we talk about what they need and what helps them. I’m confident that “arbitrary code in a .py file referenced by a TOML file” vs. “arbitrary code in a .pth file” is completely irrelevant to them.
Manual analysis is the only thing that’s realistically possible with a Python package, yes. It’s actually the only thing that’s been realistically possible with any potentially malicious code up until the last few months.
And your entire post is attempting to put words in my mouth. Please stop (or at least get them right).
Thus quote a movie:
No one said this job was supposed to be easy.
Nobody said it’s supposed to be that hard, neither!
Right, but at 3.18, anybody who wants to continue to support 3.14 until EOL now has unavoidable warnings.
We could leave import line warnings behind -v either forever or for another two years (5 years in total). Either could be done while still keeping the 3-year deprecation for processing import lines.
One more update before I ask the SC for pronouncement. Mostly minor changes.
Delaying any errors until the last version that doesn’t support start files is no longer supported certainly makes sense.
The PEP could also be more explicit that the intended migration mechanism is:
x.y:z)import x.y; x.y.z())So there is some duplication, but not a lot.
It could also suggest (non-normatively) that build tools may want to emit an error or warning if a start file exists with a parallel pth file, and the latter defines code lines that do something other than invoking the entry point.