In gh-118673, I’ve observed that the tarfile module has a shebang and as a result was marked as executable, a pattern that’s dangerous (tempting execution on an incompatible Python) and non-portable.
Many other modules (such as those touched in gh-64135) and probably others, may be due to reconsideration based on modern factors (credit to @storchaka):
runpy invocation (aka python -m) now exists.
It’s typical to have multiple Python installations on a system.
Modules (like pydoc) can no longer be turned into a command by a simple symlink.
Stdlib code is more lenient about breaking changes across (minor) versions.
I’d like to proceed with removing these shebangs and executable bits for modules that don’t have a clear use-case. I’ll do this for Python 3.14 with enough time to roll back for use-cases that are identified during the pre-releases. If there’s general consensus, I’ll prepare the pull request.
An alternative approach could be to apply the change piecemeal to modules as needed (such as tarfile, which is currently affecting the backport).
On Fedora 40 with python 3.12 I see 14 .py files in /usr/lib64/python3.12 with execute permissions. I would think that Fedora would like this change to happen.
I vote “just do it”. There is plenty of time during the entire 3.14 alpha and beta process to find out if anyone testing 3.14 system integration actually depends on these stdlib .py files being executable. I expect that to be exceedingly rare.
That’s not how we remove features in Python. The normal policy is to first issue due notice (usually under the form of deprecation warnings), and after one or two feature releases, to remove the feature.
There’s no urgency here so I don’t see why we wouldn’t follow this policy.
Propose code to detect use of this situation and emit a friendly deprecation message to stderr when any of the executable Lib/*.py files invoked via an executable bit and #! line but NOT when invoked via python -m and nobody is going to object doing this as a normal pre-announced deprecation either.
I don’t believe these were ever an intentional features. Are any documented? It is impossible to claim that someone isn’t depending on it, but I intuitively expect the Hyrum’s Law uses of this particular “feature” to be extremely low as it is rather painful to even find the stdlib .py files (they’re in a different directory for every minor version and installation and differ per distribution) So I personally lean towards taking the risk on pushing this change rather than drawing it out for a couple years.
It is perfectly fine for us to do this over years as a normal deprecation. Just more complicated than our typical ones because this is an unusual scenario.
I only fixed inconsistencies, because it does not make sense to have the executable bit set without shebang or shebang without the executable bit. According to @doko42 this was done in the Debian/Ubuntu packaging anyway.
I removed the executable bit and shebang in cases where this obviously did not make sense (like tests files).
We should consider the value proposition. It’s conceivable that adding a deprecation warning could be just as disruptive as removing the executable bit (that is, changing the stdio signature for an executable is probably a breaking change in many cases), so adding a deprecation period draws out the maintenance effort to achieve the desired endpoint and may still not provide any soft landing.
Personally, I was hoping to limit the scope of this change to just the tarfile module and not adopting a project. If we’re talking about someone committing to overseeing a multi-year deprecation project, I’ll probably opt to merely patch the symptom that led to the discovery.
This approach will only emit the warning in test suites (or if deprecation warnings are otherwise enabled). I think we can safely assume that users are unlikely to have deprecation warnings enabled in whatever environment that might be executing these scripts. In that case, perhaps a UserWarning could work, assuming we’re willing to alter the stdio.
I thought the idea was to only warn about using the executable bit? Running python ./mod.py should be okay, it’s ./mod.py relying on being +x that is being deprecated.
@pitrou , given Serhiy’s response, it does seem as if these behaviors leaked unintentionally into the codebase. Given that these changes were likely unintentional, would you be okay with making the shift without the deprecation period? I’m mainly interested in avoiding a protracted effort if it’s not needed.
Good call. We’d need to inject a bit more state to track whether the invocation was explicit or implicit:
#!/usr/bin/env DEPRECATED_INVOCATION=1 python3
if __name__ == '__main__':
import os
if __spec__ is None and os.environ.get('DEPRECATED_INVOCATION'):
import warnings
warnings.warn("Direct invocation is deprecated", DeprecationWarning)
@ py -m foo
@ py ./foo.py
@ ./foo.py
/Users/jaraco/draft/./foo.py:10: DeprecationWarning: Direct invocation is deprecated
warnings.warn("Direct invocation is deprecated", DeprecationWarning)
Before @ronaldoussoren submitted his example I did not think that deprecation is even possible here. And now we see that this requires adding some fairly non-trivial code to each shebanged file. I think that we can just remove shebangs if this is needed. We should left them in scripts which are supposet to be run by external Python.
The hard part is how to make de-shebangization without sending a false message that shebang in the Python file is now condered bad.
