Removing executable bits and shebangs from incidental modules

In gh-118673, I’ve observed that the tarfile module has a shebang and as a result was marked as executable, a pattern that’s dangerous (tempting execution on an incompatible Python) and non-portable.

Many other modules (such as those touched in gh-64135) and probably others, may be due to reconsideration based on modern factors (credit to @storchaka):

  • runpy invocation (aka python -m) now exists.

  • It’s typical to have multiple Python installations on a system.

  • Modules (like pydoc) can no longer be turned into a command by a simple symlink.

  • Stdlib code is more lenient about breaking changes across (minor) versions.

For similar reasons @gpshead had previously suggested that such modules should not be implying they are executable.

I’d like to proceed with removing these shebangs and executable bits for modules that don’t have a clear use-case. I’ll do this for Python 3.14 with enough time to roll back for use-cases that are identified during the pre-releases. If there’s general consensus, I’ll prepare the pull request.

An alternative approach could be to apply the change piecemeal to modules as needed (such as tarfile, which is currently affecting the backport).

4 Likes

On Fedora 40 with python 3.12 I see 14 .py files in /usr/lib64/python3.12 with execute permissions. I would think that Fedora would like this change to happen.

$ ll *.py | grep '^...x'
-rwxr-xr-x. 1 root  20602 2024-04-09 09:09:14 base64.py*
-rwxr-xr-x. 1 root  34418 2024-04-09 09:09:14 cgi.py*
-rwxr-xr-x. 1 root   6555 2024-04-09 09:09:14 cProfile.py*
-rwxr-xr-x. 1 root  69459 2024-04-09 09:09:14 pdb.py*
-rwxr-xr-x. 1 root  43331 2024-04-09 09:09:14 platform.py*
-rwxr-xr-x. 1 root  23092 2024-04-09 09:09:14 profile.py*
-rwxr-xr-x. 1 root 112795 2024-04-09 09:09:14 pydoc.py*
-rwxr-xr-x. 1 root   7183 2024-04-09 09:09:14 quopri.py*
-rwxr-xr-x. 1 root  43531 2024-04-09 09:09:14 smtplib.py*
-rwxr-xr-x. 1 root  11530 2024-04-09 09:09:14 tabnanny.py*
-rwxr-xr-x. 1 root 106883 2024-04-09 09:09:14 tarfile.py*
-rwxr-xr-x. 1 root  13463 2024-04-09 09:09:14 timeit.py*
-rwxr-xr-x. 1 root  29182 2024-04-09 09:09:14 trace.py*
-rwxr-xr-x. 1 root  23627 2024-04-09 09:09:14 webbrowser.py*

I vote “just do it”. There is plenty of time during the entire 3.14 alpha and beta process to find out if anyone testing 3.14 system integration actually depends on these stdlib .py files being executable. I expect that to be exceedingly rare.

3 Likes

How about a deprecation period before removal? Most people and organizations don’t test pre-release versions.

If you found that someone depended on the executable .py then you could fix that in a patch update.

Also any distro that is packaging python could patch it quickly before you reelase a patch with such a change.

That’s not how we remove features in Python. The normal policy is to first issue due notice (usually under the form of deprecation warnings), and after one or two feature releases, to remove the feature.

There’s no urgency here so I don’t see why we wouldn’t follow this policy.

Propose code to detect use of this situation and emit a friendly deprecation message to stderr when any of the executable Lib/*.py files invoked via an executable bit and #! line but NOT when invoked via python -m and nobody is going to object doing this as a normal pre-announced deprecation either.

I don’t believe these were ever an intentional features. Are any documented? It is impossible to claim that someone isn’t depending on it, but I intuitively expect the Hyrum’s Law uses of this particular “feature” to be extremely low as it is rather painful to even find the stdlib .py files (they’re in a different directory for every minor version and installation and differ per distribution) So I personally lean towards taking the risk on pushing this change rather than drawing it out for a couple years.

It is perfectly fine for us to do this over years as a normal deprecation. Just more complicated than our typical ones because this is an unusual scenario.

1 Like

I was thinking this was not so much a feature as an over sight.

Since @storchaka originally introduced some of these executable bits, perhaps he can tell us more.

1 Like

I only fixed inconsistencies, because it does not make sense to have the executable bit set without shebang or shebang without the executable bit. According to @doko42 this was done in the Debian/Ubuntu packaging anyway.

I removed the executable bit and shebang in cases where this obviously did not make sense (like tests files).

1 Like

This should do the trick:


if __name__ == "__main__" and __spec__ is None:
    import warnings
    warnings.warn("nice message goes here", DeprecationWarning)

We should consider the value proposition. It’s conceivable that adding a deprecation warning could be just as disruptive as removing the executable bit (that is, changing the stdio signature for an executable is probably a breaking change in many cases), so adding a deprecation period draws out the maintenance effort to achieve the desired endpoint and may still not provide any soft landing.

Personally, I was hoping to limit the scope of this change to just the tarfile module and not adopting a project. If we’re talking about someone committing to overseeing a multi-year deprecation project, I’ll probably opt to merely patch the symptom that led to the discovery.

This approach will only emit the warning in test suites (or if deprecation warnings are otherwise enabled). I think we can safely assume that users are unlikely to have deprecation warnings enabled in whatever environment that might be executing these scripts. In that case, perhaps a UserWarning could work, assuming we’re willing to alter the stdio.

3 Likes
$ python3.12 ./mod.py
/......././mod.py:3: DeprecationWarning: nice message goes here
  warnings.warn("nice message goes here", DeprecationWarning)

This is with a plain install of Python 3.12.3 and without python related environment variables set.

Which warning to pick is left as an exercise for the reader though, my point was that it is easily possible to detect this case.

BTW. I have no strong opinion on actually emitting a warning.

1 Like