Remove outdated Tools/scripts/ scripts

Hi,

The Python Security Response Team received a report of a shell code injection vulnerability in the Tools/scripts/get-remote-certificate.py script. I decided to fix it: issue #97612.

Honestly, I don’t think that it’s still relevant to maintain such script in Python, and I would suggest to remove it in Python 3.12. For example, this one is a thin wrapper to the openssl s_client -connect HOST:PORT -showcert command. IMO using directly the openssl command is simple enough.

There are 74 Python scripts in Tools/scripts/. How many are outdated? The dutree.py comes with a documentation (an email) written in… 1992. Many scripts are short examples of what can be done in a few lines of Python to enhance some existing Unix commands. The which.py script can now be replaced with the shutil.which() function which has a better implementation.

I’m not sure how to proceed. Should we go through each script and decide if it should be removed or not? Or remove all of them and then wait for complains? :smiley:

Some scripts are used to build Python, like check_extension_modules.py and deepfreeze.py. Maybe they should live in a separated directory? Is win_add2path.py used by the Windows installer?

  • macOS installs these scripts
  • The Windows installer don’t install them (correct me if I’m wrong)
  • On Linux, ./configure && make && make install installs them, but many Linux distributions package these examples in a separated package which is not installed by default (ex: python3-examples on Debian)

If some scripts are useful, maybe the code should be moved into the stdlib? For example, should diff.py becomes python3 -m difflib CLI? Currently, python3 -m difflib runs a self-test, IMO such test should belong to Lib/test/test_difflib.py instead.

Or should we move these scripts to a separated project/Git repository?

Well, what do you think?

Victor

7 Likes

All the things!

3 Likes

+1

LOL diff.py, let’s just get rid of that while also moving -m difflib test code into a unit test. No need for -m difflib to do anything. We’d have to support it’s use as a diff tool forever I’d we did that. Every command line system has a diff tool already.

Just as a kind of data point, a “mature” (20+ yrs) project (written in Python) I work on has been moving old snippets of this nature off into a separate repository of unsupported tools and tricks, and out of the main repo where it might end up getting distributed with the project and considered something that needs supporting.

2 Likes

Okay, consider yourself corrected. (Though only for the executable and Nuget packages, not the Store package.)

But I’d be more than happy to remove them. There’s so little benefit in including them, and non-zero risk, as it turns out.

4 Likes

I think they should all be removed. Too much risk, for things that are never looked at.

If something is used to build Python, that’s another story. But I think it still doesn’t need to be installed on end user systems.

3 Likes

I would have no objection to also removing them from our macOS installers. I added them years ago as a convenience to users but deliberately did not make them very visible. I don’t recall ever seeing any feedback from users; I doubt anyone would miss them there. And if anyone does, I suppose they could be bundled up as a PyPI-installable package.

1 Like

JFDI! I removed Tools/pynche and nobody complained :wink:

5 Likes

What else should go from the Tools directory (when installed for users)?

I think we should remove everything from those directories that isn’t needed to build. I don’t know how to tell what that is. For sure the Tools/demo directory can go – there’s even a demo of remote execution there! :slight_smile:

There are also some benchmarks (ccbench, iobench, stringbench) that probably ought to be moved to pyperformance so they can be used and maintained.

Etc., etc.

3 Likes

I think we need to be a bit more careful here, since some scripts are
useful to (re)generate source files, update or fix them.

None of these need to be user installed. At least on Unix platforms,
“make install” does not install them anywhere, AFAIK.

2 Likes

Just my data point: besides the tools that are required to build, regenerate, etc., the only thing I use here is summarize_stats.py.

It converts the counters dumped by --with-pystats builds into a nice markdown format. I use it at least several times monthly.

I filed https://github.com/python/cpython/issues/97649 to remove the files from being installed on Windows. Not planning to touch anything in the repository Tools directory.

I suggest any of us who actually use these tools (aside from the ones invoked by build scripts which are thus clearly referred to by our build systems) edit them to add a top level comment with our name and date describing what we use it for and why. Consider it a freshness indicator. We don’t need to be shipping/installing most of these, but that at least helps indicate which subset are owned and used.

At worst we can write up the list of files on a whiteboard at the core dev sprints and people can erase the files and directories they want to keep. That might the fastest way to get consensus on the list of things to drop.

2 Likes

File a GitHub issue with the current list as check box bullet items and let all of us committers modify the checkboxes for a week. That way it is global.

6 Likes

I created Remove outdated Tools/scripts/ example scripts · Issue #97669 · python/cpython · GitHub issue to remove these scripts.

Here is a first PR to remove 42 scripts, IMO the least controversial scripts: gh-97669: Remove outdated example scripts by vstinner · Pull Request #97674 · python/cpython · GitHub

I prefer to write separated PRs for the remaining scripts that I plan to remove.

It’s more complicated than what I expected: diff.py and ndiff.py are copied in the difflib documentation. I will propose a separated PR for these scripts.

“win_add2path.py” is not used by the installer. I don’t know whether any project uses this script. PR 1645 improves it in several ways, but this PR has been open for over 5 years without getting merged. The one thing this script does that the installer doesn’t do is add the user scripts directory to PATH. That’s for --user installations, which pip enables automatically if it isn’t permitted to install to the system site packages.

2 Likes