Do we still need `Tools/{io,cc,string}bench`?

Core Development
1

There are Tools that look quite unused:

  1. cpython/Tools/iobench at main · python/cpython · GitHub
  2. cpython/Tools/stringbench at main · python/cpython · GitHub original issue: Issue 13165: Integrate stringbench in the Tools directory - Python tracker
  3. cpython/ccbench.py at main · python/cpython · GitHub

They still claim to support python2 and haven’t been touched for over a decade (except some minor global refactorings like https://github.com/python/cpython/commit/8f943ca25732d548cf9f0b0393ba8d582fb93e29)

So, the question is: do we still need them? Aren’t these cases covered by GitHub - python/pyperformance: Python Performance Benchmark Suite ? Are they required for faster-cpython project?

I see several ways forward:

  1. Do nothing, they are good as-is
  2. Modernize them to be fully useful inside Tools/: add new feaftures to the test (like new str methods), improve their docs, maybe mention them in devguide
  3. Move them to pyperformance suite if needed and delete them from Tools/
  4. Just delete them - they are outdated (I doubt that)

CC @pitrou @vstinner @storchaka who worked on them initially.

3 Likes
2

Existing discussion in cpython: Move Benchmarks From Tools/ to the pyperformance Repo · Issue #97680 · python/cpython · GitHub

3

I’m in favor of just deleting them. They’re in version control, if another repo (pyperformance is a good example) wants to adopt any of them for any particular purpose they’ll always exist there.

2 Likes
4

Please see PR #101853 to remove them.

5

Some context:

  • iobench was developed (by me) when we rewrote the Python 3 IO stack in C
  • ccbench was developed (by me) when I rewrote the GIL implementation

None of them are of course necessary to keep, though the maintenance cost must be close to zero.

6

I would have commented sooner had I known that this was about to happen. It would have been helpful if someone had commented in my PR to ccbench.py that this was being considered so that I would have been notified and might have had a chance to comment before it happened.

I believe it was clear from the extensive discussion about the shortcomings of the current GIL in faster-cpython that the ccbench script is still useful - because it clearly identifies an ongoing performance issue when multithreading with one or more CPU-heavy threads.

That said, I do understand that performance scripts do not really have a place in the cpython source code repo, but unfortunately I do not have the time to rework the updated ccbench script for the pyperformance suite, so I guess it is going to just disappear.