Tachyon: 97% error rate

ed9001 · June 3, 2026, 9:31pm

Hello,

I am testing the new sampling profiler Tachyon. For my workload, I am consistently seeing error-rates exceeeding 97%, producing poor quality results. Is this due to me using Tachyon incorrectly, or is something about my particular workload throwing it off?

I am using python 3.15.0 beta 2, built with pyenv:

PYTHON_CONFIGURE_OPTS=‘–enable-optimizations --with-lto’ PYTHON_CFLAGS=‘-march=native -mtune=native -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer’ ~/.pyenv/bin/pyenv install -v 3.15.0b2

The target is a pure-python webserver:

git clone https://github.com/9001/copyparty/; cd copyparty; ~/.pyenv/versions/3.15.0b2/bin/python -m profiling.sampling run --heatmap -o a.html -d 10 -r 1khz --blocking -a -m copyparty -q --no-htp --hash-mt=0 --mtag-mt=1 --th-mt=1 -v srv::ra

Load is generated by sending http-requests from another terminal:

while true; do curl -s 127.1:3923/?ls{,,,,,,,,,,,,,,,,,,,,,,,,,,,,}{,,,,,,,,,,,,,,,,,,,,,,,,,,,,} >/dev/null; done

I have tried with/without `–blocking`, and various sampling-rates (1k through 20k), all with no difference. The arguments given to the profiling target (`–no-htp --hash-mt=0 --mtag-mt=1 --th-mt=1`) are to reduce the number of threads, but there is still 8 threads at idle, listed at http://127.0.0.1:3923/?stack .

With --realtime-stats I get:

Stats: 1,000.2Hz (999.8µs) Min: 930.8Hz Max: 1,082.5Hz N=9003 Cache: 98.7% (2135+82/30)
Captured 10,001 samples in 10.00 seconds
Sample rate: 1,000.09 samples/sec
Error rate: 97.52
Unwinder Statistics:
  Frame Cache:
    Total samples:    10,001
    Full hits:        2,309 (94.8%)
    Partial hits:     95 (3.9%)
    Misses:           32 (1.3%)
  Frame Reads:
    From cache:       9,924 (74.6%)
    From memory:      3,375 (25.4%)
  Code Object Cache:
    Hits:             16,236 (59.9%)
    Misses:           10,851 (40.1%)
  Batched Reads:
    Attempts:         12,187
    Successes:        12,167 (99.8%)
    Misses:           20 (0.2%)
    Segments read:    34,171/34,191 (99.9%)
  Memory:
    Read operations:  51,643 (13.6 MB)

if I attach to a running process rather than run it directly under tachyon, the error rate drops to 80%, however the output is still confusing; I know that almost all of the cpu-time is spent in httpcli.run, yet that does not get a single hit.

That aside, I really like everything else about Tachyon so far! The UI looks awesome, and I’m certain it’ll be a fantastic way to see where the cycles are going, as soon as I can get it to sing

maurycy · June 6, 2026, 3:23pm

Alexander,

What a nice issue! No, it’s completely not your fault. I’ve created a Github issue detailing what happened:

github.com/python/cpython

`_remote_debugging`: raising `Invalid bytes length` on codeobjects with linetable > 4096

opened 03:22PM - 06 Jun 26 UTC

maurycy

type-bug

# Bug report ### Bug description: I've reproduced the issue reported in https:…//discuss.python.org/t/tachyon-97-error-rate/107619 The issue is caused by: https://github.com/python/cpython/blob/884ac3e3ec02347301939ff1f124972d4973f015/Modules/_remote_debugging/code_objects.c#L389-L390 The linetable for `HttpCli.tx_browser` and `HttpCli.run` is above 4096: ```bash 2026-06-06T17:13:54.694708000+0200 maurycy@gimel /Users/maurycy/work/cpython (main fded34d*?) % uv run linetable_dist.py ~/Desktop/copyparty objects=3472 median=51 max=19536 >4096=7 19536 AuthSrv._reload /Users/maurycy/Desktop/copyparty/copyparty/authsrv.py:1762 10640 HttpCli.tx_browser /Users/maurycy/Desktop/copyparty/copyparty/httpcli.py:6749 9131 HttpCli.run /Users/maurycy/Desktop/copyparty/copyparty/httpcli.py:335 6245 <module> /Users/maurycy/Desktop/copyparty/copyparty/util.py:1 5087 Up2k._handle_json /Users/maurycy/Desktop/copyparty/copyparty/up2k.py:3035 4146 SvcHub.__init__ /Users/maurycy/Desktop/copyparty/copyparty/svchub.py:136 4133 HttpCli.handle_plain_upload /Users/maurycy/Desktop/copyparty/copyparty/httpcli.py:3648 3753 HttpCli.dump_to_file /Users/maurycy/Desktop/copyparty/copyparty/httpcli.py:2444 ``` The 4096 limit is too small even for our stdlib: ```bash 2026-06-06T17:14:47.179006000+0200 maurycy@gimel /Users/maurycy/work/cpython (main fded34d*?) % uv run linetable_dist.py ~/work/cpython/Lib objects=92035 median=49 max=37416 >4096=37 37416 <module> /Users/maurycy/work/cpython/Lib/html/entities.py:1 20958 <module> /Users/maurycy/work/cpython/Lib/test/test_ast/snippets.py:1 17060 <module> /Users/maurycy/work/cpython/Lib/locale.py:1 11158 <module> /Users/maurycy/work/cpython/Lib/test/test_dis.py:1 10063 SuggestionFormattingTestBase.test_name_error_suggestions_do_not_trigger_for_too_many_locals.<locals>.func /Users/maurycy/work/cpython/Lib/test/test_traceback.py:4899 9953 <module> /Users/maurycy/work/cpython/Lib/stringprep.py:1 9400 <module> /Users/maurycy/work/cpython/Lib/test/re_tests.py:1 5892 <module> /Users/maurycy/work/cpython/Lib/encodings/aliases.py:1 5791 <module> /Users/maurycy/work/cpython/Lib/test/test_socket.py:1 5730 <module> /Users/maurycy/work/cpython/Lib/encodings/mac_arabic.py:1 5730 <module> /Users/maurycy/work/cpython/Lib/encodings/cp866.py:1 5730 <module> /Users/maurycy/work/cpython/Lib/encodings/cp865.py:1 5730 <module> /Users/maurycy/work/cpython/Lib/encodings/cp863.py:1 5730 <module> /Users/maurycy/work/cpython/Lib/encodings/cp862.py:1 5730 <module> /Users/maurycy/work/cpython/Lib/encodings/cp861.py:1 ``` # Reproduction Using real file from stdlib: ```python import os, _remote_debugging src = open("/Users/maurycy/work/cpython/Lib/html/entities.py").read() src += "\n_remote_debugging.RemoteUnwinder(os.getpid()).get_stack_trace()\n" exec(compile(src, "entities.py", "exec")) print("OK") ``` We get: ```bash 2026-06-06T17:17:34.919627000+0200 maurycy@gimel /Users/maurycy/work/cpython (main fded34d*?) % uv run repro.py Traceback (most recent call last): File "/Users/maurycy/src/github.com/maurycy/cpython/repro.py", line 5, in <module> exec(compile(src, "entities.py", "exec")) ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "entities.py", line 2515, in <module> RuntimeError: Invalid bytes length (37445) at 0xc5091c000 ``` # Bonus (thx Claude) <details> <summary><code>linetable_dist.py</code></summary> ```python #!/usr/bin/env python3 import warnings from pathlib import Path from statistics import median from sys import argv from types import CodeType warnings.simplefilter("ignore") SKIP = {"venv", "site-packages", "__pycache__"} def codes(co): yield co for c in co.co_consts: if isinstance(c, CodeType): yield from codes(c) rows = [] for arg in argv[1:]: p = Path(arg).expanduser() for f in [p] if p.is_file() else p.rglob("*.py"): if SKIP & set(f.parts): continue try: co = compile(f.read_text(errors="replace"), str(f), "exec") except Exception: continue rows += [(len(c.co_linetable), str(f), c.co_qualname, c.co_firstlineno) for c in codes(co)] rows.sort(reverse=True) vals = [s for s, *_ in rows] print(f"objects={len(vals)} median={median(vals):.0f} max={max(vals)} " f">4096={sum(v > 4096 for v in vals)}") for s, fn, qual, lineno in rows[:15]: print(f"{s:8d} {qual:24s} {fn}:{lineno}") ``` </details> ### CPython versions tested on: CPython main branch ### Operating systems tested on: macOS

Please drop by there, if you want

Thank you again for creating such a detailed report. It was delightful to reproduce.

Have a great Saturday!

maurycy · July 19, 2026, 9:57pm

Alexander,

I hope that everything is going well!

Pablo has just merged a great fix by Goutam Adwant: gh-151022: Fix remote debugging linetable reads by goutamadwant · Pull Request #151036 · python/cpython · GitHub

Do you want to give it a try?

I tried to reproduce it again, and the error rate on my machine went to around 0,1%.

Thank you once again for a great bug report.

Have a great Monday

ed9001 · August 6, 2026, 9:38pm

Hello again :>

Good news; tried 3.15.0-rc1 just now, and can confirm it’s looking great now – the numbers look realistic, and already spotted a tiny optimization too

Just some quick observations, probably nothing important:

I compared blocking vs. non-blocking; they largely agree, but I noticed some functions had notably more hits in one mode compared to the other. Likely to be expected due to sampling bias, but some examples (number of samples) just in case:

function	block	nonblk
util.py: def _quotep3b	6339	1324
util.py: def absreal	4653	8779
util.py: def humansize	484	1119

As for error rates and missed samples,
With blocking: 0.1% error and 24.4% missed
Nonblocking: 10.7% error and 0.0% missed