# Bug in tokenizer # comments

**URL:** <https://discuss.python.org/t/bug-in-tokenizer-comments/110169>\
**Category:** Python Help\
**Created:** [November 18, 2026, 1:57pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169 "2026-11-18T13:57:59Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![NickC](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nickc/32/37293_2.png) [@NickC](https://discuss.python.org/u/NickC)\
**Post date:** [November 18, 2026, 1:57pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/1 "2026-11-18T13:57:59Z")

</div>

Have found a small bug in \_tokenize.TokenizerIter with comments getting indented & dedented incorrectly. This may seek slightly irrelevant but has an effect on the ‘compiler’ pre-processor I am working on using tokenize/untokenize.

```python
import tokenize as t
from io import BytesIO

source_code = """
if True:
    # comment one <- gets incorrectly dedented, indent doesn't happen until after this line
    x = 1
# comment two -> gets incorrectly indented, dedent doesn't happen until after this line
x = 2
# comment three
"""
print(f"{source_code=}")
modi_toks = []
toks = t.tokenize(BytesIO(source_code.encode()).readline)
for tok in toks:
    print(f"{tok=}")
    modi_toks.append((tok.type, tok.string))
modi_source = t.untokenize(modi_toks)
print(modi_source.decode())

```

---

<div class="post-metadata">

**Author:** ![Rosuav](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rosuav/32/3429_2.png) [@Rosuav](https://discuss.python.org/u/Rosuav)\
**Post date:** [November 18, 2026, 2:05pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/2 "2026-11-18T14:05:35Z")

</div>

There are a lot of (usually insignificant) differences in the source code. Compare your initial and final, and you’ll see quite a few things (I notice a space in “if True :”, no space in “x =1”, and that’s just what first catches the eye). The resultant code will have the same effect as the original, but it may look different.

(BTW, if anyone else tries to run this, add `from io import BytesIO` at the top.)

Is the unindenting of comments going to cause you problems? If so, I would recommend using something other than `tokenize`; you may want a simple line-by-line processor instead.

---

<div class="post-metadata">

**Author:** ![NickC](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nickc/32/37293_2.png) [@NickC](https://discuss.python.org/u/NickC)\
**Post date:** [November 18, 2026, 2:25pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/3 "2026-11-18T14:25:11Z")

</div>

@Rosuav sorry about missing out the BytesIO import, original post now fixed.

I started out trying to use `ast` but hit problems of going line-by-line with an `if` as it was a single line and needed a following indented line to avoid an error. Then there was the problem of multi-line statements which was a separate can-of-worms to contend with.

I was hoping that perhaps someone might be able to see the bug and fix this `#comments` issue in CPython’s tokenizer.

---

<div class="post-metadata">

**Author:** ![Rosuav](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/rosuav/32/3429_2.png) [@Rosuav](https://discuss.python.org/u/Rosuav)\
**Post date:** [November 18, 2026, 3:16pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/5 "2026-11-18T15:16:40Z")

</div>

> [@NickC](#):
>
> I started out trying to use `ast` but hit problems of going line-by-line with an `if` as it was a single line and needed a following indented line to avoid an error. Then there was the problem of multi-line statements which was a separate can-of-worms to contend with.

When I said “line by line”, I really meant line by line. Not through the AST. Lines of source code.

> [@NickC](#):
>
> I was hoping that perhaps someone might be able to see the bug and fix this `#comments` issue in CPython’s tokenizer

It’s not a bug though. The relevant tokens are being emitted. Those tokens are sufficient to compile the code, but comments don’t care about indentation.

Why do you need the token stream? What kinds of tokens are you looking at? There’s a reason that line-by-line preprocessors (knowing nothing about the syntax) are by far the most common.

> [@mfile\_bay](#):
>
> I don’t know if this difference is a bug or a won’t-fix, but what’s for sure is that [the docs](https://docs.python.org/3/library/tokenize.html#tokenize.untokenize) are wrong in saying that omitting fields past the first two don’t make a difference:

It won’t make a difference to the compiled code. Note the guarantee in the docs you’re quoting:

“”"  
The result is guaranteed to tokenize back to match the input so that the conversion is lossless and round-trips are assured. The guarantee applies only to the token type and token string as the spacing between tokens (column positions) may change.  
“”"

The guarantee does not apply to spacing between tokens, but it does apply to things that will be code-relevant.

---

<div class="post-metadata">

**Author:** ![mfile\_bay](https://avatars.discourse-cdn.com/v4/letter/m/d78d45/32.png) [@mfile\_bay](https://discuss.python.org/u/mfile_bay)\
**Post date:** [November 18, 2026, 3:19pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/6 "2026-11-18T15:19:34Z")

</div>

> [@Rosuav](#):
>
> It won’t make a difference to the compiled code. Note the guarantee in the docs you’re quoting:

Sorry, I think I missed that and misinterpreted “Any additional sequence elements are ignored”. I think I’ll delete my post.

* * *

I think I’ll add back the method that untokenizes losslessly: pass back tokens unchanged ([online runner](https://ato.pxeger.com/run?1=jZKxTsQwDIYlxjyFVQYa6RoJJoQ4BjYmFnZUNc41oo2jxBWUE0_Ccgs8FE9DQltxOhguSyLb-T_7l98__cgtud3uY2BTXX6dCNt7CgxMT-jsK0IdgYUJ1IMlmJO3I2O8uxci0hAafGxII6yhKAphDTyEAa8EpHMKDfU9OgZyCNcVbJAjWNdQCNhwN4JGndKoVymaX6AJoztjaGvv0cHg2HZQG8YA3NoInXX4o_2SgOfil8DPBNXNX8KkmwkT6zhCVr_YV28DosgD-mAdl6bY7s2-fiukSI7F9KuzkUtWi3_l7FW5V67Q5auUUgWsdQZKKQyF7HpqOF9xcnChAWxTMGOmCKvU94Io00vx6HGVf6rIqWQj4UBQKo0z9j-RwxIhppWYN2PZkG8))

---

<div class="post-metadata">

**Author:** ![inventshah](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/inventshah/32/29822_2.png) [@inventshah](https://discuss.python.org/u/inventshah)\
**Post date:** [November 18, 2026, 3:29pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/7 "2026-11-18T15:29:48Z")

</div>

You may want [`tokenize-rt`](https://pypi.org/project/tokenize-rt/).

> The stdlib `tokenize` module does not properly roundtrip. This wrapper around the stdlib provides two additional tokens `ESCAPED_NL` and `UNIMPORTANT_WS`, and a `Token` data type. Use `src_to_tokens` and `tokens_to_src` to roundtrip.  
> This library is useful if you’re writing a refactoring tool based on the python tokenization.

---

<div class="post-metadata">

**Author:** ![NickC](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nickc/32/37293_2.png) [@NickC](https://discuss.python.org/u/NickC)\
**Post date:** [November 18, 2026, 3:31pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/8 "2026-11-18T15:31:48Z")

</div>

@mfile_bay That’s a shame I was just reading that.

@Rosuav `but comments don’t care about indentation.`. Yep but just because the docs seem to suggest this ‘incorrect’ behaviour is allowed wouldn’t it be better if the indentation of comments was actually corrected in the first place? It is only the order the indent / dedent tokens are getting written after a #comments line which cause this, surely this can’t be too difficult to fix can it.

I am using these comments as part of the compiler directives so in this case the faulty indentation may actually be important.

---

<div class="post-metadata">

**Author:** ![aroberge](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aroberge/32/6200_2.png) [@aroberge](https://discuss.python.org/u/aroberge)\
**Post date:** [November 18, 2026, 3:46pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/9 "2026-11-18T15:46:34Z")

</div>

@inventshah mentions using tokenize-rt. Another option is to use [token-utils](https://github.com/aroberge/token-utils) which also guarantees perfect round-trip. Adapting your code to use token-utils:

```python
from token_utils import tokenize, untokenize

source_code = """
if True:
    # comment one <- gets incorrectly dedented, indent doesn't happen until after this line
    x = 1
# comment two -> gets incorrectly indented, dedent doesn't happen until after this line
x = 2
# comment three
"""
modi_toks = []
toks = tokenize(source_code)

for tok in toks:
    modi_toks.append(tok)
modi_source = untokenize(modi_toks)

assert modi_source == source_code

```

In addition, since you want to modify the source code, tokens from token-utils are mutable.

---

<div class="post-metadata">

**Author:** ![jrivers](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jrivers/32/12134_2.png) [@jrivers](https://discuss.python.org/u/jrivers)\
**Post date:** [November 18, 2026, 3:47pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/10 "2026-11-18T15:47:56Z")

</div>

A line with a comment by itself does not actually need to be correctly indented, so the tokenizer is right to not emit `INDENT`/`DEDENT` tokens there.  
For example:

```python
if True:
# should the INDENT still come before the comment here?
    pass

if True:
        # where should the INDENT be emitted in this case?
    pass

```

It’s fine for the comment to go anywhere on the line, so these examples are both correct.

---

<div class="post-metadata">

**Author:** ![mfile\_bay](https://avatars.discourse-cdn.com/v4/letter/m/d78d45/32.png) [@mfile\_bay](https://discuss.python.org/u/mfile_bay)\
**Post date:** [November 18, 2026, 3:56pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/11 "2026-11-18T15:56:55Z")

</div>

I deleted my post because the post is based on a wrong understanding except for the working method and I think it would be distraction. I can send it to you privately if you want 😶

---

<div class="post-metadata">

**Author:** ![NickC](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nickc/32/37293_2.png) [@NickC](https://discuss.python.org/u/NickC)\
**Post date:** [November 18, 2026, 4:32pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/12 "2026-11-18T16:32:35Z")

</div>

Just tried that but fails with `AttributeError: 'tuple' object has no attribute 'src'`

```python
import tokenize_rt as t
from io import BytesIO

source_code = """
if True:
    # comment one <- gets incorrectly dedented, indent doesn't happen until after this line
    x = 1
# comment two -> gets incorrectly indented, dedent doesn't happen until after this line
x = 2
# comment three
"""
print(f"{source_code=}")
modi_toks = []
# toks = t.tokenize(BytesIO(source_code.encode()).readline)
toks = t.src_to_tokens(source_code)
for tok in toks:
    modi_toks.append((tok.name, tok.src))
modi_source = t.tokens_to_src(modi_toks)
print(modi_source

```

---

<div class="post-metadata">

**Author:** ![NickC](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nickc/32/37293_2.png) [@NickC](https://discuss.python.org/u/NickC)\
**Post date:** [November 18, 2026, 4:53pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/13 "2026-11-18T16:53:24Z")

</div>

@aroberge tested that but when creating the new tokens using `(tok.type, tok.string)` it fails with

> File “D:\Dev\Python\ImportHook.venv\Lib\site-packages\token\_utils\tokenizing.py”, line 270, in \_untokenize  
> assert isinstance(tokens[0], (Token, str))  
> ~~~~~~~~~~ ^^^^^^^^^^^^^^^^^^^^^^^^^  
> AssertionError

```python
from token_utils import tokenize, untokenize

source_code = """
if True:
    # comment one <- gets incorrectly dedented, indent doesn't happen until after this line
    x = 1
# comment two -> gets incorrectly indented, dedent doesn't happen until after this line
x = 2
# comment three
"""

modi_toks = []
toks = tokenize(source_code)

for tok in toks:
    # modi_toks.append(tok)
    modi_toks.append((tok.type, tok.string))
modi_source = untokenize(modi_toks)

```

---

<div class="post-metadata">

**Author:** ![inventshah](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/inventshah/32/29822_2.png) [@inventshah](https://discuss.python.org/u/inventshah)\
**Post date:** [November 18, 2026, 4:56pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/14 "2026-11-18T16:56:14Z")

</div>

Based on the usage guide `tokens_to_src` takes an iterable of `Token` objects not of tuples. Did you try `modi_toks.append(tok)` instead? You can modify the token objects too. `tokenize-rt` is specifically not a direct API compatible drop-in for `tokenize`.

---

<div class="post-metadata">

**Author:** ![NickC](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nickc/32/37293_2.png) [@NickC](https://discuss.python.org/u/NickC)\
**Post date:** [November 18, 2026, 5:00pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/15 "2026-11-18T17:00:43Z")

</div>

@jrivers When it is just a plain comment it obviously doesn’t matter, although it would still be nice to see it correct. However as I am using comments as part of the pre-process compiler directive their correct indentation is important.

---

<div class="post-metadata">

**Author:** ![aroberge](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aroberge/32/6200_2.png) [@aroberge](https://discuss.python.org/u/aroberge)\
**Post date:** [November 18, 2026, 5:09pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/16 "2026-11-18T17:09:14Z")

</div>

Tokens from token-utils are NOT two-tuples and cannot be replaced by two-tuples.

The [documentation](https://aroberge.github.io/token-utils/docs/html/), and in particular the [“recipes”](https://aroberge.github.io/token-utils/docs/html/tips.html), should have all that you need to find out how to modify a token stream. In particular, if you simply want to append a new token, you can append a simple **string**.

---

<div class="post-metadata">

**Author:** ![NickC](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nickc/32/37293_2.png) [@NickC](https://discuss.python.org/u/NickC)\
**Post date:** [November 18, 2026, 5:24pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/17 "2026-11-18T17:24:15Z")

</div>

@inventshah Yes it works using `modi_toks.append(tok)` but then how do I modify the source?

Also will it allow me to add/remove tokens as well.

---

<div class="post-metadata">

**Author:** ![NickC](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nickc/32/37293_2.png) [@NickC](https://discuss.python.org/u/NickC)\
**Post date:** [November 18, 2026, 5:36pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/18 "2026-11-18T17:36:49Z")

</div>

@aroberge I can see how that would allow me to change the token string but would it also allow me to both add tokens and remove tokens without leaving orphaned newlines?

Edit:  
Replacement works but I can’t see how to delete tokens at the moment.

---

<div class="post-metadata">

**Author:** ![aroberge](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aroberge/32/6200_2.png) [@aroberge](https://discuss.python.org/u/aroberge)\
**Post date:** [November 18, 2026, 5:54pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/19 "2026-11-18T17:54:09Z")

</div>

> [@NickC](#):
>
> I can see how that would allow me to change the token string but would it also allow me to both add tokens and remove tokens without leaving orphaned newlines?

Yes, you can add and remove tokens without leaving orphaned newlines. In addition to that which is found in token-utils’ documentation, you can find many additional examples of modifying source code using token-utils in my [ideas repository](https://github.com/aroberge/ideas)

Instead of using `tokenize()`, I’d suggest using `get_logical_lines()`, so that you can focus on modifying/removing one new [logical line](https://docs.python.org/3/reference/lexical_analysis.html#logical-lines) at a time.

---

<div class="post-metadata">

**Author:** ![NickC](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/nickc/32/37293_2.png) [@NickC](https://discuss.python.org/u/NickC)\
**Post date:** [November 18, 2026, 6:18pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/20 "2026-11-18T18:18:53Z")

</div>

@aroberge Just had a go at removing some tokens but, unless I am missing something obvious, they are getting added back in again:

```python
from token_utils import tokenize, untokenize

source_code = """
if True:
    # comment one <- gets incorrectly dedented, indent doesn't happen until after this line
    x = 1
# comment two -> gets incorrectly indented, dedent doesn't happen until after this line
x = 2
# comment three
"""

modi_toks = []
toks = tokenize(source_code)
for tok in toks:
    if tok.string[:9] == "# comment":
        print(f" **************************************** {tok.string=}")
    else:
        modi_toks.append(tok)
for tok in modi_toks:
    print(tok)
print(modi_toks)
modi_source = untokenize(modi_toks)
print(modi_source)

```

---

<div class="post-metadata">

**Author:** ![aroberge](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aroberge/32/6200_2.png) [@aroberge](https://discuss.python.org/u/aroberge)\
**Post date:** [November 18, 2026, 6:25pm UTC](https://discuss.python.org/t/bug-in-tokenizer-comments/110169/21 "2026-11-18T18:25:36Z")

</div>

> [@NickC](#):
>
> Just had a go at removing some tokens but, unless I am missing something obvious, they are getting added back in again

As shown in some examples in the documentation, to remove a token, simply set its string to the empty string: `tok.string = ''`; alternatively, as mentioned in [untokenizing](https://aroberge.github.io/token-utils/docs/html/untokenizing.html) … _If you don’t want to set a token’s string to the empty string to effectively remove it, and absolutely want to remove a token, then use `stringify()` instead of `untokenize()` to obtain a string version; note however that all token characters will be replaced by spaces._

In the link above, you will find the explanation as to why attempting to remove a token and use `untokenize()` does not produce the expected result.

[Next page](https://discuss.python.org/t/bug-in-tokenizer-comments/110169.md?page=2)
