Have found a small bug in _tokenize.TokenizerIter with comments getting indented & dedented incorrectly. This may seek slightly irrelevant but has an effect on the ‘compiler’ pre-processor I am working on using tokenize/untokenize.
import tokenize as t
from io import BytesIO
source_code = """
if True:
# comment one <- gets incorrectly dedented, indent doesn't happen until after this line
x = 1
# comment two -> gets incorrectly indented, dedent doesn't happen until after this line
x = 2
# comment three
"""
print(f"{source_code=}")
modi_toks = []
toks = t.tokenize(BytesIO(source_code.encode()).readline)
for tok in toks:
print(f"{tok=}")
modi_toks.append((tok.type, tok.string))
modi_source = t.untokenize(modi_toks)
print(modi_source.decode())
There are a lot of (usually insignificant) differences in the source code. Compare your initial and final, and you’ll see quite a few things (I notice a space in “if True :”, no space in “x =1”, and that’s just what first catches the eye). The resultant code will have the same effect as the original, but it may look different.
(BTW, if anyone else tries to run this, add from io import BytesIO at the top.)
Is the unindenting of comments going to cause you problems? If so, I would recommend using something other than tokenize; you may want a simple line-by-line processor instead.
@Rosuav sorry about missing out the BytesIO import, original post now fixed.
I started out trying to use ast but hit problems of going line-by-line with an if as it was a single line and needed a following indented line to avoid an error. Then there was the problem of multi-line statements which was a separate can-of-worms to contend with.
I was hoping that perhaps someone might be able to see the bug and fix this #comments issue in CPython’s tokenizer.
When I said “line by line”, I really meant line by line. Not through the AST. Lines of source code.
It’s not a bug though. The relevant tokens are being emitted. Those tokens are sufficient to compile the code, but comments don’t care about indentation.
Why do you need the token stream? What kinds of tokens are you looking at? There’s a reason that line-by-line preprocessors (knowing nothing about the syntax) are by far the most common.
It won’t make a difference to the compiled code. Note the guarantee in the docs you’re quoting:
“”"
The result is guaranteed to tokenize back to match the input so that the conversion is lossless and round-trips are assured. The guarantee applies only to the token type and token string as the spacing between tokens (column positions) may change.
“”"
The guarantee does not apply to spacing between tokens, but it does apply to things that will be code-relevant.
The stdlib tokenize module does not properly roundtrip. This wrapper around the stdlib provides two additional tokens ESCAPED_NL and UNIMPORTANT_WS, and a Token data type. Use src_to_tokens and tokens_to_src to roundtrip.
This library is useful if you’re writing a refactoring tool based on the python tokenization.
@mfile_bay That’s a shame I was just reading that.
@Rosuavbut comments don’t care about indentation.. Yep but just because the docs seem to suggest this ‘incorrect’ behaviour is allowed wouldn’t it be better if the indentation of comments was actually corrected in the first place? It is only the order the indent / dedent tokens are getting written after a #comments line which cause this, surely this can’t be too difficult to fix can it.
I am using these comments as part of the compiler directives so in this case the faulty indentation may actually be important.
@inventshah mentions using tokenize-rt. Another option is to use token-utils which also guarantees perfect round-trip. Adapting your code to use token-utils:
from token_utils import tokenize, untokenize
source_code = """
if True:
# comment one <- gets incorrectly dedented, indent doesn't happen until after this line
x = 1
# comment two -> gets incorrectly indented, dedent doesn't happen until after this line
x = 2
# comment three
"""
modi_toks = []
toks = tokenize(source_code)
for tok in toks:
modi_toks.append(tok)
modi_source = untokenize(modi_toks)
assert modi_source == source_code
In addition, since you want to modify the source code, tokens from token-utils are mutable.
A line with a comment by itself does not actually need to be correctly indented, so the tokenizer is right to not emit INDENT/DEDENT tokens there.
For example:
if True:
# should the INDENT still come before the comment here?
pass
if True:
# where should the INDENT be emitted in this case?
pass
It’s fine for the comment to go anywhere on the line, so these examples are both correct.
I deleted my post because the post is based on a wrong understanding except for the working method and I think it would be distraction. I can send it to you privately if you want
Just tried that but fails with AttributeError: 'tuple' object has no attribute 'src'
import tokenize_rt as t
from io import BytesIO
source_code = """
if True:
# comment one <- gets incorrectly dedented, indent doesn't happen until after this line
x = 1
# comment two -> gets incorrectly indented, dedent doesn't happen until after this line
x = 2
# comment three
"""
print(f"{source_code=}")
modi_toks = []
# toks = t.tokenize(BytesIO(source_code.encode()).readline)
toks = t.src_to_tokens(source_code)
for tok in toks:
modi_toks.append((tok.name, tok.src))
modi_source = t.tokens_to_src(modi_toks)
print(modi_source
@aroberge tested that but when creating the new tokens using (tok.type, tok.string) it fails with
File “D:\Dev\Python\ImportHook.venv\Lib\site-packages\token_utils\tokenizing.py”, line 270, in _untokenize
assert isinstance(tokens[0], (Token, str))
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
from token_utils import tokenize, untokenize
source_code = """
if True:
# comment one <- gets incorrectly dedented, indent doesn't happen until after this line
x = 1
# comment two -> gets incorrectly indented, dedent doesn't happen until after this line
x = 2
# comment three
"""
modi_toks = []
toks = tokenize(source_code)
for tok in toks:
# modi_toks.append(tok)
modi_toks.append((tok.type, tok.string))
modi_source = untokenize(modi_toks)
Based on the usage guide tokens_to_src takes an iterable of Token objects not of tuples. Did you try modi_toks.append(tok) instead? You can modify the token objects too. tokenize-rt is specifically not a direct API compatible drop-in for tokenize.
@jrivers When it is just a plain comment it obviously doesn’t matter, although it would still be nice to see it correct. However as I am using comments as part of the pre-process compiler directive their correct indentation is important.
Tokens from token-utils are NOT two-tuples and cannot be replaced by two-tuples.
The documentation, and in particular the “recipes”, should have all that you need to find out how to modify a token stream. In particular, if you simply want to append a new token, you can append a simple string.
@aroberge I can see how that would allow me to change the token string but would it also allow me to both add tokens and remove tokens without leaving orphaned newlines?
Edit:
Replacement works but I can’t see how to delete tokens at the moment.
Yes, you can add and remove tokens without leaving orphaned newlines. In addition to that which is found in token-utils’ documentation, you can find many additional examples of modifying source code using token-utils in my ideas repository
Instead of using tokenize(), I’d suggest using get_logical_lines(), so that you can focus on modifying/removing one new logical line at a time.
@aroberge Just had a go at removing some tokens but, unless I am missing something obvious, they are getting added back in again:
from token_utils import tokenize, untokenize
source_code = """
if True:
# comment one <- gets incorrectly dedented, indent doesn't happen until after this line
x = 1
# comment two -> gets incorrectly indented, dedent doesn't happen until after this line
x = 2
# comment three
"""
modi_toks = []
toks = tokenize(source_code)
for tok in toks:
if tok.string[:9] == "# comment":
print(f" **************************************** {tok.string=}")
else:
modi_toks.append(tok)
for tok in modi_toks:
print(tok)
print(modi_toks)
modi_source = untokenize(modi_toks)
print(modi_source)
As shown in some examples in the documentation, to remove a token, simply set its string to the empty string: tok.string = ''; alternatively, as mentioned in untokenizing … If you don’t want to set a token’s string to the empty string to effectively remove it, and absolutely want to remove a token, then use stringify() instead of untokenize() to obtain a string version; note however that all token characters will be replaced by spaces.
In the link above, you will find the explanation as to why attempting to remove a token and use untokenize() does not produce the expected result.