# Merge typed\_ast back into CPython **URL:** \ **Category:** Ideas\ **Created:** [November 4, 2018, 5:03pm UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377 "2018-11-04T17:03:34Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\ **Post date:** [November 4, 2018, 5:03pm UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/1 "2018-11-04T17:03:34Z")
There’s a fork of the ast module (in C) named [typed\_ast](https://github.com/python/typed_ast) used by [mypy](https://github.com/python/mypy/), [pytype](https://github.com/google/pytype) and (IIRC) also by some linters. Its redeeming quality is that it preserves _certain_ comments (currently only `# type:` comments; I could imagine that it might be extended to support `# noqa` comments too). We’ve found that it’s hard work to keep this code up to date with developments in the language’s grammar. (E.g. mypy still [doesn’t support](https://github.com/python/typed_ast/issues/60) all new Python 3.7 syntax.) I propose to merge this code back into the ast module, thereby simplifying maintenance of the typed\_ast module and ensuring that it stays compatible with new syntax added. Thoughts? PS. I’m not sure this belongs in Ideas or Committers – if it’s in Ideas it would be simpler for other users of the ast (e.g. linter owners) to provide feedback or support for this idea. But in Committers the people who will end up maintaining it are reached. ---
**Author:** ![dstufft](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dstufft/32/23_2.png) [@dstufft](https://discuss.python.org/u/dstufft)\ **Post date:** [November 4, 2018, 5:16pm UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/2 "2018-11-04T17:16:32Z")
> [@guido](#): > > PS. I’m not sure this belongs in Ideas or Committers – if it’s in Ideas it would be simpler for other users of the ast (e.g. linter owners) to provide feedback or support for this idea. But in Committers the people who will end up maintaining it are reached. I think the intent is that it belongs in “Ideas”, giving everyone a chance to be a part of the discussion. We don’t have a direct parallel to python-dev in Discourse, and there is some discussion [here](https://discuss.python.org/t/where-should-python-development-discussions-move-to/172/3) about why that is. It seems like the intent is for Ideas to mirror python-ideas, and Users to sort of mirror python-list _and_ python-dev. I don’t really understand that and feel like we need a Development category, but I’m not a Discourse admin, maybe @ambv can weigh in more? With regards to the proposal itself, I don’t really have an opinion, sorry! ---
**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\ **Post date:** [November 4, 2018, 5:36pm UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/3 "2018-11-04T17:36:40Z")
OK, moved it to Ideas. ---
**Author:** ![gpshead](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/gpshead/32/54_2.png) [@gpshead](https://discuss.python.org/u/gpshead)\ **Post date:** [November 5, 2018, 5:17am UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/4 "2018-11-05T05:17:33Z")
I’m generally +1 on the idea of merging in and maintaining this as part of CPython along with the latest grammar. But can we _please_ keep maintaining a current release of this on PyPI? That way it can be used under older Python versions enabling tooling running on them to analyze the latest syntax. Otherwise we’re stuck in the heck that things like pylint find themselves in of only supporting the language version that the linter is running under. ---
**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\ **Post date:** [November 5, 2018, 7:24am UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/5 "2018-11-05T07:24:53Z")
> [@gpshead](#): > > Otherwise we’re stuck in the heck that things like pylint find themselves in of only supporting the language version that the linter is running under. Yeah, we don’t want that. Though right now mypy is generally able to parse Python 3.4 and up with a single typed\_ast version. Hmm, except that `async` and `await` are always keywords. So you’re right. Anyway, we also need the PyPI version for Python 2.7. (But that version doesn’t evolve. 🙂 Still I think it would be an improvement if this was maintained as part of CPython and kept up to date with CPython. ---
**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\ **Post date:** [November 5, 2018, 9:47am UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/6 "2018-11-05T09:47:29Z")
> [@guido](#): > > Its redeeming quality is that it preserves _certain_ comments (currently only `# type:` comments; I could imagine that it might be extended to support `# noqa` comments too). How problematic would it be to keep _all_ comments? If only `type` and `noqa` are special, I expect that people will be tempted to use them for unrelated things (like attribute docstrings or blockly coordinates). ---
**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\ **Post date:** [November 5, 2018, 6:06pm UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/7 "2018-11-05T18:06:47Z")
> [@encukou](#): > > How problematic would it be to keep _all_ comments? RIght now typed\_ast preserves type annotations in extra fields on the AST. It only has slots for these in specific types. Preserving _all_ comments would require a very different architecture (more like the AST nodes used by lib2to3). So this feels out of scope for a simple “move the code to upstream” proposal; I don’t want to complicate the proposal with scope creep. ---
**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\ **Post date:** [November 5, 2018, 10:33pm UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/8 "2018-11-05T22:33:07Z")
Currently comments are entirely ignored. The following code is represented as a single AST node `Constant(value='foobar')`: ```auto ('foo' # type: int 'bar') ``` ---
**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\ **Post date:** [November 6, 2018, 12:42am UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/9 "2018-11-06T00:42:31Z")
I meant currently in typed-ast. It most definitely stores type comments. ---
**Author:** ![njs](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/njs/32/204_2.png) [@njs](https://discuss.python.org/u/njs)\ **Post date:** [November 6, 2018, 5:31am UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/10 "2018-11-06T05:31:11Z")
It’s probably worth spending a few minutes considering how this proposal relates to @ambv’s proposal to provide a fully comment-preserving parser in the stdlib ([bpo-33337](https://bugs.python.org/issue33337)). I guess that like most things they’re less related than the summaries make them sound, but I’m not super familiar with the details myself. ---
**Author:** ![ilevkivskyi](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ilevkivskyi/32/26_2.png) [@ilevkivskyi](https://discuss.python.org/u/ilevkivskyi)\ **Post date:** [November 7, 2018, 2:02pm UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/11 "2018-11-07T14:02:59Z")
This is an interesting idea, but it will require some work to merge it back. Another idea about arbitrary comments: we can preserve a mapping {line\_number: comment\_string}, something like `# type: ignore` comments, that are just a list of line numbers in `typed_ast` currently. Anyway, I am +1 on this. @njs The original goal/motivation of that issue was a bit different, but there is a large overlap, in particular `typed_ast` also tokenizes all versions down to 2.7. So if we are exposing the possibility to parse all versions by same parser, we can also expose the possibility to tokenize all versions. ---
**Author:** ![ambv](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ambv/32/25884_2.png) [@ambv](https://discuss.python.org/u/ambv)\ **Post date:** [November 7, 2018, 7:43pm UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/12 "2018-11-07T19:43:46Z")
As others said, this doesn’t actually fix the problem for mypy as there will always be a need for external tools to be compatible with more than a single version of Python. There are two cases of this: supporting older _and_ newer versions than the currently used one. This is the main (but not the only) reason why Black keeps its own bundled version of lib2to3’s pytree. The “# noqa” idea is intriguing but I wonder if it could be made to work as easily as type comments since noqa is line-based and therefore can meaningfully come literally after any token. And if it could, at this point it probably would make sense to preserve all comments because why not. But this makes the AST bigger. More importantly, I’m not sure what @guido is asking for here. The built-in AST for Python 3.8 could support type comments but it doesn’t need them for anything, there are annotations after all. The only type comment that is not expressible as an annotation is “type: ignore” which, again, Python itself does not need. I might be misunderstanding what is discussed here. Merging type comment support to the built-in AST just to simplify typed-ast’s maintenance seems wrong to me. Python itself doesn’t need this functionality and typed-ast will need to continue existing anyway. > [@njs](#): > > provide a fully comment-preserving parser in the stdlib ([bpo-33337](https://bugs.python.org/issue33337)) This proposal is about a _concrete_ syntax tree, and specifically about taking pytree out of lib2to3, merging the two implementations of tokenizer.py and documenting the result (which it currently is not). ---
**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\ **Post date:** [November 20, 2018, 12:41am UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/13 "2018-11-20T00:41:12Z")
> [@ambv](#): > > I might be misunderstanding what is discussed here. Merging type comment support to the built-in AST just to simplify typed-ast’s maintenance seems wrong to me. Python itself doesn’t need this functionality and typed-ast will need to continue existing anyway. True, but the maintenance burden for typed\_ast is really high (basically it might take a person-month to do the port, since the original maintainer has left). And once we have this, creating a separate typed\_ast would be much simpler (just copy the files). ---
**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\ **Post date:** [November 20, 2018, 10:03am UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/14 "2018-11-20T10:03:02Z")
Sorry, this reads really wrong to me. I must be misunderstanding. A high-maintenance external project losing its maintainer is sad, but that shouldn’t be a reason to merge the project into the stdlib (i.e. ask core developers are now asked to do the maintenance instead). Who’d be the core dev responsible for maintaining it in stdlib, and why can’t this person do it in an external library? Would the stdlib version be open to extending for other use cases? In bpo-33337, a fully comment-preserving \*ST (improved pgen2) was asked to become a third-party library. I fail to see how typed\_ast is different – it even seems typed\_ast could be built on top of such a comment-preserving tree, so adding typed\_ast to stdlib first seems backwards. I am confused. I must be misunderstanding something. I hope this post doesn’t read as an attack. ---
**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\ **Post date:** [November 21, 2018, 12:34am UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/15 "2018-11-21T00:34:03Z")
@encukou Maybe you didn’t realize typed\_ast is a fork of ast, with exactly two things added? It adds fields to certain nodes that hold the type comment, and it adds a bitmap indicating which lines contain `# type: ignore` comments (which look like type comments but really are a different thing – they can occur in places where a type comment would not be legal). Note that both type comments and `# type: ignore` are described in a PEP. This is not your ordinary third-party package. Most of the maintenance to typed\_ast is keeping it up to date with changes to Python’s syntax for each new Python release. Currently this must either be done by starting afresh with a new fork of ast, and then re-applying the (very stable) changes to support type comments, or by taking the entire diff between ast for two adjacent Python versions and manually applying it to typed\_ast. Bot of these processes are difficult due to code reformatting and unrelated other changes to the ast module in the CPython repo. In addition, flake8 currently uses the ast module, and it would benefit from having the type comments: an import that is only used in a type comment should not be counted as unused. There are probably other tools that would benefit from having the type comments available too (though IIRC pylint has its own parser, Astroid). The feature could be extended to e.g. keeping track of `# noqa` comments (which flake8 currently parses separately). Those are more like `# type: ignore` than like type comments, in that they apply to arbitrary lines. They do optionally allow additional text, but that’s simple to solve. ---
**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\ **Post date:** [November 21, 2018, 9:17am UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/16 "2018-11-21T09:17:32Z")
Ah, that cleared it up for me (and hopefully some others). Thank you! > [@guido](#): > > The feature could be extended to e.g. keeping track of `# noqa` comments (which flake8 currently parses separately). Those are more like `# type: ignore` than like type comments, in that they apply to arbitrary lines. They do optionally allow additional text, but that’s simple to solve. In what way do you imagine solving it? Would the additional text be preserved? If yes, what’s holding the parser back from also retaining other comments, like attribute/variable documentation? ---
**Author:** ![ambv](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ambv/32/25884_2.png) [@ambv](https://discuss.python.org/u/ambv)\ **Post date:** [November 21, 2018, 10:39pm UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/17 "2018-11-21T22:39:06Z")
@guido, AFAICT if we merge type comment support to 3.8, typed\_ast will continue to live because: - it will be essentially a backport of AST 3.8+ to previous versions; - it will be a forward port of old, incompatible ASTs to 3.8+ (3.4 without type hints and async/await, 3.5 without var annotations and async gens, 3.6 without real async/await kwds, and so on). That said, both mypy and typed\_ast are now PSF projects so it makes sense to share maintenance cost, especially if this move actually **decreases** required maintenance. If having type comment/noqa support really makes it much easier to create 3.8, 3.9, and later variants of the AST in typed\_ast, I say make a PR @guido and I will merge it. I hope you and Ivan will be around to help on the cpython end if there are any issues on the type comment front 🙂 ---
**Author:** ![njs](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/njs/32/204_2.png) [@njs](https://discuss.python.org/u/njs)\ **Post date:** [November 22, 2018, 12:03am UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/18 "2018-11-22T00:03:16Z")
It [looks like](https://github.com/python/typed_ast/blob/89242344f18f94dc109823c0732325033264e22b/typed_ast/ast3.py#L13-L25) the special features of typed\_ast are: - A number of AST classes like `FunctionDef` have an extra node attached with the type comment - There’s a new `func_type` parse mode - There’s an option to parse to selectively disable new syntax - `Module` objects have an attached list of lines that had a `# type: ignore` comment - `Str` objects have a `kind` field that lets you distinguish between `u"foo"` and `"foo"` I’m not sure why the Python-3-mode type-checker needs to distinguish between `u"foo"` and `"foo"`, but I guess the `kind` field would be a harmless addition to the stdlib in any case. I don’t know what the `func_type` parse mode does, so I can’t comment on that. (What does it do?) I’d be interested to hear whether the disable-new-syntax feature is something you think the stdlib parser should add and maintain going forward. The thing that got me looking into this though, is: for the parts involving comments, shouldn’t it be pretty straightforward to implement those without modifying the parser? It’s easy to use `tokenize` to find the locations and contents of all the comments in a source file. That’s trivially enough to let you calculate which lines have `# type: ignore` on them. And for the type comments, if you know which lines the type comments are on, and you know which lines the `FunctionDef`s and friends are on (because the AST already tracks that information), it should be pretty easy to match them up? Of course I’m fully prepared for the answer to be “yes it _should_ be but in practice it _isn’t_ because \_\_\_\_.” 🙂 (Alternatively, I can also see an argument for teaching the parser to treat type comments as an alternative syntax for type annotations, and report them directly in the `annotation` field as if they were PEP 563 deferred annotations.) ---
**Author:** ![ambv](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ambv/32/25884_2.png) [@ambv](https://discuss.python.org/u/ambv)\ **Post date:** [November 22, 2018, 12:56am UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/19 "2018-11-22T00:56:25Z")
> [@njs](#): > > if you know which lines the type comments are on, and you know which lines the `FunctionDef` s and friends are on (because the AST already tracks that information), it should be pretty easy to match them up? Answering your immediate question: this is harder than it should due to some off by one errors in line/column reporting in the AST. As a more general note, and please don’t take it personally, I sometimes see discussions mutate in this fashion: - hey, can we do $SIMPLE\_THING so that it solves $PROBLEM? - how about we explore $THING\_OF\_UNKNOWN\_SCOPE\_AND\_RISK instead which doesn’t directly solve $PROBLEM but avoids doing $SIMPLE\_THING? - nothing gets done, $PROBLEM persists There are various flavors of this but the point I’m trying to make is that this pattern derails the discussion. In our particular case my gut feeling is that it is very unlikely for mypy to spend time rewriting what typed-ast enables using the tokenizer. And it would need a non-standard parser for the multi-version support anyway. Let’s focus on how what Guido is asking for would actually help typed-ast (and mypy by extension) and how is it useful for others to have this functionality in core Python. ---
**Author:** ![njs](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/njs/32/204_2.png) [@njs](https://discuss.python.org/u/njs)\ **Post date:** [November 22, 2018, 2:19am UTC](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/20 "2018-11-22T02:19:07Z")
Sure, I don’t want to derail… I actually went back and forth a few times on whether to put a disclaimer about that in :-). It is sometimes useful to get new eyes on things before merging them into the stdlib, just in case someone notices something new :-). And it’s actually pretty unclear to me right now what exactly the proposal here is. Guido says typed\_ast adds “exactly 2 features”, but the docs list 5. Guido’s suggesting possibly expanding the scope further to track `# noqa` comments – is that the right amount of expansion? You seem to be assuming that typed\_ast’s features for parsing old 3.x features won’t be ported over, but that’s not obvious from the topic title, and in that case I don’t understand how mypy is planning to maintain its old version support… But you all certainly understand this part of the interpreter better than I do though so if my comments are distracting rather than helpful, please ignore me 🙂 It would be nice to have more accurate line/column tracking in the AST module in any case. (I’d love to see start/end positions.) [Next page](https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377.md?page=2)