Help building python parser

Python Help
1

Hi
I’m trying to build the python peg parser following the instructions on the developers pages. I get some errors, see below.

I’m using windows 10 and python 3.10.4 and I’m building from the source for that release.
I’m trying to do this as a standalone. I dont want to completely rebuild python on windows and I dont want to use linux where I can build python.
First I try yo build parser,py - this gives errors but the file is generated. But its broken as well.

I’ve tried to make sure all the code and tools are from 3.10.4
The errors are below
Thanks in advance
John


H:.\peg_generator> py -m pegen -v python python.gram
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\__main__.py", line 185, in <module>
    main()
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\__main__.py", line 129, in main
    grammar, parser, tokenizer, gen = args.func(args)
                                      ^^^^^^^^^^^^^^^
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\__main__.py", line 58, in generate_python_code
    grammar, parser, tokenizer, gen = build_python_parser_and_generator(
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\build.py", line 259, in build_python_parser_and_generator
    gen = build_python_generator(grammar, grammar_file, output_file, skip_actions=skip_actions,)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\build.py", line 192, in build_python_generator
    gen.generate(grammar_file)
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\python_generator.py", line 154, in generate
    self.visit(rule)
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\grammar.py", line 32, in visit
    return visitor(node, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\python_generator.py", line 181, in visit_Rule
    self.visit(rhs, is_loop=is_loop, is_gather=is_gather)
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\grammar.py", line 32, in visit
    return visitor(node, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\python_generator.py", line 202, in visit_Rhs
    self.visit(alt, is_loop=is_loop, is_gather=is_gather)
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\grammar.py", line 32, in visit
    return visitor(node, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\python_generator.py", line 218, in visit_Alt
    self.visit(item)
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\grammar.py", line 32, in visit
    return visitor(node, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\pegen\python_generator.py", line 188, in visit_NamedItem
    name, call = self.callmakervisitor.visit(node.item)
    ^^^^^^^^^^
TypeError: cannot unpack non-iterable NoneType object

H:.\peg_generator> py parse.py test.py
Requested Python version (3.8) is not installed

H:.\peg_generator> py parse.py test.py
  File "H:\Computers\PythonTools\ideas\MyLisp\peg_generator\parse.py", line 33
    return _PyAST_Interactive ( a , p -> arena )
                                      ^^
SyntaxError: invalid syntax

H:.\peg_generator>
2

Going out on a limb here since I don’t know anything about the peg parser, so excuses in advance if this is just wildly off –
The python code in parse.py is definitely wrong. Should “p → arena” have been a quoted string? Is this error caused by not using appropriate quotes in a grammar file? (Does peg not have a way perhaps to first test the formal validity of grammars before trying to parse them?)

1 Like
3

Thanks, it does look wrong, but its computer generated code, so i think its a consequnce of an earlier error. ‘->’ looks like typing
J

1 Like
4

Yes, I assumed it was generated code. “p → arena” didn’t look like typing to me, but rather looks like some grammar rule that needs to be parsed. Since the stack trace also seems to indicate an error on a node that probably is None, I’m guessing that in the input Lisp grammar there must be some typo.
If so – again going further out on a limb – it might be strange that peg doesn’t give a better error message.

5

That’s obviously a bit of C.

In the build it is used to generate a parser in C, of course, and python.gram is full of fragments of C (actions) that should appear in the generated parser.

But it does claim to generate them in Python too. Clearly some C action is leaking out where there should have been a bit of Python. I suspect this is simply a bug that has crept in where changes were made, and no-one noticed because it isn’t used in the build.

Or it may be a misunderstanding. When generating in Python, where are the actions defined? I have only experimented with a modified pegen that takes this one as a starting point.

1 Like
6

Thanks all.
I’ve just tried it on linux with pygen from pypi and I get exactly the same error with the python.gram file.
J

7

According to the Guide to the Parser, you need a grammar in which the actions are written in Python.

There is an option --skip-actions to suppress the actions from the grammar, which ought to mean you don’t get C leaking through, but it makes no real difference with the python option. (Recall I’m using a borrowed version and this may be different in the official repo.)

8

Thanks, I went back and looked at what I had done. Using pegen from PyPi I was able to build a parser when I used a grammar without C actions. This parsed a simple program to generate an textual ast tree. Now I can move on. The trick seems to be to ignore content from the python source files and work with the pegen package only.
Thanks again
John