Code generation before packaging, migration from to pyproject.toml

I want to migrate a project which has some code generation (build-time dependency) using jinja2, json5 and grpcio-tools . Previously I had a requirements-build.txt and requirements.txt, and a with a generate ‘target’. Now I would like to migrate to a more modern approach (setuptools complains when I directly invoke the ‘generate’ target about being deprecated). I am however a bit at a loss of how to approach this.

I first started reading pyproject.toml related PEPs and then ran across flit. This seems nice and makes sense, but was somewhat limited for my purpose.

Then I found poetry. From what I can tell, I can define build dependencies, run pre-build scripts and such. But reading the documentation makes me very confused, as it seems like it sort of hijacks the pyproject.toml. Afaict it sort-of forcing pipx, and uses version dependencies with ^ (instead of > / >= ?), which I don’t think is defined in PEP 508. And why does the default poetry pyproject.toml not use PEP621 and replace [project] with [tool.poetry], having similar meta-data, for, far as I can tell, no specific reason? So, maybe I’m all wrong here, but I’m quite a newbie to this all, and if it confuses me, I don’t think I am the only one. /rant

Is there a ‘standard practice’ to add pre-packaging code generation, and specify build and run-time dependencies without Which tools would you recommend?

1 Like

Poetry predates PEP 621, and there is a bug open to implement it: Support for PEP 621 · Issue #3 · python-poetry/roadmap · GitHub. Unfortunately, it doesn’t seem to be progressing. As part of that, removal of ^ was suggested as well.

I think hatchling supports running custom code as part of the build process. Unfortunately, I haven’t used it and I don’t have the docs handy.

There is also the option of writing a custom PEP 517 backend that basically does any extra work you need it to do, then calls the function of another build backend. This is e.g. how pkgcore combines flit-core with custom build commands.


So far, there is no standard practice for that.
Depending on the build backend you select, you will have to use a different strategy.

If your code generation requires you to write imperative code you might end up with a different .py file in your project implementing the necessary steps.

In the case you want to use setuptools you can try to add a custom build step (which would require you to have a - however please have in mind that the file itself is not deprecated, it is a perfectly valid configuration file, what is deprecated is running it directly as a script or CLI tool - you can always keep all your static metadata in pyproject.toml and use just for the imperative code that describes the custom build step).

1 Like

I am going to have the same requirement on a $work project, although I’m planning on using PDM.

Can something like this help choosing the right build back-end? Packaging tools comparisons — Sinoroc KB (feel free to leave comments so that I can improve the document).

This page might also help: Python package structure information — Python Packaging Guide

And then a question… Does the dynamic part (the code generation) happen before the sdist, or between the sdist and the wheel(s)?

That is correct and is one of the core features of the Hatch ecosystem!

1 Like

This would certainly be handy do add. A good overview of the different build/packaging tools with their capabilities is good to have. For me it was not easy to piece everything together.

Thanks everyone for replying! I will try the different solutions provided and see what best fits my use-case.

For code generation I had to invoke it as a script, i.e.:

python ./ generate

And then get the deprecated warning. Maybe there is a way to disable it, but for me it meant I wasn’t on the right track.

Hi @plinnie, running python xxxxxx directly is indeed deprecated.

But you can still have a file with a custom build step, and that would be automatically invoked when you run python -m build or pip install ..

The link I provided before has a more complete skeleton, but the main idea is something along the following lines:

from setuptools import Command, setup
from import build

class Generate(Command):

class CustomBuild(build):
    sub_commands = [("generate", None), *build.sub_commands]
    # ^-- this will result in `generate` being called automatically when `build` is called.

    cmdclass={"build": CustomBuild, "generate": Generate},
    # The rest of the static metadata can go in `pyproject.toml`

This will probably get you half way there, for a full integration with editable installs you might need to implement a few extra methods, but if you are interested in pursuing this root please have a look on our docs. You can submit setuptools specific questions in the setuptools discussions.

We had similar code-gen steps [1] and had a bit of a journey getting to a happy place. But everyone here was extremely helpful and we did eventually arrive at a good solution that no longer relied on invoking directly. In case it is useful, here is that thread:

  1. typescript compilation, specifically ↩︎


Hi Ofek,

I stated using hatch, and I’m very happy with it. After some initial confusion about dependencies, and environments, it has been smooth sailing. Great work!