Code generation before packaging, migration from setup.py to pyproject.toml

Packaging
1

I want to migrate a project which has some code generation (build-time dependency) using jinja2, json5 and grpcio-tools . Previously I had a requirements-build.txt and requirements.txt, and a setup.py with a generate ‘target’. Now I would like to migrate to a more modern approach (setuptools complains when I directly invoke the ‘generate’ target about being deprecated). I am however a bit at a loss of how to approach this.

I first started reading pyproject.toml related PEPs and then ran across flit. This seems nice and makes sense, but was somewhat limited for my purpose.

Then I found poetry. From what I can tell, I can define build dependencies, run pre-build scripts and such. But reading the documentation makes me very confused, as it seems like it sort of hijacks the pyproject.toml. Afaict it sort-of forcing pipx, and uses version dependencies with ^ (instead of > / >= ?), which I don’t think is defined in PEP 508. And why does the default poetry pyproject.toml not use PEP621 and replace [project] with [tool.poetry], having similar meta-data, for, far as I can tell, no specific reason? So, maybe I’m all wrong here, but I’m quite a newbie to this all, and if it confuses me, I don’t think I am the only one. /rant

Is there a ‘standard practice’ to add pre-packaging code generation, and specify build and run-time dependencies without setup.py? Which tools would you recommend?

1 Like
2

Poetry predates PEP 621, and there is a bug open to implement it: Support for PEP 621 · Issue #3 · python-poetry/roadmap · GitHub. Unfortunately, it doesn’t seem to be progressing. As part of that, removal of ^ was suggested as well.

I think hatchling supports running custom code as part of the build process. Unfortunately, I haven’t used it and I don’t have the docs handy.

There is also the option of writing a custom PEP 517 backend that basically does any extra work you need it to do, then calls the function of another build backend. This is e.g. how pkgcore combines flit-core with custom build commands.

3 Likes
3

So far, there is no standard practice for that.
Depending on the build backend you select, you will have to use a different strategy.

If your code generation requires you to write imperative code you might end up with a different .py file in your project implementing the necessary steps.

In the case you want to use setuptools you can try to add a custom build step (which would require you to have a setup.py - however please have in mind that the setup.py file itself is not deprecated, it is a perfectly valid configuration file, what is deprecated is running it directly as a script or CLI tool - you can always keep all your static metadata in pyproject.toml and use setup.py just for the imperative code that describes the custom build step).

1 Like
4

I am going to have the same requirement on a $work project, although I’m planning on using PDM.

5

Can something like this help choosing the right build back-end? Packaging tools comparisons — Sinoroc KB (feel free to leave comments so that I can improve the document).

This page might also help: Python package structure information — Python Packaging Guide

And then a question… Does the dynamic part (the code generation) happen before the sdist, or between the sdist and the wheel(s)?

6

That is correct and is one of the core features of the Hatch ecosystem!

1 Like
7

This would certainly be handy do add. A good overview of the different build/packaging tools with their capabilities is good to have. For me it was not easy to piece everything together.

8

Thanks everyone for replying! I will try the different solutions provided and see what best fits my use-case.

9

For code generation I had to invoke it as a script, i.e.:

python ./setup.py generate

And then get the deprecated warning. Maybe there is a way to disable it, but for me it meant I wasn’t on the right track.

10

Hi @plinnie, running python setup.py xxxxxx directly is indeed deprecated.

But you can still have a setup.py file with a custom build step, and that would be automatically invoked when you run python -m build or pip install ..

The link I provided before has a more complete skeleton, but the main idea is something along the following lines:

from setuptools import Command, setup
from setuptools.command.build import build


class Generate(Command):
    ...


class CustomBuild(build):
    sub_commands = [("generate", None), *build.sub_commands]
    # ^-- this will result in `generate` being called automatically when `build` is called.


setup(
    cmdclass={"build": CustomBuild, "generate": Generate},
    # The rest of the static metadata can go in `pyproject.toml`
)

This will probably get you half way there, for a full integration with editable installs you might need to implement a few extra methods, but if you are interested in pursuing this root please have a look on our docs. You can submit setuptools specific questions in the setuptools discussions.

11

We had similar code-gen steps [1] and had a bit of a journey getting to a happy place. But everyone here was extremely helpful and we did eventually arrive at a good solution that no longer relied on invoking setup.py directly. In case it is useful, here is that thread:

  1. typescript compilation, specifically ↩︎

3 Likes
12

Hi Ofek,

I stated using hatch, and I’m very happy with it. After some initial confusion about dependencies, and environments, it has been smooth sailing. Great work!

Cheers,
Vincent

2 Likes