Dynamic metadata populated by entry-points

Packaging
1

I have projects with (some) PEP 621 metadata fields populated by Setuptools entry points. How can I satisfy the (looming) hard requirement to declare these as dynamic, and that they are populated by an entry-point function?

I am busily migrating projects from executable ‘setup.py’-based configuration to PEP 621, and some of the metadata is dynamically generated.

This is working fine, until the PEP 517 conformant build tool wants to know how these fields are defined; I don’t know what is needed in ‘pyproject.toml’ to tell it the situation.

Examples:

  • The distribution version field is derived from the ChangeLog document as a single point of truth. After migrating from a function called from ‘setup.py’, this is now done with an entry-point, declared in ‘pyproject.toml’:
[project.entry-points."setuptools.finalize_distribution_options"]
# Set the distribution version.
version = "util.packaging:derive_version"

The function util.packaging.derive_version(distribution) correctly determines the version string, and sets it on distribution.metadata.version. This works great.

  • The distribution description and readme fields are derived from the main package’s docstring, as a single point of truth for that information. (This is done so that in order to update the project description, one updates the docstring, which ensures that aligns with the description.)
[project.entry-points."setuptools.finalize_distribution_options"]
# Set the description fields ‘description’, ‘readme’.
description_fields = "util.packaging:derive_dist_description"

Again, this works great and the correct metadata is generated in the distribution files. But:

##########################################################################
# configuration would be ignored/result in error due to `pyproject.toml` #
##########################################################################

The following seems to be defined outside of `pyproject.toml`:

`description = 'Lorem ipsum, dolor sit amet.'`

According to the spec (see the link below), however, setuptools CANNOT
consider this value unless `description` is listed as `dynamic`.

https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

However, when I follow that instruction and set the fields in project.dynamic:

dynamic = [
    "version",
    "description",
    "readme",
    ]

then setuptools.build_meta crashes:

distutils.errors.DistutilsOptionError: No configuration found for dynamic 'description'.
Some dynamic fields need to be specified via `tool.setuptools.dynamic`
others must be specified via the equivalent attribute in `setup.py`.

Since it’s satisfied in a project.entry-points."setuptools.finalize_distribution_options" entry point, how do I inform the build system of this so builds will succeed?

2

Hatchling supports this easily with build hooks:

3

Supports what easily? You’re showing me a custom build hook, but I already have that implemented, using the setuptools.finalize_distribution_options entry-point.

So I already have the metadata generated correctly, and it appears correctly in the build artefacts.

What I don’t have, and what the tools tell me I need this month before everything breaks, is a way to specify in ‘pyproject.toml’ that metadata has been generated dynamically.

4

I’m not familiar with the setuptools way but I guarantee what I linked will not produce an error.

5

I’m imagining something like:

[tool.setuptools.dynamic]
version = { entry-point = "setuptools.finalize_distribution_options" }
description = { entry-point = "setuptools.finalize_distribution_options" }
readme = { entry-point = "setuptools.finalize_distribution_options" }

Of course, there’s no such entry-point value type defined at Configuring setuptools using pyproject.toml files - setuptools 69.0.2.post20231122 documentation — yet I don’t see any other way to specify that it’s an entry point satisfying these dynamic fields?

6

This is a setuptools issue - you’d need some way to tell setuptools not to raise that error. I suggest raising this on the setuptools tracker as you’ll get more focused help there.

Edit: Having read the linked documentation on tools.setuptools.dynamic it looks like setuptools doesn’t currently allow plugins (entry points) to supply dynamic metadata, so this is probably going to be a feature request.

2 Likes
7

You’re talking past each other. You have a problem with setuptools, @ofek is saying that you could use hatchling instead, where what you’re trying to do is easy.

Personally, I’ve found setuptools much harder to understand and configure than flit or hatchling (the former doesn’t support plugins), so if I were you, I’d follow the advice to switch to hatchling (the PEP 621 metadata is the same, what differs is the non PEP 621 metadata, such as configuration of dynamic values). But if you prefer setuptools, then as @pf_moore said, the best place to ask would be setuptools’ issue tracker.

8

I can’t speak for the OP, but as they are migrating an existing project with custom setuptools code, switching to a new build system is potentially a lot bigger exercise than simply changing how the data is provided.

9

The problem is the following:

When you declare entry-points, they are “guaranteed” be available after your project is built and installed. But you want to use entry-points while building the package.

Normally that would not work (e.g. in the past python setup.py build was likely to fail in a fresh checkout). The reason why this partially works is because coincidentally when calling pip install . or python -m build, the first of the many build API hooks that runs (specified in PEP 517/660) will cause setuptools create/cache an .egg-info folder at the root of your project (for some specific project layouts). The coincidence happens because the follow up hooks will call importlib.metadata.entry_points() that happens to also consider the .egg-info directory you recently generated.

The reason why it does not fully work is because the first build API hook to be called (the one that eventually generates the .egg-info folder before the other hooks run) will still be called with an imperfectly constructed setuptools.Distribution object, which is error prone.

Please note that documentation mentions the setuptools.finalize_distribution_options entry-point in the context of external plugins, not in the context of local customisations. The following admonition, marked as important in the documentation, is relevant:

Any entry-point defined in your setup.cfg, setup.py or pyproject.toml files are not immediately available for use. Your package needs to be installed first, then setuptools will be able to access these entry points. For example consider a Project-A that defines entry points. When building Project-A, these will not be available. If Project-B declares a build system requirement on Project-A, then setuptools will be able to use Project-A’ customizations.

Please note however that you can still mix and match setup.py and pyproject.toml. So when adopting pyproject.toml you can move all your static configuration to the new file, however you can keep all the imperative code for dynamic metadata in setup.py (as long as you fill in pyproject.toml' project.dynamic accordingly)[1].

Contrary to the urban legend, setup.py is not deprecated. What is deprecated is running it as a CLI script.

Think about setup.py the same way as conftest.py for pytest or noxfile.py for nox. You don’t run them directly, but they are still perfectly fine configuration files, that happen to be written using the Python programming language.

  1. Also note that setuptools will not add CWD to sys.path. If you are importing other files in setup.py you have to manipulate sys.path accordingly before doing the import. The reason why that would work in the past with python setup.py without explicit user intervention is that Python automatically adds the directory of the script it is running to sys.path. When using the build API hooks the setup.py is not executed with python setup.py – that is what is deprecated after all… ↩︎

10 Likes
10

This is very helpful, thank you!

Okay, I was treating the setuptools.finalize_distribution_options entry-point as a build hook for the purpose of ‘pyproject.toml’. You’ve convinced me that’s not a good use of this.

But for now, in order to satisfy the PEP 517 build tools and the pending deadline at the end of this month, I will lean on that entry-point in these projects, leave the entry-points as Setuptools-specific configuration, and:

This is exactly my position, yes. I will consider some other build system in future when there is leisure to do so.

2 Likes
11

Thank you very much to everyone who helped, and for the prompt responses to a time-sensitive problem.

In particular, thank you everyone for avoiding the common trope of “what you want is wrong, you should want something else” which would not have been helpful. I’m impressed with the attention of everyone here to the actual issues and framing of advice in that context.

3 Likes
12

Consider the following:

# pyproject.toml

[build-system]
# ...

[project]
name = # ... 
author = # ...
keywords = # ...
license = # ...
urls = # ...
classifiers = #...
requires-python = # ...
dynamic = ["version", "description", "readme"]

# setup.py
from setuptools import setup

def derive_version():
      ...

def derive_description():
      ...

def derive_readme():
      ...

setup(
   version=derive_version(),
   description=derive_description(),
   long_description=derive_readme(),  # unfortunately the nomenclature varies a bit,
                                      # but PEP 621 text shows the equivalence.
)

After replacing the placeholders in the TOML with the actual data, this should give you a valid configuration[1].

It is not deprecated and it is a well-known/supported pattern.
(Also it if you have a build-system table in your pyproject.toml, adding a [project] table is optional, you can also keep using setup.py to configure your build - again not deprecated, supported and PEP 517-ready).

  1. Remembering to add os.path.dirname(__file__) to sys.path if you need to do local imports. Since setup.py is not directly run with python setup.py, Python does not automatically add the directory containing the script to sys.path, as it does when you run scripts. ↩︎

1 Like