# Limitations of PEP 621? **URL:** \ **Category:** Packaging\ **Created:** [January 6, 2022, 2:44pm UTC](https://discuss.python.org/t/limitations-of-pep-621/12934 "2022-01-06T14:44:34Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Conchylicultor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/conchylicultor/32/4052_2.png) [@Conchylicultor](https://discuss.python.org/u/Conchylicultor)\ **Post date:** [January 6, 2022, 2:44pm UTC](https://discuss.python.org/t/limitations-of-pep-621/12934/1 "2022-01-06T14:44:34Z")
I really like [PEP 621](https://www.python.org/dev/peps/pep-0621/). However, I think there are 2 restrictions which seems too limited for some of our projects that could prevent adoptions. Each of those 2 points are complementary and can be discussed independently. > (1) A build back-end MUST raise an error if the metadata specifies the name in dynamic. In some cases, we want the same project released under 2 different names. The main use-case for this is projects having a nightly version like: - `tensorflow` / `tf-nightly` - `tensorflow_datasets` / `tfds-nightly` - `kubric` / `kubric-nightly` > (2) Build back-ends MUST raise an error if the metadata specifies a field statically as well as being listed in dynamic. I think there are valid use-cases where the info should be defined **both** statically and dynamically. For example: - Poetry automatically update `classifiers` to append `"License :: OSI Approved :: Apache` and `Programming Language :: Python :: 3` - Tools might want to update the static version with a dynamic extension (e.g. `1.0.0+{datetime.now().isoformat()}`, `1.0.0+{git_hash}`) - Projects might want to dynamically extend the `optional-dependencies`. For examples, automatically add a `all = []` extra. - Similarly, in `tensorflow_datasets`, we could dynamically generate dataset-specific optional deps (`tensorflow_datasets[]`), but still want to keep a static `tensorflow_datasets[dev]` I’m not sure if this is the right place to open those discussion. But I’m interested about what the PEP author think. ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [January 6, 2022, 8:45pm UTC](https://discuss.python.org/t/limitations-of-pep-621/12934/2 "2022-01-06T20:45:22Z")
> [@Conchylicultor](#): > > In some cases, we want the same project released under 2 different names. The main use-case for this is projects having a nightly version like: > > - `tensorflow` / `tf-nightly` > - `tensorflow_datasets` / `tfds-nightly` > - `kubric` / `kubric-nightly` As part of your CD to build your nightly versions you could update `pyproject.toml` before doing your build, thus not requiring the name be dynamic in the version of `pyproject.toml` checked into source control. > [@Conchylicultor](#): > > Poetry automatically update `classifiers` to append `"License :: OSI Approved :: Apache` and `Programming Language :: Python :: 3` You could specify the classifiers out-of-band (e.g. in a `[tool]` section) and then append to it as necessary. > [@Conchylicultor](#): > > Tools might want to update the static version with a dynamic extension (e.g. `1.0.0+{datetime.now().isoformat()}` , `1.0.0+{git_hash}` ) That suggests then that your version number isn’t actually known statically and thus it is dynamic as it’s based on information provided at build time (see `setuptools_scm`). > [@Conchylicultor](#): > > Projects might want to dynamically extend the `optional-dependencies` . For examples, automatically add a `all = []` extra. Is maintaining that by hand too difficult? Or having part of your CI making sure that the `all` extra is up-to-date? > [@Conchylicultor](#): > > Similarly, in `tensorflow_datasets` , we could dynamically generate dataset-specific optional deps ( `tensorflow_datasets[]` ), but still want to keep a static `tensorflow_datasets[dev]` I would say you either validate/update that outside of the file or your leave it as dynamic. A key part of PEP 621 is that anything which is specified statically can be directly put into `PKG-INFO`/`METADATA` w/o modification. What you’re suggesting breaks that promise. It also means that all of that metadata is at best a hint and can’t be relied on to match what will actually be used later on which is what we were explicitly trying to get away from with PEP 621. The PEP does not prevent you from modifying the file **before** you use it, so you can always create tooling to do just that. But otherwise the suggestions above are not enough to suggest to me that we should change the spec to allow for amending/modifying something that is statically defined. ---
**Author:** ![Conchylicultor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/conchylicultor/32/4052_2.png) [@Conchylicultor](https://discuss.python.org/u/Conchylicultor)\ **Post date:** [January 6, 2022, 11:29pm UTC](https://discuss.python.org/t/limitations-of-pep-621/12934/3 "2022-01-06T23:29:27Z")
Thank you for your detailed answer. > A key part of PEP 621 is that anything which is specified statically can be directly put into PKG-INFO/METADATA w/o modification. What you’re suggesting breaks that promise. It also means that all of that metadata is at best a hint and can’t be relied on to match what will actually be used later on which is what we were explicitly trying to get away from with PEP 621. I completly agree with keeping statically defined metadata. My proposal doesn’t change this. Like today, everything not defined in `dynamic = []` is static, everything defined in `dynamic` is provided by tools. This would be exactly the same before or after my proposal. The user could simply give a hint/partial info of the `dynamic` fields. I don’t think that’s unreasonable. > You could specify the classifiers out-of-band (e.g. in a [tool] section) and then append to it as necessary. Of course, but then I feel this defeat the motivation of PEP 621 to: > Allow for more code sharing between build back-ends for the “boring parts” of a project’s metadata I think ```toml [project] name = "etils" version = "1.0.0" classifiers = [ "Intended Audience :: Developers", "Intended Audience :: Science/Research", "Topic :: Software Development :: Libraries :: Python Modules", ] # Add license classifier & version scm suffix dynamic = ["classifiers", "version"] ``` Feel more standard than: ```toml [project] name = "etils" dynamic = ["classifiers", "version"] [tool.my_tool] classifiers = [ "Intended Audience :: Developers", "Intended Audience :: Science/Research", "Topic :: Software Development :: Libraries :: Python Modules", ] ``` From the tools, there is no semantic difference between the 2 options (the `dynamic` field indicate which field is provided by tool). I just feel the first one is more “generic”. > Is maintaining that by hand too difficult? Or having part of your CI making sure that the all extra is up-to-date? Activelly maintaining many python projects is actually much more work than I wish it was, so I’m always looking to optimise my workflow as much as I can. Having a `[all]` is a very popular feature expected/requested by many users. I feel it’s a little sad if the best Python offer is to ask project mainteners to maintain it by hand. Here are some example of `[all]` requested by users. > > > As I see, \`flit install\` supports a special \`all\` dependency, i.e. \`flit install… --extras=all\`. > > However, the similar extra requirement doesn't work in the \`flit\_core\`, i.e. \`pip install '.\[all\]'\` doesn't install all extra requirements. > > Additionally, unlike \`setuptools\` + \`setup.cfg\`, it is apparently not possible to specify the \`all\` extra requirements DRYly, like \`all = %(tests)s %(dev)s\`. > > Would it make sense to support the same logic as \`flit install --extras=all\` in the \`flit\_core\`? > > > \- \[x\] I have searched the \[issues\](https://github.com/python-poetry/poetry/issue…s) of this repo and believe that this is not a duplicate. > \- \[x\] I have searched the \[documentation\](https://python-poetry.org/docs/) and believe that my question is not covered. > > \## Feature Request > > It would be handy to have a flag like \`--all-extras\` to install every package listed in any of the extras. > > The specific use case is that I want the IDE to index the packages for every subsystem (AWS Lambda jobs in this case) for easier development. > > Workarounds: > > 1. Creating a separate "extra" section with all the packages from all the other extras (\[example\](https://github.com/biocatchltd/yellowbox/pull/38)). This means duplication, which means it would be really easy for the sections to get out of sync. > 2. Listing each extra specifically in the developer instructions. > 3. \[This ugly code\](https://stackoverflow.com/a/68216833/96588), used in anger \[here\](https://github.com/linz/geostore/blob/67ac5a6f248aedb5da11f7e37d11b04a28292e8a/.github/workflows/test.yml#L39-L40): \`poetry install --extras="$(sed --quiet '/\\\[tool\\.poetry\\.extras\\\]/,/^\\\[/{s/^\\(.\*\\) = \\\[/\\1/p}' pyproject.toml | sed --null-data 's/\\n/ /g;s/ $//')" --remove-untracked\` > > > Don’t get me wrong. I’m really glad that PEP 621 exists and I really want to use it. However, for complex project, I think the specs should allow more flexible workflow. As you pointed out, there are workaround but in the end those add more complexity vs keeping a plain `setup.py` (where I can dynamically set `name`, `[all]`,… without having to hack around the limitations of PEP 621). ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [January 6, 2022, 11:42pm UTC](https://discuss.python.org/t/limitations-of-pep-621/12934/4 "2022-01-06T23:42:59Z")
> [@Conchylicultor](#): > > From the tools, there is no semantic difference between the 2 options (the `dynamic` field indicate which field is provided by tool). Perhaps, but then why even suggest you know what the dynamic fields should are when you’re admitting upfront that you don’t? > [@Conchylicultor](#): > > Having a `[all]` is a very popular feature expected/requested by many users. I feel it’s a little sad if the best Python offer is to ask project mainteners to maintain it by hand. That problem does not need to be solved by `pyproject.toml`. You can always propose a PEP for an implicit `all` for installers to support. ---
**Author:** ![Conchylicultor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/conchylicultor/32/4052_2.png) [@Conchylicultor](https://discuss.python.org/u/Conchylicultor)\ **Post date:** [January 7, 2022, 12:09am UTC](https://discuss.python.org/t/limitations-of-pep-621/12934/5 "2022-01-07T00:09:00Z")
> [@brettcannon](#): > > Perhaps, but then why even suggest you know what the dynamic fields should are when you’re admitting upfront that you don’t? If I know a field entirely: ```python [project] my_field = 'data' ``` If I don’t know a field: ```python [project] dynamic = ['my_field'] ``` If I know a field partially: ```python [project] my_field = 'data (part 1)' dynamic = ['my_field'] ``` I think defining a field partially when the data is partially know is better than nothing and feel quite intuitive for me. I also think it’s better for the readers (which gets a hint of what the data is about while being warned this data can be augmented) .I don’t think I’m being unreasonable here. ---
**Author:** ![Conchylicultor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/conchylicultor/32/4052_2.png) [@Conchylicultor](https://discuss.python.org/u/Conchylicultor)\ **Post date:** [January 7, 2022, 12:14am UTC](https://discuss.python.org/t/limitations-of-pep-621/12934/6 "2022-01-07T00:14:33Z")
> [@brettcannon](#): > > You can always propose a PEP for an implicit `all` for installers to support. I think there are good reason why this hasn’t been done/don’t exists yet (e.g. extras can be mutually incompatible). However for the projects who want this feature, I think Python should make it as easy as possible to opt-in. This was easy to automate with `setup.py`. But now, with PEP 621, this has to be done manually which feel like a step backward. Also `[all]` is just an example but there are other cases where users might want to define some deps statically and some deps dynamically. ---
**Author:** ![layday](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/layday/32/2771_2.png) [@layday](https://discuss.python.org/u/layday)\ **Post date:** [January 7, 2022, 7:38am UTC](https://discuss.python.org/t/limitations-of-pep-621/12934/7 "2022-01-07T07:38:50Z")
> [@Conchylicultor](#): > > As you pointed out, there are workaround but in the end those add more complexity vs keeping a plain `setup.py` (where I can dynamically set `name` , `[all]` ,… without having to hack around the limitations of PEP 621). FWIW, as of pip 21.2 (I think it’s 21.2), you can self-depend, so you don’t have to duplicate all of your dependencies in `all`: ```toml [project] name = "my-package" optional-dependencies.all = ["my-package[foo]", "my-package[bar]"] ``` ---
**Author:** ![Conchylicultor](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/conchylicultor/32/4052_2.png) [@Conchylicultor](https://discuss.python.org/u/Conchylicultor)\ **Post date:** [January 7, 2022, 4:15pm UTC](https://discuss.python.org/t/limitations-of-pep-621/12934/8 "2022-01-07T16:15:01Z")
Yes, this is what I’m using already for example in: [etils/pyproject.toml at 8d5981e4fcee1c4c7dc3c49cac3bbf3ef21fe46d · google/etils · GitHub](https://github.com/google/etils/blob/8d5981e4fcee1c4c7dc3c49cac3bbf3ef21fe46d/pyproject.toml#L76) In my ideal world, rather than having to manually maintain the list manually, they would be some plugin/hook which would mutate the TomlDict to automatically add those info. The most natural implementation would be something like: ```python @plugin def add_deps_all_extras(pyproject: TomlDict) -> TomlDict: pyproject['optional-deps']['all'] = list( itertools.chain(*pyproject['optional-deps'].values()) ) return pyproject @plugin def add_scm_suffix_to_version(pyproject: TomlDict) -> TomlDict: version = pyproject['version'] pyproject['version'] = f'{version}+{get_git_hash()}' return pyproject ``` Those implenentations feel very natural I think, but only works if we allow PEP 621 field to be mutated (field both in static and dynamic). ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [January 7, 2022, 11:31pm UTC](https://discuss.python.org/t/limitations-of-pep-621/12934/9 "2022-01-07T23:31:03Z")
> [@Conchylicultor](#): > > I don’t think I’m being unreasonable here. It’s not a question of being unreasonable, I just personally disagree with your proposal. Since PEP 621 has been accepted and is being used in the community you will need to write a PEP to propose your changes and get general buy-in from folks to support your PEP. I’m just one person so me disagreeing is very much not a showstopper, but I would advise you wait to see if at least a few people step forward in support before putting in the time to write a PEP. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [January 8, 2022, 10:33am UTC](https://discuss.python.org/t/limitations-of-pep-621/12934/10 "2022-01-08T10:33:36Z")
> [@brettcannon](#): > > It’s not a question of being unreasonable, I just personally disagree with your proposal. FWIW, I’m in agreement with @brettcannon here. But I will say that my view is mostly theoretical, as I’ve not used PEP 621 much in practice. So views from people who have found this to be a significant practical issue would be useful if you want to try to write a PEP proposing the change you’re suggesting. ---
**Author:** ![CAM-Gerlach](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cam-gerlach/32/3688_2.png) [@CAM-Gerlach](https://discuss.python.org/u/CAM-Gerlach)\ **Post date:** [January 8, 2022, 5:27pm UTC](https://discuss.python.org/t/limitations-of-pep-621/12934/11 "2022-01-08T17:27:26Z")
As a disclaimer, I’m not a packaging expert or one of the PEP 621 author like the others here, just someone who has read and re-read it as part of my intersecting work on PEP 639. The `dynamic` key and its semantics provide a guarantee that the source metadata specified in the `project` table of `pyproject.toml` can be relied upon by other tools, at and after package build time. At present, if a key is marked as dynamic, the field(s) it maps to can be modified in arbitrary ways by individual build tools, and thus you cannot > [@Conchylicultor](#): > > Allow for more code sharing between build back-ends for the “boring parts” of a project’s metadata because the input data cannot be relied upon to not be modified in arbitrary ways by different build tools, and the handling is defined to be inherently bespoke and _not_ shared between different build tools. While a key could be still be specified under `project` rather than `tool`, it would no longer be static and would be tool-specific, and thus breaks the `project` table’s current guarantee that anything specified there would be static and not modified in arbitrary tool-specific ways. That said, given good cause and a PEP, there might be a few mechanisms that could potentially allow most of the behavior you mention, while still retaining many of the guarantees provided by PEP 621 and benefits thereof; however, the latter do (which more specifically do what you’re asking) do raise backward compat concerns and would come at a cost of tool and specification complexity. First, the functionality could be provided via other mechanisms than including it in the static metadata, as suggested by others here. Second, the guarantee could be relaxed very slightly to allow deviations from verbatim copying for one or more specific fields, if explicitly specified by a PEP, are suitably straightforward and have a deterministic outcome (i.e. given a project source tree and `project` table contents, all tools following the specification will produce the same output every time. This is more or less the approach I specify for a couple of particular (new) cases in PEP 639, with clear rationale for why it needed to be taken. To note, however, for PEP 621 keys/core metadata fields that aren’t newly defined (as both were in PEP 639), this creates potential backward compatibility issues, as older tooling not aware of the specification will produce different metadata in the build distribution, meaning the metadata isn’t truly “static” after all. Third, specific fields (for example,`classifiers`) could be allowed to be both listed as `dynamic` and included under `project`, which would relax the guarantee in a particular defined way, so that the provided metadata can still be relied on to _some_ extent by other tools, rather than the intended output being completely tool-dependent and non-interoperable. For a field like `classifiers`, you could specify, for example, that if `classifiers` is both included and marked as `dynamic`, individual classifiers can be added but not removed or modified. Existing tools _should_ (per the spec) error out, since the same field is included in both `dynamic` and the `project` table, but that’s better then them silently handling the metadata incorrectly—however, if any tools rely on PEP 621’s existing guarantee that any key included under `project` must map statically to a metadata field, then they will silently behave incorrectly. Finally, as a more radical form of the above, you could specify that if any key is both specified under `dynamic` and included under `project`, tools should only assume that it is the user input in a standardized form, but the actual output metadata is arbitrary and tool-dependent. This would accomplish the limited goal of being a standard _user input_ format for project source metadata, but given different tools could handle any such keys in arbitrary ways in the output fields, it would be no more than a hint to tools about what the user intends, and users switching tools may not expect the output metadata to silently change in arbitrary ways and/or the input metadata to need modification, plus the above concerns about any tools currently relying on such fields. Not saying these are necessarily good or bad ideas, but hopefully they are helpful in giving you some different directions in which to take a potential proposal that achieves what you’re looking for while having a better chance of being considered.