Help testing experimental features in setuptools

Packaging
62

I tested this with https://github.com/pypa/sampleproject, including the PEP 621 support, and it seems to work as expected.

One question though: am I understanding correctly from https://setuptools.pypa.io/en/latest/userguide/datafiles.html that using data_files (e.g. this pattern) is completely deprecated and there is no replacement?

(That’s totally fine with me if so and makes sense, just wanted to confirm my understanding)

63

Hi Dustin, thank you very much for testing the changes! Glad to hear everything went fine.

Regarding data_files, the existing docs were already mentioning that the users should move away from it, so what I did at the end of the last year was to remove the examples from the docs, because it was very common to have people asking about how to install files in arbitrary locations of the users’ machine.

That said, I think we can review the docs and expose this feature in a more controlled way instead of considering it completely deprecated.

My personal opinion is that there are genuine use cases for data files (e.g. jupyter extensions), but they are very advanced and rare. We should not encourage new-comers/first-time-packagers to adopt it.

TLDR: The behaviour of installing things at arbitrary locations is definitely deprecated. For the time being let’s assume that data_files is at least discouraged and a power-user-only feature.

64

Speaking from the Jupyter side, we are currently trying to migrate away from setuptools to use flit because it recently added support for external_data, which is a constrained version of data_files. Maybe we could consider adding an equivalent feature to setuptools instead?

2 Likes
65

That sounds very reasonable to me.

I would still consider it a power-user feature though.

It is very easy for new-comers/new-packagers to see the examples and jump into the conclusion that they should be shipping configuration files, etc as data_files, when for most use cases in-package immutable files + platformdirs solve the majority of the problems.

3 Likes
66

I agree. We should specify that it is only for files that need to exist at install time that need to be consumed by another package in a shared location, and give examples.

2 Likes
67

By the way setuptools 61.0.0 is finally out, with experimental support for pyproject.toml metadata and auto-discovery of packages.

Thank you everyone for the immense support and help!

10 Likes
68

Thanks to you, @abravalheri , for doing the immense amount of development, testing and outreach to make this very important and long awaited feature land!

3 Likes
69

I opened [FR] External Data Support · Issue #3191 · pypa/setuptools · GitHub to discuss the External Data feature.

2 Likes
70

This is awesome! And when PEP 660 support lands, I can finally do away with setup.cfg.

2 Likes
71

Thank you @abravalheri, nice job!

5 Likes
72

Many thanks for your continuous commitment, awesome job! @abravalheri
have created a tiny sample project and the configuration worked successfully with many of my projects.

3 Likes
73

Thank you very much!

I am really impressed! This example is very thoughtful and complete.
Thank you very much for putting it togheter.

2 Likes
74

For reference, I opened [BUG] pyproject.toml-only project does not support editable installation · Issue #3226 · pypa/setuptools · GitHub about editable installs. I’m not sure if editable installs were supposed to work with pyproject.toml-only?

75

Hi Moshe.
Yes this is a well know limitation because of the fact setuptools does not implement PEP 660 yet.
I will add this to the list in the first post of this thread.

The existing workaround for it is to either (a) have an empty setup.cfg or (b) add a stub-only setup.py (import setuptools; setuptools.setup()).

76

From the GitHub issue

EpicWink:
you say “some” support, but I don’t see what isn’t supported

abravalheri:
Hi @EpicWink, from the top of my head:

If it’s just limited to not running validation with the project table in the pyproject TOML only has requires-python, I’d say that’s pretty close to full support for PEP 621 / pyproject TOML, regardless of the setuptools features not supported in it. I think it would be better to communicate it as such, while continuing to say that it is experimental, with the tool.setuptools table being non-final.

  • The aspects of setuptools configuration not covered by PEP 621 are currently implemented via [tool.setuptools] .
    The final shape of [tool.setuptools] and which exact fields it will support still needs to be debated and might change.
    • Configurations that are currently not supported in setup.cfg are also not supported in pyproject.toml (e.g. ext_modules ). In the long term we might decide to support them or not.
    • setup.cfg supports even more configurations, e.g. tag_build , tag_date , etc. These need even more debate than [tool.setuptools]

I know the end goal is to replace setup.cfg with pyproject.toml, but for now pyproject TOML support seems fulfilled with some weird values left in setup.cfg.

I see setup.py as a build-script, not metadata declaration file, and as such it makes sense for me to keep custom actions in it (such as yaml-to-json conversion for some data-files, Cython (etc) extension modules). I’m not sure if it will ever make sense to put these in any declarative file, however the in-tree PEP 517 backend seems like a place to move it.

There’s quite a lot of reference to Pip in the documentation, especially with references to ignored fields. Has there been testing with other build front-ends?

2 Likes
77

Hi Laurie, all your comments make a lot of sense to me. I will add those changes in the docs to the TODO list (but if anyone would like to suggest a PR that would be really appreciated!).

The changes were tested with build and pip.

If you are referring to provides and obsoletes in Configuring setuptools using pyproject.toml files - setuptools 61.2.0.post20220327.post-20220327 documentation (pypa.io), maybe it makes more sense to ask about other installers instead of front-ends. In that case the answer would be no.

I haven’t tested with other installers, but there are tests in place that compare the generated core metadata between setup.cfg and the equivalent pyproject.toml, trying to find regressions.

The lack of support for obsoletes and provides is something that is related to the history of the core metadata format, and it was mentioned in Keywords - setuptools 61.2.0.post20220327.post-20220327 documentation (pypa.io) and Core metadata specifications — Python Packaging User Guide prior to the latest changes.

These fields are currently kept in the [tool.setuptools] table for compatibility with the setup.cfg format, since there is an announced intention of eventually deprecating setup.cfg in favour of automatically “transpiling” it to pyproject.toml (see Discussion: support for pyproject.toml configuration · Issue #1688 · pypa/setuptools · GitHub)

3 Likes
78

Also, just to note, while provides and obsoletes are metadata fields and not tool config, they are not specified in PEP 621 [project] format due to lack of significant use or general applicability (for the reasons mentioned in the core metadata spec), so they can’t be included in [project] anyway.

79

We’ve been thinking of setting up CI using a Nix Hydra to test a subset of the Nix Python packages set with setuptools master integrated. As setuptools maintainer, would you @abravalheri be interested in seeing such data? For the majority of packages we also run unit tests.

80

Hi @FRidh, thank you very much for setting it up it looks like a great idea. I would be definitely interested in having access to the data. It can be very valuable when introducing changes, testing new features, etc…

I cannot commit to help maintaining it though, that would be a bit too much on my plate :sweat_smile:.

2 Likes
81

Hello, how to properly use version read from file? I trying to create package from flask project.

Tree:

.
├── CHANGELOG.md
├── pyproject.toml
├── README.md
├── setup.py
├── src
│   └── hdmiswitcher
│       ├── create_service.sh
│       ├── hdmiswitcher.py
│       ├── hdmiswitcher.service
│       ├── __init__.py
│       ├── modules
│       │   ├── modul1.py
│       │   ├── modul2.py
│       │   └── modul3.py
│       ├── requirements.txt
│       ├── static
│       │   ├── css
│       │   │   └── style.css
│       │   └── favicon.ico
│       └── templates
│           ├── actual_status_model.html
│           ├── error.html
│           └── index.html
└── VERSION

My pyproject.toml

[build-system]
requires = ["setuptools >= 61.0.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "hdmiswitcher"
description = "hdmiswitcher testing v2"
requires-python = ">=3.7"
classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: Linux",
]
dynamic = ["version", "readme", "entry-points"]

[project.urls]
"Homepage" = "https://github.com/pypa/sampleproject"
"Bug Tracker" = "https://github.com/pypa/sampleproject/issues"

[tool.setuptools]
include-package-data = true

[tool.setuptools.dynamic]
version = {file = "VERSION"}
readme = {file = ["README.md"]}

[tool.setuptools.packages.find]
where = ["src"]

[tool.setuptools.package-data]
hdmiswitcher = ["*.txt", "*.sh", "*.service"]
"hdmiswitcher.static" = ["*"]
"hdmiswitcher.static.ccs3" = ["*"]
"hdmiswitcher.templates" = ["*"]
"hdmiswitcher.modules" = ["*.py"]

VERSION file contain 0.0.1 and is maintained by git flow.

Building with python3 -m build produce source package correctly with version 0.0.1
but building wheel produce whl with version 0.0.0

In log, I can see:

/tmp/build-env-l9mkhi6m/lib/python3.9/site-packages/setuptools/config/expand.py:142: UserWarning: File '/tmp/build-via-sdist-b12k2_z4/hdmiswitcher-0.0.1/VERSION' cannot be found
  warnings.warn(f"File {path!r} cannot be found")
...
Successfully built hdmiswitcher-0.0.1.tar.gz and hdmiswitcher-0.0.0-py3-none-any.whl