Trouble using setuptools with only a pyproject.toml file

Python Help
,
1

I’m trying to make a working pure-pyproject.toml file for my modules. I’ve run out of ideas and lost patience with the setuptools documentation.

Background: I’ve been publishing some modules to PyPI for years. A few years back I upgraded my ancient setup.py based setuptools stuff to use pyproject.toml, with an adjacent setup.cfg file. Now I’m using only the pyproject.toml file and getting broken wheel builds. I haven’t been able to see the configuration error.

As my working example, my cs.gimmicks module. Here’s the contents of an older correct wheel file:

% unzip -l cs.gimmicks-20230212-py3-none-any.whl
Archive:  cs.gimmicks-20230212-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
     3916  02-12-2023 00:37   cs/gimmicks.py
     2190  02-12-2023 00:37   cs.gimmicks-20230212.dist-info/METADATA
       92  02-12-2023 00:37   cs.gimmicks-20230212.dist-info/WHEEL
        3  02-12-2023 00:37   cs.gimmicks-20230212.dist-info/top_level.txt
      396  02-12-2023 00:37   cs.gimmicks-20230212.dist-info/RECORD
---------                     -------
     6597                     5 files

This unpacks directly into site-packages making a cs/gimmicks.py file. That’s what should happen.

With my newer builds the wheel files look like this:

% unzip -l dist/cs.gimmicks-20230331-py3-none-any.whl
Archive:  dist/cs.gimmicks-20230331-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
     4840  03-31-2023 10:02   lib/python/cs/gimmicks.py
     2529  03-15-2024 22:54   cs.gimmicks-20230331.dist-info/METADATA
       92  03-15-2024 22:54   cs.gimmicks-20230331.dist-info/WHEEL
        4  03-15-2024 22:54   cs.gimmicks-20230331.dist-info/top_level.txt
      407  03-15-2024 22:54   cs.gimmicks-20230331.dist-info/RECORD
---------                     -------
     7872                     5 files

which unpacks a lib/python/cs/gimmicks.py file. That reflects my source tree, where my Python stuff is all in a lib/python subdirectory - effectively this is the “src layout” approach, but with a different subdirectory. This is supposed to be happily supported by setuptools.

My release process generates a pyproject.toml with this content (minus the inline [project.readme] table):

[project]
name = "cs.gimmicks"
description = "Gimmicks and hacks to make some of my other modules more robust and less demanding of others."
authors = [
    { name = "Cameron Simpson", email = "cs@cskk.id.au" },
]
keywords = [
    "python2",
    "python3",
]
dependencies = []
classifiers = [
    "Programming Language :: Python",
    "Programming Language :: Python :: 2",
    "Programming Language :: Python :: 3",
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "Operating System :: OS Independent",
    "Topic :: Software Development :: Libraries :: Python Modules",
    "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
]
version = "20230331"

[project.license]
text = "GNU General Public License v3 or later (GPLv3+)"

[project.urls]
URL = "https://bitbucket.org/cameron_simpson/css/commits/all"

[build-system]
build-backend = "setuptools.build_meta"
requires = [
    "setuptools == 61.2",
    "trove-classifiers",
    "wheel",
]

["tool.setuptools"]
py-modules = [
    "cs.gimmicks",
]

["tool.setuptools".package-dir.""]
"" = "lib/python"

I’ve been hand hacking those bottom 2 tables in various ways and running:

python3 -m build -o . --wheel

all to no avail. I either get the incorrect lib/python/cs/gimmicks.py file in the wheel, or I get this error complaint:

error: Multiple top-level packages discovered in a flat-layout: ['lib', 'tmp185du45n'].

To avoid accidental inclusion of unwanted files or directories,
setuptools will not proceed with this build.

If you are trying to create a single distribution with multiple packages
on purpose, you should not rely on automatic discovery.
Instead, consider the following options:

1. set up custom discovery (`find` directive with `include` or `exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names

To find more information, look for "package discovery" on setuptools docs.

Their package discover docs have not helped me.

Can someone explain what I could be putting here? Ideally not some autodiscovery config. (Indeed, their docs same the providing py-modules or packages turns off the autodiscovery, which is as I want it.)

2

Some addenda:

  • the setuptools == 61.2 is debugging, trying to see if this is some setuptools version related issue - I normally use >=61.2 and using the latest release does not change the behaviour anyway
  • you can see all the wheel URLs (and thus hand fetch them) by running: python3 -m pip install -f -v -v -v cs.gimmicks; fetching the corresponding sdist tarball gets you the pyproject.toml (and setup.* if present) used to make the wheel file
  • wheel files are zip files, so unzip -l foo.whl will list its contents
3

This doesn’t look right at all, as a matter of TOML syntax. Pretty sure it should just be [tool.setuptools.package-dir], or possibly [tool.setuptools."package-dir"].

1 Like
4

It looks wrong to me too, but when I change it to this:

["tool.setuptools"]
py-modules = [
    "cs.gimmicks",
]

["tool.setuptools".package-dir]
"" = "lib/python"

I get the Multiple top-level packages discovered in a flat-layout error described earlier.

This output is a TOML transcription emitted by this line of my release script:

    sys.stdout.write(tomli_w.dumps(pyproject, multiline_strings=True))

So it’s whatever tomli_w thinks is sane.

The tool.setuptools stuff starts like this:

    setuptools_cfg = {
        "package-dir": {
            "": package_dir
        },
    }

which is stitched into the overall config like this:

    setuptools_cfg = {
        "package-dir": {
            "": package_dir
        },
    }
    if self.is_package:
      setuptools_cfg["packages"] = [self.name]
    else:
      setuptools_cfg["py-modules"] = [self.name]
    pyproject = {
        "project": projspec,
        "build-system": {
            "build-backend":
            "setuptools.build_meta",
            "requires": [
                ##"setuptools >= 61.2",
                "setuptools == 61.2",
                "trove-classifiers",
                "wheel",
            ],
        },
        "tool.setuptools": setuptools_cfg,
    }

so I also don’t understand where that spurious-looking "" comes from. Left untouched I get the lib/python/cs/gimmicks.py path in the wheel :frowning:

5

This sounds like it automatically discovers the other packages in your monorepo (I’ve seen it before :wink: ) that are siblings of cs/gimmicks.py. What if you try also adding tool.setuptools.packages = []?

6

No change, same Multiple top-level packages discovered in a flat-layout complaint:

["tool.setuptools"]
py-modules = [
    "cs.gimmicks",
]
packages = []

It should like I need to convince it to use a “src-layout”. Which is what I thought the:

["tool.setuptools".package-dir.""]
"" = "lib/python"

was supposed to achieve.

I’ve downloaded the setuptools code, but it’s… complicated. (Not that I should complain, my release script is a complicated evolved-over-time thing.)

7

Hmm. Maybe it can be done using MANIFEST.in?

8

You do realize that ["tool.setuptools"] is strictly wrong, right? If setuptools behaves according to the spec, it will just ignore everything in there.

You dictionary needs to be {"tool": {"setuptools": ...}}, not {"tool.setuptools": ...}.

2 Likes
9

To follow up on this: Specifically, wrapping it in quotes makes it one table name rather than referring to a setuptools table nested within the tool table.

1 Like
10

Thank you, this hits my syntactic blind spot directly. Those quotes should have been ringing alarm bells for me instead of just making me uncomfortable.

Between a mistake in my default for package-dir (which is where the "" came from) and this error in my dict setup, fix:

"tool": {
     "setuptools": setuptools_cfg,
},

I’m now good!

My pyproject.toml now contains this:

[tool.setuptools]
py-modules = [
    "cs.gimmicks",
]

[tool.setuptools.package-dir]
"" = "lib/python"

Thank you all!

Now all I have to do is push updates for every recent package where I’ve mangled things like this… :frowning:

1 Like
11

I’ve got one of them (a bit ancient, needs revisiting) but I’ve since had my core mistake pointed out.

12

Yah, this is The Clue, which didn’t get through my brain. @MegaIng made this explicit to me.

13

For reference - notes on specifying package contents in the py TOML with setuptools:

https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#package-discovery-and-namespace-packages

If you’re using an src layout and want everything in src to be included, then surely all you would need to do is to write:

[tool.setuptools]
package-dir = {"" = "src"}
...
1 Like