Good packaging/setuptools/whatever tutorial?

I’m an old school Make user, but I think I’ve exceeded its abilities (well, not without a lot of extra work) to handle building, installing and packaging some stuff. So, seeing all kinds of mentions of pyproject.toml, I went looking for details about using it. I came across this, but that really only addresses one issue, creating wheels and tar files to upload to PyPI.

I’ve got a directory structure patterned after the one in the packaging tutorial:

mpl/
├── LICENSE
├── pyproject.toml
├── README.md
└── src/
    └── csv-mpl-plot/
        ├── __init__.py
        └── mpl.py

Running python -m build does indeed build a wheel and a tar file and I can upload them to test.pypi.org, but they don’t have quite the correct structure:

% tar tvfz dist/csv_mpl_plot-1.0.0.tar.gz 
-rw-r--r-- 0/0            2430 2020-02-01 18:00 csv_mpl_plot-1.0.0/Makefile
-rw-r--r-- 0/0           21886 2020-02-01 18:00 csv_mpl_plot-1.0.0/lib/mpl.py
-rw-r--r-- 0/0               0 2020-02-01 18:00 csv_mpl_plot-1.0.0/src/csv-mpl-plot/__init__.py
-rw-r--r-- 0/0           22129 2020-02-01 18:00 csv_mpl_plot-1.0.0/src/csv-mpl-plot/mpl.py
-rw-r--r-- 0/0              44 2020-02-01 18:00 csv_mpl_plot-1.0.0/.gitignore
-rw-r--r-- 0/0            1057 2020-02-01 18:00 csv_mpl_plot-1.0.0/LICENSE
-rw-r--r-- 0/0             760 2020-02-01 18:00 csv_mpl_plot-1.0.0/README.md
-rw-r--r-- 0/0             626 2020-02-01 18:00 csv_mpl_plot-1.0.0/pyproject.toml
-rw-r--r-- 0/0            2507 2020-02-01 18:00 csv_mpl_plot-1.0.0/PKG-INFO

The src directory is replicated in the distribution (is that right?), the Makefile from the current directory is included, as is (for some strange reason) the .gitignore file from the parent directory.

mpl.py is meant to be both importable and runnable as a command line tool. When I pip install the package, I get the csv_mpl_plot.mpl module, but no mpl command line tool. That’s not surprising, as I saw nothing in `pyproject.toml which looked like it was supposed to do that. I looked around, but don’t see where/how to specify command line entry points. I am clearly missing an entire chunk of the puzzle, but don’t know what. Pointers to tutorials for that side of things would be appreciated.

1 Like

Can you post your pyproject.toml? That would help us to be able to answer your questions. Some of the answers will be different based on which build backend (hatchling, setuptools, etc.) you chose.

By Skip Montanaro via Discussions on Python.org at 22Sep2022 23:30:

I’m an old school Make user, but I think I’ve exceeded its abilities
(well, not without a lot of extra work) to handle building, installing
and packaging some stuff. So, seeing all kinds of mentions of
pyproject.toml, I went looking for details about using it. I came
across
this,
but that really only addresses one issue, creating wheels and tar files
to upload to PyPI.

That’s pretty much all that’s needed for getting stuff to PyPI (aside
from the upload itself, of course). The specifics depend on the tool
you’re using, and that tool is defined in the pyproject.toml file as
well.

I would be interested (personally) to know if you read this:
https://packaging.python.org/en/latest/flow/
and if it was of much help. It’s a conceptual overview for packaging.

I’ve got a directory structure patterned after the one in the packaging tutorial:

mpl/
├── LICENSE
├── pyproject.toml
├── README.md
└── src/
   └── csv-mpl-plot/
       ├── __init__.py
       └── mpl.py

What’s in your pyproject.toml file? That affects everything else,
including the contents of the upload files.

Running python -m build does indeed build a wheel and a tar file and I can upload them to test.pypi.org, but they don’t have quite the correct structure:

Do they install from PyPI using pip? If that works, their structure
is probably “correct”, just not identify to your source tree.

% tar tvfz dist/csv_mpl_plot-1.0.0.tar.gz
-rw-r--r-- 0/0            2430 2020-02-01 18:00 csv_mpl_plot-1.0.0/Makefile
-rw-r--r-- 0/0           21886 2020-02-01 18:00 csv_mpl_plot-1.0.0/lib/mpl.py
-rw-r--r-- 0/0               0 2020-02-01 18:00 csv_mpl_plot-1.0.0/src/csv-mpl-plot/__init__.py
-rw-r--r-- 0/0           22129 2020-02-01 18:00 csv_mpl_plot-1.0.0/src/csv-mpl-plot/mpl.py
-rw-r--r-- 0/0              44 2020-02-01 18:00 csv_mpl_plot-1.0.0/.gitignore
-rw-r--r-- 0/0            1057 2020-02-01 18:00 csv_mpl_plot-1.0.0/LICENSE
-rw-r--r-- 0/0             760 2020-02-01 18:00 csv_mpl_plot-1.0.0/README.md
-rw-r--r-- 0/0             626 2020-02-01 18:00 csv_mpl_plot-1.0.0/pyproject.toml
-rw-r--r-- 0/0            2507 2020-02-01 18:00 csv_mpl_plot-1.0.0/PKG-INFO

All the necessary bits seem to be present.

Let’s look at one of my packages as a comparison. I’m not a packaging
expert, but I do upload to PyPI. Here’s what happens when I publish
cs.app.ydl. I’m using setuptools as the build backend, BTW, but
invoking it using python -m build just as you do. Also, I’m still
using the setup.cfg file for some of the configuration, though I
believe I can move entirely to pyproject.toml with the more recent
setuptools releases - I need to check.

First, my package tree after making the upload files:

 CSS[~/hg/css/cs.app.ydl-20220318--2022-09-23T09:41:42.979984(hg:default)]fleet2*> L -R
 -rw-rw-r--    1 cameron  cameron    18 23 Sep 09:41 MANIFEST.in
 -rw-rw-r--    1 cameron  cameron  3685 23 Sep 09:41 README.md
 drwxrwxr-x    4 cameron  cameron   128 23 Sep 09:42 dist
 drwxrwxr-x    3 cameron  cameron    96 23 Sep 09:41 lib
 -rw-rw-r--    1 cameron  cameron  4066 23 Sep 09:41 pyproject.toml
 -rw-rw-r--    1 cameron  cameron  1075 23 Sep 09:41 setup.cfg
 -rw-rw-r--    1 cameron  cameron    59 23 Sep 09:41 setup.py

 ./dist:
 -rw-rw-r--  1 cameron  cameron  7102 23 Sep 09:42 cs.app.ydl-20220318-py3-none-any.whl
 -rw-rw-r--  1 cameron  cameron  6943 23 Sep 09:42 cs.app.ydl-20220318.tar.gz

 ./lib/python/cs/app:
 -rw-r--r--  1 cameron  cameron  11325 18 Mar  2022 ydl.py

 ./lib/python/cs.app.ydl.egg-info:
 total 24
 drwxrwxr-x  8 cameron  cameron   256 23 Sep 09:42 .
 drwxrwxr-x  4 cameron  cameron   128 23 Sep 09:42 ..
 -rw-rw-r--  1 cameron  cameron  3703 23 Sep 09:42 PKG-INFO
 -rw-rw-r--  1 cameron  cameron   352 23 Sep 09:42 SOURCES.txt
 -rw-rw-r--  1 cameron  cameron     1 23 Sep 09:42 dependency_links.txt
 -rw-rw-r--  1 cameron  cameron    40 23 Sep 09:42 entry_points.txt
 -rw-rw-r--  1 cameron  cameron   173 23 Sep 09:42 requires.txt
 -rw-rw-r--  1 cameron  cameron     3 23 Sep 09:42 top_level.txt

Note: the build step made the files in dist/ and in
cs.app.ydl.egg-info/, not me.

The upload files in dist/ above only need to contain the files needed
for install. For example, the cs.app.ydl-20220318-py3-none-any.whl
contains this:

   Length      Date    Time    Name
 ---------  ---------- -----   ----
     11325  03-18-2022 01:08   cs/app/ydl.py
      4035  09-22-2022 23:42   cs.app.ydl-20220318.dist-info/METADATA
        92  09-22-2022 23:42   cs.app.ydl-20220318.dist-info/WHEEL
        40  09-22-2022 23:42   cs.app.ydl-20220318.dist-info/entry_points.txt
         3  09-22-2022 23:42   cs.app.ydl-20220318.dist-info/top_level.txt
       493  09-22-2022 23:42   cs.app.ydl-20220318.dist-info/RECORD

which I gather gets unpacked directly into a site-packages tree. So the
.py file and some metadata files.

The .tar.gz is a source distribution and contains the source and
enough stuff for setuptools to build and locally install in the end
user’s python environment. So it has quite a bit more stuff:

 drwxrwxr-x       0 23 Sep 09:42 cs.app.ydl-20220318/
 -rw-rw-r--      18 23 Sep 09:41 cs.app.ydl-20220318/MANIFEST.in
 -rw-rw-r--    3703 23 Sep 09:42 cs.app.ydl-20220318/PKG-INFO
 -rw-rw-r--    3685 23 Sep 09:41 cs.app.ydl-20220318/README.md
 drwxrwxr-x       0 23 Sep 09:42 cs.app.ydl-20220318/lib/
 drwxrwxr-x       0 23 Sep 09:42 cs.app.ydl-20220318/lib/python/
 drwxrwxr-x       0 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs/
 drwxrwxr-x       0 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs/app/
 -rw-r--r--   11325 18 Mar  2022 cs.app.ydl-20220318/lib/python/cs/app/ydl.py
 drwxrwxr-x       0 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/
 -rw-rw-r--    3703 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/PKG-INFO
 -rw-rw-r--     352 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/SOURCES.txt
 -rw-rw-r--       1 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/dependency_links.txt
 -rw-rw-r--      40 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/entry_points.txt
 -rw-rw-r--     173 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/requires.txt
 -rw-rw-r--       3 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/top_level.txt
 -rw-rw-r--    4066 23 Sep 09:41 cs.app.ydl-20220318/pyproject.toml
 -rw-rw-r--    1113 23 Sep 09:42 cs.app.ydl-20220318/setup.cfg
 -rw-rw-r--      59 23 Sep 09:41 cs.app.ydl-20220318/setup.py

The project tree itself Starting from the top, here’s the
pyproject.toml:

 [project]
 name = "cs.app.ydl"
 description = "Convenient command line and library wrapper for youtube-dl."
 authors = [
     { name = "Cameron Simpson", email = "cs@cskk.id.au" },
 ]
 keywords = [
     "python3",
 ]
 dependencies = [
     "cs.cmdutils>=20220318",
     "cs.excutils>=20210123",
     "cs.fstags>=20220918",
     "cs.logutils>=20220531",
     "cs.pfx>=20220918",
     "cs.progress>=20220918",
     "cs.result>=20220918",
     "cs.upd>=20200517",
     "youtube_dl",
 ]
 classifiers = [
     "Development Status :: 4 - Beta",
     "Environment :: Console",
     "Operating System :: POSIX",
     "Operating System :: Unix",
     "Programming Language :: Python",
     "Programming Language :: Python :: 3",
     "Topic :: Internet",
     "Topic :: System :: Networking",
     "Topic :: Utilities",
     "Intended Audience :: Developers",
     "License :: OSI Approved :: GNU General Public License v3 or later 
 (GPLv3+)",
 ]
 version = "20220318"

 [project.license]
 text = "GNU General Public License v3 or later (GPLv3+)"

 [project.urls]
 URL = "https://bitbucket.org/cameron_simpson/css/commits/all"

 [project.readme]
 text = """
 .... omitted, it is quite long, ......
 """
 content-type = "text/markdown"

 [build-system]
 requires = [
     "setuptools >= 61.2",
     "trove-classifiers",
     "wheel",
 ]
 build-backend = "setuptools.build_meta"

The [build-system] says that I’m using setuptools as the build
system (which is what python -m build invokes).

For me, a lot of the package info is still in setup.cfg (as I said, I
believe that these days I can probably move all of that into the
pyproject.toml). So it says:

 [metadata]
 name = cs.app.ydl
 version = 20220318
 author = Cameron Simpson
 author_email = cs@cskk.id.au
 license = GNU General Public License v3 or later (GPLv3+)
 description = Convenient command line and library wrapper for youtube-dl.
 keywords = python3
 url = https://bitbucket.org/cameron_simpson/css/commits/all
 classifiers =
     Development Status :: 4 - Beta
     Environment :: Console
     Operating System :: POSIX
     Operating System :: Unix
     Programming Language :: Python
     Programming Language :: Python :: 3
     Topic :: Internet
     Topic :: System :: Networking
     Topic :: Utilities
     Intended Audience :: Developers
     License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
 long_description = file: README.md
 long_description_content_type = text/markdown

 [options]
 package_dir =
  = lib/python
 install_requires =
     cs.cmdutils>=20220318
     cs.excutils>=20210123
     cs.fstags>=20220918
     cs.logutils>=20220531
     cs.pfx>=20220918
     cs.progress>=20220918
     cs.result>=20220918
     cs.upd>=20200517
     youtube_dl

 [options.entry_points]
 console_scripts =
     ydl = cs.app.ydl:main

You can see it says a lot of the stuff already present in
pyproject.toml. One day, when I have the time…

Notice that the setup.cfg says where the source files are in the
[options] section and what the entry point is in the
[options.entry_points] section.

Whatwever build tool you’re using will have similar information.

Cheers,
Cameron Simpson cs@cskk.id.au

Sure. Didn’t think to as it’s basically the same as the packaging tutorial I referenced.

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "csv-mpl-plot"
version = "0.0.1"
authors = [
  { name="Skip Montanaro", email="skip.montanaro@gmail.com" },
]
description = "Matplotlib-based CSV file plotter"
readme = "README.md"
license = { file="LICENSE" }
requires-python = ">=3.10"
classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
]

[project.urls]
"Homepage" = "https://github.com/smontanaro/csvprogs/tree/main/mpl"
"Bug Tracker" = "https://github.com/smontanaro/csvprogs/issues"

Thanks, Cameron. It would appear that I’m mostly missing these bits:

This was all stuff I was doing with Make. I’ve never really used any of the Python packaging stuff before. I used Hatch because that’s what the packaging tutorial said was the default, but I have no way to judge the feature sets of the various alternatives. I’ll give these sections a try and see where I get.

Aside: Is there a reference document which says (in effect), “Here are all the valid pyproject.toml sections?” Maybe per-build-system?

OK, so you are using hatchling, not setuptools, for building the distribution. That’s fine, but it means that referring to setuptools docs won’t help you :slight_smile:

Hatch/hatchling defaults to including everything that is under version control into the distribution, so that the distribution can be used to build a wheel itself (which isn’t always possible otherwise). It can be configured to restrict the distribution contents, with examples in the documentation here.

It’s not possible to document all content that can appear in pyproject.toml, because there are dozens of tools that know how to use that file for configuration. Instead, as you indicated, if you’re looking to control the behavior of the build backend you are using, you need to consult its documentation which I linked above.

Thanks… Yes, as I indicated, the packaging tutorial gave hatchling as the default, so I just pasted those bits in. It gave me no indication how to choose between hatchling, setuptools, flit and pdm, so I dropped in hatchling.

This is probably a basic hole in my understanding about how everything is supposed to work, but how is one supposed to decide which of these build systems to use?

This page isn’t helping me:

https://packaging.python.org/en/latest/key_projects/

Just sayin’…

I did some searching, trying to find actual feature/ease-of-use comparisons of the various options. No luck so far.

Ahh… well, that’s hard to answer! Hatchling is fairly new, setuptools has been around for a long time, is mature, and there is more community knowledge about it. I don’t have any knowledge about Flit.

I’ve never seen any sort of comparisons between them, unfortunately.

The page you linked to does give you the option of switching between build backends; in the “Creating pyproject.toml” section there are tabs for them.

Yes, but unless I misread something it said Hatchling was the default. If figured, “who am I to argue?” :slight_smile:

It does use Hatchling by default, that’s true.

That is fine in the sdist, bad only if that is also included in the wheel.

The term you need to know here is “entry points”.
Hatch supports standard PEP 621 metadata, so you can look in packaging specifications docs on how to specify it Declaring project metadata — Python Packaging User Guide.
And it is mentioned in the hatch docs too Metadata - Hatch.

I don’t know if there is any good comparison article anywhere,you can just try each and see. The biggest difference between them for pure python projects is which files they include in the sdist and wheel by default, and how you have to configure them if you are not happy with the default way. The key projects page gives links to docs for all of them Project Summaries — Python Packaging User Guide

If you don’t want to do all that then the default choice of hatchling is good if you just want a standards compliant backend that is really small in size. Setuptools is also good choice if you want lots of choices and flexibility, or if you want to build C extensions.

1 Like

By Skip Montanaro via Discussions on Python.org at 23Sep2022 11:07:

Aside: Is there a reference document which says (in effect), “Here are
all the valid pyproject.toml sections?” Maybe per-build-system?

The pyproject.toml requires a [build-system] clause/table to
specify the build system you’re using. It tells python -m build (or
any PEP517 complaint command) how to invoke the build tool’s “build
backend” to make the upload files.

Beyond that, technically a tool can get its info anywhwre; all it has to
do is make the upload files (sdist and wheel) and maybe upload them for
you. The upload files contain the metadata.

Ideally, (if that’s a good choice of word), you could put everything
required into the pyproject.toml:

  • the [project] section is for Core Metadata
  • the [tool] section is for tool specific stuff (if the tool looks for
    it)

Pick a tool, then see what its docs ask you to do. I was using
setuptools in the past, so that’s what I went with, and I still need
to revisit it to see if I can move more config from its setup.cfg into
the pyproject.toml.

If I was trying to pick a tool from scratch I might try to pick one
which matches my workflow. I do not thing there is a “default”.

I’d choose the simplest tool available, given that your setup is pretty
simple.

Cheers,
Cameron Simpson cs@cskk.id.au

Thanks all. I’m going to be out of town and distracted this weekend, but will take my laptop and mess around. I’ll see if I can figure more out before pestering y’all more.

You can! setuptools, in its latest versions, can be completely configured in pyproject.toml.

1 Like

The changes required turned out to be modest. (I chose this particular application because it was very small, just one file, no test cases, etc.) I switched to setuptools as the build system and added a [project.scripts] section. Aside from the unrelated stuff (renames, adding required packages), there was nothing else to do.

Color me “happy camper blue”. Thanks for the help.

1 Like

FYI, I just saw this go by in my twitter timeline:
https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html

FYI, I just saw this go by in my twitter timeline:
https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html

Thanks, yes, I started working my way through that…

Skip

I’m biased, but I like Simple Python packaging - Scikit-HEP :slight_smile:

Hatchling (and Flit) are much simpler than Setuptools (Flit’s around 2-4K LoC while setuptools is 40K+). It’s also faster. The usage of PEP 621 is more mature in both Hatchling and Flit than setuptools, and they aren’t affected by the distutils removal and other historical issues. Error messages tend to be good on Hatchling; setuptools has so many layers that it’s a bit more painful to parse. Hatchling uses .gitignore as the source of truth for including files in the SDist, which is really nice, IMO - Flit is terrible at this and setuptools tends to miss files unless configured (typing related files for example). Not to say any of the four are bad choices. Setuptools has has some fantastic work done on it recently. If you need to add compiled extensions, setuptools is still probably the only way to go (but I’m working on changing that!).

The tutorial intentionally doesn’t have any details about the backend, it’s not supposed to matter for the purposes of the tutorial - all four given will work with the tutorial. There’s also not an example of a command line app - that would complicate it but it’s also often useful. I think the issue here is there’s not a good place to go for “what next”.