By Skip Montanaro via Discussions on Python.org at 22Sep2022 23:30:
I’m an old school Make user, but I think I’ve exceeded its abilities
(well, not without a lot of extra work) to handle building, installing
and packaging some stuff. So, seeing all kinds of mentions of
pyproject.toml
, I went looking for details about using it. I came
across
this,
but that really only addresses one issue, creating wheels and tar files
to upload to PyPI.
That’s pretty much all that’s needed for getting stuff to PyPI (aside
from the upload itself, of course). The specifics depend on the tool
you’re using, and that tool is defined in the pyproject.toml
file as
well.
I would be interested (personally) to know if you read this:
https://packaging.python.org/en/latest/flow/
and if it was of much help. It’s a conceptual overview for packaging.
I’ve got a directory structure patterned after the one in the packaging tutorial:
mpl/
├── LICENSE
├── pyproject.toml
├── README.md
└── src/
└── csv-mpl-plot/
├── __init__.py
└── mpl.py
What’s in your pyproject.toml
file? That affects everything else,
including the contents of the upload files.
Running python -m build
does indeed build a wheel and a tar file and I can upload them to test.pypi.org
, but they don’t have quite the correct structure:
Do they install from PyPI using pip
? If that works, their structure
is probably “correct”, just not identify to your source tree.
% tar tvfz dist/csv_mpl_plot-1.0.0.tar.gz
-rw-r--r-- 0/0 2430 2020-02-01 18:00 csv_mpl_plot-1.0.0/Makefile
-rw-r--r-- 0/0 21886 2020-02-01 18:00 csv_mpl_plot-1.0.0/lib/mpl.py
-rw-r--r-- 0/0 0 2020-02-01 18:00 csv_mpl_plot-1.0.0/src/csv-mpl-plot/__init__.py
-rw-r--r-- 0/0 22129 2020-02-01 18:00 csv_mpl_plot-1.0.0/src/csv-mpl-plot/mpl.py
-rw-r--r-- 0/0 44 2020-02-01 18:00 csv_mpl_plot-1.0.0/.gitignore
-rw-r--r-- 0/0 1057 2020-02-01 18:00 csv_mpl_plot-1.0.0/LICENSE
-rw-r--r-- 0/0 760 2020-02-01 18:00 csv_mpl_plot-1.0.0/README.md
-rw-r--r-- 0/0 626 2020-02-01 18:00 csv_mpl_plot-1.0.0/pyproject.toml
-rw-r--r-- 0/0 2507 2020-02-01 18:00 csv_mpl_plot-1.0.0/PKG-INFO
All the necessary bits seem to be present.
Let’s look at one of my packages as a comparison. I’m not a packaging
expert, but I do upload to PyPI. Here’s what happens when I publish
cs.app.ydl
. I’m using setuptools
as the build backend, BTW, but
invoking it using python -m build
just as you do. Also, I’m still
using the setup.cfg
file for some of the configuration, though I
believe I can move entirely to pyproject.toml
with the more recent
setuptools
releases - I need to check.
First, my package tree after making the upload files:
CSS[~/hg/css/cs.app.ydl-20220318--2022-09-23T09:41:42.979984(hg:default)]fleet2*> L -R
-rw-rw-r-- 1 cameron cameron 18 23 Sep 09:41 MANIFEST.in
-rw-rw-r-- 1 cameron cameron 3685 23 Sep 09:41 README.md
drwxrwxr-x 4 cameron cameron 128 23 Sep 09:42 dist
drwxrwxr-x 3 cameron cameron 96 23 Sep 09:41 lib
-rw-rw-r-- 1 cameron cameron 4066 23 Sep 09:41 pyproject.toml
-rw-rw-r-- 1 cameron cameron 1075 23 Sep 09:41 setup.cfg
-rw-rw-r-- 1 cameron cameron 59 23 Sep 09:41 setup.py
./dist:
-rw-rw-r-- 1 cameron cameron 7102 23 Sep 09:42 cs.app.ydl-20220318-py3-none-any.whl
-rw-rw-r-- 1 cameron cameron 6943 23 Sep 09:42 cs.app.ydl-20220318.tar.gz
./lib/python/cs/app:
-rw-r--r-- 1 cameron cameron 11325 18 Mar 2022 ydl.py
./lib/python/cs.app.ydl.egg-info:
total 24
drwxrwxr-x 8 cameron cameron 256 23 Sep 09:42 .
drwxrwxr-x 4 cameron cameron 128 23 Sep 09:42 ..
-rw-rw-r-- 1 cameron cameron 3703 23 Sep 09:42 PKG-INFO
-rw-rw-r-- 1 cameron cameron 352 23 Sep 09:42 SOURCES.txt
-rw-rw-r-- 1 cameron cameron 1 23 Sep 09:42 dependency_links.txt
-rw-rw-r-- 1 cameron cameron 40 23 Sep 09:42 entry_points.txt
-rw-rw-r-- 1 cameron cameron 173 23 Sep 09:42 requires.txt
-rw-rw-r-- 1 cameron cameron 3 23 Sep 09:42 top_level.txt
Note: the build step made the files in dist/
and in
cs.app.ydl.egg-info/
, not me.
The upload files in dist/
above only need to contain the files needed
for install. For example, the cs.app.ydl-20220318-py3-none-any.whl
contains this:
Length Date Time Name
--------- ---------- ----- ----
11325 03-18-2022 01:08 cs/app/ydl.py
4035 09-22-2022 23:42 cs.app.ydl-20220318.dist-info/METADATA
92 09-22-2022 23:42 cs.app.ydl-20220318.dist-info/WHEEL
40 09-22-2022 23:42 cs.app.ydl-20220318.dist-info/entry_points.txt
3 09-22-2022 23:42 cs.app.ydl-20220318.dist-info/top_level.txt
493 09-22-2022 23:42 cs.app.ydl-20220318.dist-info/RECORD
which I gather gets unpacked directly into a site-packages tree. So the
.py
file and some metadata files.
The .tar.gz
is a source distribution and contains the source and
enough stuff for setuptools
to build and locally install in the end
user’s python environment. So it has quite a bit more stuff:
drwxrwxr-x 0 23 Sep 09:42 cs.app.ydl-20220318/
-rw-rw-r-- 18 23 Sep 09:41 cs.app.ydl-20220318/MANIFEST.in
-rw-rw-r-- 3703 23 Sep 09:42 cs.app.ydl-20220318/PKG-INFO
-rw-rw-r-- 3685 23 Sep 09:41 cs.app.ydl-20220318/README.md
drwxrwxr-x 0 23 Sep 09:42 cs.app.ydl-20220318/lib/
drwxrwxr-x 0 23 Sep 09:42 cs.app.ydl-20220318/lib/python/
drwxrwxr-x 0 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs/
drwxrwxr-x 0 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs/app/
-rw-r--r-- 11325 18 Mar 2022 cs.app.ydl-20220318/lib/python/cs/app/ydl.py
drwxrwxr-x 0 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/
-rw-rw-r-- 3703 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/PKG-INFO
-rw-rw-r-- 352 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/SOURCES.txt
-rw-rw-r-- 1 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/dependency_links.txt
-rw-rw-r-- 40 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/entry_points.txt
-rw-rw-r-- 173 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/requires.txt
-rw-rw-r-- 3 23 Sep 09:42 cs.app.ydl-20220318/lib/python/cs.app.ydl.egg-info/top_level.txt
-rw-rw-r-- 4066 23 Sep 09:41 cs.app.ydl-20220318/pyproject.toml
-rw-rw-r-- 1113 23 Sep 09:42 cs.app.ydl-20220318/setup.cfg
-rw-rw-r-- 59 23 Sep 09:41 cs.app.ydl-20220318/setup.py
The project tree itself Starting from the top, here’s the
pyproject.toml
:
[project]
name = "cs.app.ydl"
description = "Convenient command line and library wrapper for youtube-dl."
authors = [
{ name = "Cameron Simpson", email = "cs@cskk.id.au" },
]
keywords = [
"python3",
]
dependencies = [
"cs.cmdutils>=20220318",
"cs.excutils>=20210123",
"cs.fstags>=20220918",
"cs.logutils>=20220531",
"cs.pfx>=20220918",
"cs.progress>=20220918",
"cs.result>=20220918",
"cs.upd>=20200517",
"youtube_dl",
]
classifiers = [
"Development Status :: 4 - Beta",
"Environment :: Console",
"Operating System :: POSIX",
"Operating System :: Unix",
"Programming Language :: Python",
"Programming Language :: Python :: 3",
"Topic :: Internet",
"Topic :: System :: Networking",
"Topic :: Utilities",
"Intended Audience :: Developers",
"License :: OSI Approved :: GNU General Public License v3 or later
(GPLv3+)",
]
version = "20220318"
[project.license]
text = "GNU General Public License v3 or later (GPLv3+)"
[project.urls]
URL = "https://bitbucket.org/cameron_simpson/css/commits/all"
[project.readme]
text = """
.... omitted, it is quite long, ......
"""
content-type = "text/markdown"
[build-system]
requires = [
"setuptools >= 61.2",
"trove-classifiers",
"wheel",
]
build-backend = "setuptools.build_meta"
The [build-system]
says that I’m using setuptools
as the build
system (which is what python -m build
invokes).
For me, a lot of the package info is still in setup.cfg
(as I said, I
believe that these days I can probably move all of that into the
pyproject.toml
). So it says:
[metadata]
name = cs.app.ydl
version = 20220318
author = Cameron Simpson
author_email = cs@cskk.id.au
license = GNU General Public License v3 or later (GPLv3+)
description = Convenient command line and library wrapper for youtube-dl.
keywords = python3
url = https://bitbucket.org/cameron_simpson/css/commits/all
classifiers =
Development Status :: 4 - Beta
Environment :: Console
Operating System :: POSIX
Operating System :: Unix
Programming Language :: Python
Programming Language :: Python :: 3
Topic :: Internet
Topic :: System :: Networking
Topic :: Utilities
Intended Audience :: Developers
License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
long_description = file: README.md
long_description_content_type = text/markdown
[options]
package_dir =
= lib/python
install_requires =
cs.cmdutils>=20220318
cs.excutils>=20210123
cs.fstags>=20220918
cs.logutils>=20220531
cs.pfx>=20220918
cs.progress>=20220918
cs.result>=20220918
cs.upd>=20200517
youtube_dl
[options.entry_points]
console_scripts =
ydl = cs.app.ydl:main
You can see it says a lot of the stuff already present in
pyproject.toml
. One day, when I have the time…
Notice that the setup.cfg
says where the source files are in the
[options]
section and what the entry point is in the
[options.entry_points]
section.
Whatwever build tool you’re using will have similar information.
Cheers,
Cameron Simpson cs@cskk.id.au