Backends declared in the pyproject.toml files during 2018 to 2023 on PyPI

fcodvpt · December 8, 2023, 2:51pm

Hi,

I made some code to extract the backends declared in the pyproject.toml files during 2018 to 2023 (1)

I used @sethmlarson great article : “Querying every file in every release on the Python Package Index” (2)

I called the gist (3) to dowload the files and everything worked fine.

Only the root pyproject.toml is analyzed.

Here is the bar chart with all “build-backend” found (log scale used) :

Here is the bar chart with the “build-backend” declared more than 5 times (log scale) :

I was just curious about it and hope you will find this useful.

Regards
Françoise

(1) https://gitlab.liris.cnrs.fr/fconil-small-programs/packaging/get-pypi-packages-backends
(2) Querying every file in every release on the Python Package Index
(3) Simple module for querying data from py-code.org · GitHub

mgorny · December 8, 2023, 5:46pm

Do I understand correctly that the counts are per sdist archive and not per project? i.e. projects that had more releases count more towards the total than projects that had fewer releases?

ofek · December 8, 2023, 6:33pm

That is my understanding as well i.e. there is no way 8K projects are using Hatchling since it is so new. Unless, of course, I don’t have a firm grasp of how many projects are actually on PyPI.

jeanas · December 8, 2023, 10:11pm

https://github.com/search?q=hatchling.build%20path%3Apyproject.toml&type=code

10.4k results (note that GitHub code search excludes forks by default). Now, of course, not all of these will be published to PyPI.

cemici · December 8, 2023, 11:38pm

This doesn’t seem to be true, but you can add “NOT is:fork” to the search

fcodvpt · December 11, 2023, 4:42pm

I used the dataset : Metadata about every file uploaded to PyPI.

This dataset contains information about every file within every release uploaded to PyPi

But I tried to get only the latest release uploaded (with the uploaded_on metadata). I fetched only the pyproject.toml files.

Some projects contains several pyproject.toml files (often in tests folders).

+--------------+-----------------+-------------------------+------------------------------------------------------------------------------------------------------------+
| project_name | project_version |       uploaded_on       |                                                    path                                                    |
+--------------+-----------------+-------------------------+------------------------------------------------------------------------------------------------------------+
| pyink        | 23.10.0         | 2023-10-24 17:24:34.499 | packages/pyink/pyink-23.10.0.tar.gz/pyink-23.10.0/pyproject.toml                                           |
| pyink        | 23.10.0         | 2023-10-24 17:24:34.499 | packages/pyink/pyink-23.10.0.tar.gz/pyink-23.10.0/tests/data/empty_pyproject.toml                          |
| pyink        | 23.10.0         | 2023-10-24 17:24:34.499 | packages/pyink/pyink-23.10.0.tar.gz/pyink-23.10.0/tests/data/include_exclude_tests/pyproject.toml          |
| pyink        | 23.10.0         | 2023-10-24 17:24:34.499 | packages/pyink/pyink-23.10.0.tar.gz/pyink-23.10.0/tests/data/invalid_gitignore_tests/pyproject.toml        |
| pyink        | 23.10.0         | 2023-10-24 17:24:34.499 | packages/pyink/pyink-23.10.0.tar.gz/pyink-23.10.0/tests/data/invalid_nested_gitignore_tests/pyproject.toml |
| pyink        | 23.10.0         | 2023-10-24 17:24:34.499 | packages/pyink/pyink-23.10.0.tar.gz/pyink-23.10.0/tests/data/nested_gitignore_tests/pyproject.toml         |
| pyink        | 23.10.0         | 2023-10-24 17:24:34.499 | packages/pyink/pyink-23.10.0.tar.gz/pyink-23.10.0/tests/data/project_metadata/both_pyproject.toml          |
| pyink        | 23.10.0         | 2023-10-24 17:24:34.499 | packages/pyink/pyink-23.10.0.tar.gz/pyink-23.10.0/tests/data/project_metadata/neither_pyproject.toml       |
| pyink        | 23.10.0         | 2023-10-24 17:24:34.499 | packages/pyink/pyink-23.10.0.tar.gz/pyink-23.10.0/tests/data/project_metadata/only_black_pyproject.toml    |
| pyink        | 23.10.0         | 2023-10-24 17:24:34.499 | packages/pyink/pyink-23.10.0.tar.gz/pyink-23.10.0/tests/data/project_metadata/only_metadata_pyproject.toml |
+--------------+-----------------+-------------------------+------------------------------------------------------------------------------------------------------------+

I tried to keep only the pyproject.toml files at the root of the project.

You can now access the files (csv or SQLite) I produced for these plots : https://dataset-dl.liris.cnrs.fr/pypi-projects-backends/