I developed lockfile which I was recently informed is critical. It appears criticality depends on downloads. Is there some way to identify other PyPI packages which list lockfile as a dependency? Is that detail buried somewhere in the Big Query database? If so, is there a tutorial on getting started with that beast?
2 Likes
The answer is No, but also sort of, yes?
The fundamental problem you’ll run into here is that for a lot of historical reasons PyPI doesn’t have reliable metadata on what dependencies a package has, the problems with this data stem from a few things:
- Historically this information wasn’t tracked at all (i.e. distutils didn’t have this information at all until setuptools added it)
- PyPI doesn’t introspect the files to pull metadata out [1], but rather expects the tool doing the uploading to send all the metadata alongside the file, and older tools didn’t support the dependency metadata even after it was added to our standards (modern tools do though).
- While Wheels have always had static metadata, the metadata inside of a sdist historically could not be trusted, because authors could do anything they wanted inside of their
setup.pyso you’d possibly get different dependencies depending on the context that you were building that sdist in (common for say different deps on windows vs linux). - PyPI’s data model for dependencies is wrong, it attaches dependency information directly to the Version, but every file on PyPI is able to have it’s own unique set of dependencies, so in practice when PyPI does know of the dependencies, it’s the dependencies of the first file to be uploaded of that version.
All that being said, there is some data available, there’s just a lot of caveats to it. For what I think you want here (I assume you’re tracking down packages using lockfile for either your own curiosity or to ask them to switch off of it) you don’t need perfectly accurate metadata, so that should be fine.
As @pradyunsg pointed out, libraries.io has a pretty reasonable set of data available for easy access which I think pulls data from several sources.
If you want to ask PyPI specifically what it knows about dependencies, you can use the legacy json api, but you’ll have to iterate over the entire package space of PyPI (this won’t bother PyPI at all, but it won’t be a super fast thing since you’ll be making a ton of HTTP requests).
If you want the most accurate information, then your only real choices currently are to download the wheels themselves and inspect the METADATA file inside of them, and for sdists there’s no real way to generically get accurate information across every sdist, so you’re kinda SOL there no matter what.
A lot of work has gone, and is going, into improving this situation though! So hopefully in the future this will be a lot easier.
-
We want to make it do that at some point, but it doesn’t right now. ↩︎
Similarly, is there a way to see what built the wheels e.g. the Generator in WHEEL files?
1 Like
For that the only current option is to inspect the wheel file itself.
How about Wheelodex? Does Wheelodex — lockfile — Reverse Dependencies give you the information you want?
I’ve found libraries.io to be a bit buggy and missing some data.
1 Like
Libraries.io show the number of dependant packages (171) and repos (2.71k) but only seems to have a webpage to see the packages (for example and not repos). They have an API where you can access both lists, and as it happens I wrote a quick script to use it on Saturday.
Another option is to download @sethmlarson’s database of “Mostly up-to-date data about almost every package on PyPI” and:
$ sqlite3 'pypi.db' 'SELECT name FROM deps WHERE dep_name LIKE "lockfile" GROUP BY name;' | wc -l
97
3 Likes
Any idea where these services are pulling their data from? Flask and the other Pallets projects have been retaining a setup.py file only containing name and install_requires because GitHub’s dependency graph only knows how to read that. However, I’m thinking of dropping support for GitHub’s dependency graph and use a newer management tool like PDM. Knowing that there are other graphs out there helps me make that decision.
Not true, GitHub supports pyproject.toml (Poetry format only I think), Pipfile and Pipfile.lock, poetry.lock, and requirements.txt as well. It doesn’t support setup.cfg.
Guess they’ve added more since I last checked, but I was considering using PDM, so that will still be unsupported. Weird that they specifically support poetry, not the standard metadata.
GitHub supported pyproject.toml for dependencies when Poetry was the only one using it, and Poetry have their own format, before PEP 621 (storing project metadata) was accepted. I wonder if GitHub even knows about PEP 621, being the canonical way to declare dependencies.
Nope Standard Python support · Issue #3290 · dependabot/dependabot-core · GitHub
That would be a good fundable task imo
An empty setup.py would be enough for dependabot to get dependants(who depends on me?). You can use whatever metadata you like. But it can’t know what the dependencies are.
Did you add it to the fundables repo?
No I didn’t
I don’t think funding work on dependabot is a good fit for that list – it’s feature development for cross-language tooling that backs a commercial product. AFAIK, we haven’t added things to the fundables repository where there’s no concensus among the project maintainers that this is a good improvement, or when the packages being updated aren’t in the Python space, or aren’t primarily volunteer maintained.
That said, I don’t wear any hats that give me authority over what goes in that list so… take this with a grain of salt. 
(updates to fix typos, that’s what I get for going on d.p.o first thing in the morning)
1 Like