Since PEP 665 is moving forward with the decision to only support wheels, I am creating a new proposal that’s separate to, but depending on PEP 665 to provide sdist and source tree support, as this is evidently desired from the discussion in Supporting sdists and source trees in PEP 665.
Speaking as an author of PEP 665, my personal motivation to limit the PEP’s scope to wheels is mainly from my impression, based on discussions around the PEP, that a lock without strict reproducibility guarantees is unlikely to be accepted by the community, due to their inherent nature of not actually providing locked-down dependencies, based on “locking down” being defined to mean “guaranteeing reproducibility”. This, however, does leave a use case unsatisfied. In certain workflow, users do not want to actually lock their dependencies down to strict reproducibility in a file. Instead, they assume the file’s consumer is able to somehow maintain reproducibility.
Where PEP 665 produces this flow:
┌──────┐ ┌─────────────┐
│ file ├──►│ environment │
└──────┘ └─────────────┘
so in order for the environment to be reproducible, the file must fully describe reproducibility.
While this alternative workflow described in the previous paragraph produces this:
external ┌─────────────┐
inputs ──────►│ │
│ environment │
┌──────┐ ┌──────►│ │
│ file ├─┘ └─────────────┘
└──────┘
and wants a file that, assuming external inputs are reproducible, guarantees reproducibility, while considering this reproducibility of external inputs out of scope.
This leads me into two natural conclusions:
- Like how we don’t cram sdist and wheel into one single format, the two flows need two different formats.
- Like how we defined wheels first and (slowly) apply usable parts of wheels to sdists, the file format for the latter flow can and should reuse syntaxes defined in PEP 665, while being kept semantically distinct.
In other words, PEP 665 is analogous to wheel for application dependency specification, and we now need an sdist equivalent for the same purpose.
So I’m proposing a new format that will
- Have a distinct filename suffix to PEP 665.
- Reuse all PEP 665 syntaxes and semantics where possible.
- Give certain reused fields additional or alternative semantics to support sdist and source tree.
Top-level fields; [tool]
and [metadata]
sections
These all have exactly the same syntaxes and semantics as proposed in PEP 665.
[package._name_._version_]
sections
The section name scheme uses exactly the same syntaxes and semantics as PEP 665.
Fields direct
, requires-python
, and requires
use exactly the same syntaxes and semantics as PEP 665. This also implies that a package must either define metadata statically, or is able to generate consistent metadata dynamically in a reproducible build requirement.
The url
field, in addition to a wheel file, may specify a URL to an sdist or source tree. As in PEP 665, there are no special restrictions on the URL format, and the installer is free to not use the URL for actual artifact retrieval.
The following additional rules are introduced:
- If
url
specifies a file…-
filename
must be the name to use for the downloaded file. -
hashes
must contain at least one hash for the file.
-
- If
url
specifies a VCS location…-
filename
must be the name to download the directory into. -
hashes
must contain a field namedcommit_id
, with the value being the fully resolved revision identifier, as defined in PEP 610.
-
- If
url
specifies a (non-VCS) directory…-
filename
must be an empty string. -
hashes
must contain a field namededitable
, with the value being a boolean.
-
VCS example
[package.packaging."20.3"]
url = "git+https://github.com/pypa/packaging.git@20.3"
filename = "packaging"
hashes.commit_id = "b10b90a7d1c3afac6852462fa2548db231810adf"
specifies cloning the repository into a directory named packaging
, and the clone must point to commit b10b90a7d1c3afac6852462fa2548db231810adf
.
Installation
The next natural conclusion I drew is that, like how we install an sdist by turning it into a wheel first, this new format should also be installed by being turned into a PEP 665 lock file (plus additional information), so we can reuse how a PEP 665 lock file can be installed into an environment.
All source trees and sdists included in the new format must be able to be locally built into a wheel with PEPs 517 or 610, or the legacy bdist_wheel
. We explicitly exclude support for things requiring setup.py install
and setup.py develop
. This means, I believe, we can turn all specifications of this new format into a combination of zero or more wheels built from non-wheel specifications in the file (we might need to be able to formally prove this? I think it’s possible), and one PEP 665 lock file that references those built wheels. Installation is therefore achieved by installing this converted PEP 665 lock file.
Since this new format will have a different, distinct filename scheme to PEP 665, such a file can always be distinctively identified by tools. This means that an installer, if it wishes to, can seamlessly take either file formats, and perform the necessary conversion behind the scenes, like how pip can install either source or wheel with the same PEP 508 format.
Proposed format name and filename suffix
I’m proposing the name pin file and the .pypin.toml
suffix, since this kind of dependency specifying is commonly called “pinning versions”. Alternative proposals are welcomed.
This covers everything, I think? I’d be very interested in hearing thoughts about this.