Thanks for all the work on this Brett. I’ve reviewed the PEP and this discussion thread and I think this has real potential. I have some thoughts/concerns that might be misguided, I’ll try to keep them short:
I share this concern with Seth and others in this thread – I’m still struggling a bit to convince myself that I understand the difference on first read. Is it fair to say that “package locking” is just “file locking”, but inclusive of every possible environment, like wheel-tags = [*]? Would that be a better way to explain it to regular users? If so, I think I agree with @frostming that the two can probably be unified.
This makes sense to me.
This PEP has no opinion as to the location of lock files (i.e. in the root or the subdirectory of a project).
I’m concerned that we’re losing the ability to reliably predict where a lockfile/requirements file is for a given source checkout. The requirements.txt format has the same lack of opinion, but also has the ability to link to other requirements files with -r. Some examples of why this is useful:
a cloud provider can say “you must have a requirements.txt in the root”, but that file can just use -r ./subdirectory/requirements.txt to link to requirements files in any subdirectory.
PyPI’s source repository uses a single requirements.txt file in the root which combines all our other requirements files, so we can keep them separate, but Dependabot can find them all in subdirectories and keep them up to date.
I’m not sure what the solution would be: some equivalent of -r that allows an installer to combine multiple lock files? I’m hesitant to propose this.
Installers MAY support installation of non-binary files (i.e. source distributions, source trees, and VCS), but are not required to.
It’s not clear to me from the PEP how these things would be included in this lockfile format. Specifically, I don’t see how file locking and specifically file-locks.wheel-tags would support installing anything that isn’t a wheel? I feel as if I may be missing something.
I think that’s the correct interpretation. I would like to say though that for many regular users, they don’t really understand what a wheel tag or environment marker is. I think most in these channels who are deeply familiar with packaging are of course very knowledgeable in them. Most regular users can probably get the gist of them when they see them, but I really think a lot of newer python developers just “pip install” or “poetry add” or “rye add” etc.
So even if we think it’s simpler to explain it in terms of tags or markers, it still might be better to explain it in terms of outcomes than technical details.
File lock: for installation into a fixed set of supported environments. Since environments have previously been checked, installation into the same environments should succeed.
Package lock: for installation into all possible environments. The installation environments have not been checked, so there could be some environments where installation fails.
To me, it’s actually very user-centric to let users make this trade off explicitly. The alternative seems like we take a decision away from users and let them down later when installation into an environment fails.
Maybe less realistic, but it also might be a good way for some open source projects to indicate what environments they are willing to support. Contributions from unsupported environments can still happen etc, but its explicit when they’re in a currently unsupported environment.
Would it be a helpful simplification for the package lock to be “the lock” and the file lock to be “the set of file (name/hashes) that are allowed to be installed for each package”? So an installer would follow the package lock and then only select files listed in the file lock (if provided).[1]
I can’t say I’m hugely encouraged by the creep in supporting various sets of extras and the like, but at least if those were to be just boolean filters in the package lock (with the locker, not the installer, handling transitivity… good luck! ) then at least it’s somewhat contained. Allow the same environment tags as we already have, maybe define a new “custom” one that can be provided by a user at install time, and let the lockers figure out how to calculate it.
The file lock is then the outer limit of what files may ever be installed. So if you audit just the file lock, you see the worst-case scenario. Installers would have to order the files (as today, and as under a package lock) and take the best one (i.e. that matches the platform).
Allowing a wildcard on filename would then say “lock to numpy=1.2.3 but install any file that matches numpy-*.whl”, which is enough flexibility to get post/local versions if available (and block sdists, in this case). Equally, “lock to numpy=1.2.3 if on win32 and install numpy-1.2.3-win32.whl” gives a condition tied to the package, not the file, but still limits the files.
So there’s only a single lock in each file - not two different types of lock. Packages are locked by name+version, included/excluded based on markers, and constrained by the names/hashes of allowable files.
I’m pretty sure one of the other existing lock formats works like this? I’m sure I’ve seen it, though I don’t remember whose it was. ↩︎
I might be backwards, but to me it seems like this is backwards; i.e. that a “file lock” is just a specific variant variant of a “package lock”. In a previous comment I wrote a theoretical poetry export --extra foo --group bar --environment {environment marker}, which in my mind is taking some preexisting package lock and producing a file-lock from the result for the given target installation command.
And that is why i feel like there’s no distinction, perhaps beyond the desire for an easy path for the simple case, between the two.
I feel like a key thing though, if this “security” aspect is your goal and you’re only producing a file-lock anyway, then you have a fixed extra/group selection anyways. you’re varying by environment or not at all. So a “package lock” that happens to resolve away the extras/groups and lock to a specific environment is a file lock, as far as i can tell.
i feel like solving for package locking concerns, and then also providing a way to predefine environments (like you did) solves a bunch of file-locking problems also.
you, as a person that wants a file-lock would simply pick a file-lock-style producing tool that would have no need to include any information about the selected extras/groups because they are irrelevant to your installation scenario. And by doing so, that will retain auditability, because your particular file wont have those axes of metadata variance.
I broadly agree with the way @DanCardin frames the distinction here.
Over the course of the thread, I have come to view package locks as the overall description of the dependency set in an environment independent way, and then file locks (aka “supported environments” aka “resolved environments” - this bikeshed is still being painted) are a way of precalculating resolved artifact lists for particular target environments and recording them in the lock file for the purposes of:
lock file auditing against security expectations
installation with simplifed installers that can either be directly told “install this particular set of artifacts” (by naming the file lock/supported environment/resolved environment to install) or else choose the right set of tagged artifacts based on Python environment markers (and potentially OS environment variable values). These simplified installers will then fail if they don’t find a suitable named artifact set rather than attempting to figure out a new solution based on the full package lock.
So I think @brettcannon’s idea of renaming file locks to ‘<adjective> environments’ (exact name TBD) is a good one, there are just some details to work out in terms of how much of the work of resolving a package lock to an exact named artifact list is handled at lock file generation time, and how much is left to be handled at installation time (since those design choices will dictate what fields are needed to record these already resolved artifact sets).
This opinion seems to be trending towards consensus, but I’d (also) strongly prefer that the lockfile settle on a single format, rather than trying to support two with a single schema. And I’m open to that single format being focused on either the Package Locking or File Locking use-cases. I think we’ll end up with a much better format, and a much simpler story for users.
Since the latter has faded into the background in recent discussion, I do want to point out that the previous iteration of the PEP had what it framed as several advantages that are no longer present. Namely, the PEP was exclusively focused on File Locking, so it proposed a format that was optimized for that use-case: each environment was a flat list of packages, so you didn’t need to (e.g.) think through markers or resolve any indirection with environment names. Extremely simple for installers and for humans to audit. In that world, the PEP would be neatly solving for the “replace requirements.txt as a lockfile format” (as in pip-compile) use-case.
The biggest downside to standardizing that format is that (at least from my perspective) it will make it harder to standardize the Package Locking later, since we’ll then have two lockfile concepts, and we’d be setting ourselves up for all the confusion that comes with those two distinct concepts. I think users would also be disappointed to learn that the lockfile PEP doesn’t support the use-cases that they most-often associate with a lockfile right now (e.g., Poetry, PDM, or uv lockfiles).
Brett, can you walk through how the proposed idea solves for extras? Perhaps with an example? It’s not obvious to me from the description.
Assuming that you’re tying in platform details for e.g., “dev server” and “production”, then that makes sense. At that point you’re just expanding what an “environment” is, and thus why you made your “resolved environments” terminology suggestion.
I’m a little concerned of the impact on auditing as you now have to reference another file to get a complete view of what would be installed (and even assuming the installation would work).
I would not be opposed to a “SHOULD” line about where to put the files to help guide users towards common directory locations outside of the root of the project.
All file-locks.wheel-tags does is say, “you need the environment you’re installing into to have equivalent support of these tags”. It doesn’t require you use those tags in some way. But I assume it would be used to list the wheel tags used. But it could be the wheel tags used in your locked build requirements for your sdists, for instance.
Potentially. The trick is if you’re thinking of having a separate list of files/hashes to condense the details of the environment then you have to come up with a way to reference things like sdists w/ locked build requirements, VCS checkouts, etc. You can probably do it with an array of tables with critical keys specifying file, vcs, etc. and then the package version being referenced.
Or you say only wheel files are supported for file locking and if you want sdists you have to use package locking. (From a security standpoint that would the best approach, but considering what happened to me with PEP 665, I’m a little scared to propose that).
I’m starting to end up in that line of thinking if sticking with file locking is the way to go, then it either needs to be its own thing or stem from package locking in a nice way instead of the other way around (which is how these topics evolved).
I can’t because I don’t have a solution yet. I have some thoughts, but they haven’t solidified enough to show yet, especially if the format might fluctuate more heavily into file locking or package locking specifically.
The tricky bit with the whole extras thing is when the resolve varies w/ or w/o a certain extra being present (and this is even considering it being different per platform). This instantly shows me the appeal of what you’ve suggested you’re doing with uv.lock and doing edges from package to package and there’s a consistent package version for all scenarios; not having to care about which package version to pull in is a very nice simplification. But Paul pointed out how he didn’t expect that behaviour and would want different package versions if called for if the extras influenced things that way. Toss in different versions for different platforms or varying requirements per file and it isn’t so simple to write down in a readable, compact form.
But I have an idea around package locking that takes what you seem to be asking for based on uv.lock where package locking is the overall format, and adds in stuff @steve.dower and @ncoghlan suggested that acts like extra bookkeeping on top of that instead of an entirely separate format in the same file (and thus would make @DanCardin happy ). But since I just came up with this idea while writing this reply I will have to mull it over on the weekend to see if it solidifies into something worth mentioning here.
I think having the two locking levels gives you an escape hatch here, especially if file locks are reframed as being particular installation scenarios preresolved and the results integrated into the lock file.
You’d still keep the source references at the package lock level, but any resolved lock would need to do the source builds at locking time.
Personally I feel like this would negate much of the purpose of using a lockfile. Because how do I know for sure (or someone coming across a new repo) that this file plus some other file with a different path & name plus some third file are actually consistent? Plus auditability as Brettt said, not just for what gets installed but also seeing what the original set of requirements was.
Or many other files, yeah. I think this probably wouldn’t be a huge issue if it’s standardized (presumably an ‘auditor’ that already has the ability to read the lockfile that references other lockfiles should be able to read them too?) but does introduce complexity around paths (do relative paths work? can you reference lockfiles outside the root directory?).
I was originally thinking the PEP should just be opinionated about lockfiles only existing in the root directory, but thinking about this more, I think this is a non-issue: cloud providers can say “you must have a pylock.toml in the root”, and services like dependabot can just look for all pylock\.(.+)\.toml in the project.
(To be clear though, I’m just talking about putting lockfiles in subdirectories within the root, not above the root – not sure which you mean by “outside”, but I don’t really see a use case for the latter.)
Got it, so assuming a locker is listing the wheel tags used to generate the locks for that environment, [[packages]].files could contain source distributions as well in this example if there wasn’t an adequate wheel for that set of tags, correct? If so it might be helpful to include sdists in the examples.
Is the example here missing the equivalent of packages.file.lock? This seems to be necessary for this part of the installer workflow:
For each [[packages]] entry, iterate through [[packages.files]] to look for any files with file-locks.name listed in packages.files.lock.
Which I assume would become something like:
For each [[packages]] entry, iterate through [[packages.files]] to look for any files with supported-environments.name listed in packages.files.environment.
urllib.parse — Parse URLs into components — Python 3.12.4 documentation describes the relationship between the WHATWG spec and the RFC relationship “Following some of the WHATWG spec that updates RFC 3986, […]”. The spec itself says “Align RFC 3986 and RFC 3987 with contemporary implementations and obsolete the RFCs in the process.” and “Standardize on the term URL. URI and IRI are just confusing.”; It seems that the web is converging towards using URL over URI. Python seems to not have a distinct URI library either, but urllib seems to be mixing both. ↩︎
I mostly get that, but I think the part that’s giving me pause is: why is security about “reproducing environments” rather than “restricting the inputs to environments”? (In what I say below, I’ll use the word “environment” to mean a Python environment into which something is installed, and “context” to mean “the situation in which you want to install stuff into such environments and you may have concerns about security”.)
What I mean is that, if I were worried about making a secure environment that were restricted to specific files, what I would want would be to simply distribute a “greenlist” of files that I know are safe. This might be via a private package index or just a directory full of files or potentially a list of hashes of PyPI artifacts. But it wouldn’t be a statement about what can be installed in a particular Python environment, it would be a more global statement to the effect that all installation in this context must draw from the greenlist. This would mean that, in this context, not only could pip install mylockfile.lock not accidentally install something insecure, but even pip install insecurepackage could also not do so, because insecurepackage would not be on the greenlist.
From that perspective, “file locking” as contemplated by this PEP would be a two-layer thing: you package-lock, and then you also restrict the repository so that the package lock can only draw on known-safe files. Or you could just give a big list of filenames with hashes — but, again, this would be done at a level higher than resolving any individual environment, it would be done at the level of the context where security matters.
I’m speaking from a position of major ignorance here as I don’t actually do security-focused stuff like this, though. (Maybe that’s obvious just from reading my thoughts above? ) But in thinking about this PEP I’m struggling to understand why security of this sort should be handled in a per-environment manner (with a lock file) rather than at the “context” level, by globally blocking all installation (whether via lockfile or not) of packages that aren’t pre-approved.
Reading the comments above, I’m certainly +1 on having a single file format for superset and ci cases. Let’s make things easier for our users—they have enough on their plates already.
I just realised that existing tools are kind of lying when they create superset lock files. Surely most packages don’t come with s/390 or risc-v or java wheels.
IMHO, a superset lock must define a set of architectures or fall back on the escape hatch of compiling from source at install time, the latter may be wheels or sdists or a mix, and I don’t believe that compilation at install time is broadly compatible with the security aspect.
My gut tells me that likewise, the superset lock must explicitly list all possible extras (or link to pyproject.toml by hash) and make the promise that installation with all these extras is possible (on any listed arch), which in turn means that installation with any subset of listed extras is possible.
Then, a ci lock is a superset lock with a single platform, single architecture and a specific set of extras specified.
P.S. I think that a PEP for ci use case only would be aiming too low. Although it could make sense if the intention is to promptly follow with a PEP for the more general case.
Correct me if I’m wrong but I think the following is missing in the spec:
Selecting an extra in this package may affect extras in direct dependencies, which in turn may affect extras in transient dependencies… turtles all the way down.
I didn’t immediately see how that would be expressed in the spec.
I think there should be a guarantee that no matter what extras are selected, the packages that are in installed in both (pairwise) cases would be exact same versions. IMHO that’s required to test effectively.
P.S. on the second thought, perhaps it’s enough to state that it’s the lock tool’s responsibility to resolve all transient extras and list leaf extras? I’m assuming that explicitly installing leaf extras and the ln installing intermediaries packages without extras is equivalent…
My understanding is that what you are describing and file locking as described by the PEP may both be part of a security strategy and aren’t mutually exclusive.
For example, an organization may maintain an internal mirror with green listed packages, but with several projects being developed by different teams it will need to have several versions of the same dependency. Further, the mirror may be updated to introduce new versions and packages without input from every team / project that is using it. Additionally, security auditing is likely performed by yet another team independently of the teams developing the projects or the team maintaining the mirror.
In a situation like that, the easiest way for me (as a developer in this hypothetical situation) to know exactly what dependencies my application will be running with, and to show that to the security team or other external parties, is to have a list of the actual files that get installed. And to know that if those exact files aren’t available then the installer fails instead of trying to fall back to something else. “Exactly this or bust”, if you will. Not only guaranteed to be one of two deterministic outcomes, but perhaps more importantly it is easily provable to any other human on visual inspection that this is the case. I don’t have to manually trace through the possibilities in a package lock or trust a tool to do that for me, and nothing can change in any part of the environment or deployment process that will change what gets installed, ever.
Because a file lock is so exact, it not only describes what to install (future tense), but describes what is installed right now in a running deployment (present tense), and can be consumed downstream for other artifacts like an SBOM (past tense).
This, from my experience, is because security folks desire minimal ambiguity and maximum ease of auditing. For example, if you require some repository as a dependency you would never use Git directly but rather point to an archive and verify the hash after download. If you don’t, your security teams will make you.
edit: to put it more plainly, there is a strong incentive for introspection to act upon files rather than have additional runtime logic for audits
Package locking lists the packages and their versions that may apply to any environment being installed for.
I now believe that creating such an “allow-any” cross-platform lock is either impossible or extremely difficult to do correctly. A lock can only be applied on a constrained environment, the difference is whether the constraint is loose or not. This is also why I advocate unifying the two lock formats. In either package lock or file locks in the current PEP, the environment is specified by wheel-tags and/or markers, that gives pretty much flexibility on how “specific” they are. If they are restrict enough to be only applicable on a single environment, it’s a file locking, otherwise it is a package locking using the terms in the PEP.
I know package locking gets some inspiration from tools like PDM. From 2.17.0, PDM also changes to a new way to specify lock files. The previous “cross-platform” style lock files can be considered to no longer exist. And the lock file is restricted by one or more “lock targets” to describe the environments it can apply on. For example:
[[metadata.targets]]
requires_python = ">=3.8"
Means this lock file apply on all environments and platforms with Python >= 3.8, and:
means this lock file applies on only MacOS ARM64, CPython while still allows all python versions in range “>=3.8”
I am not suggesting to adopt the same metadata fields as PDM, and I believe the existing wheel-tags and markers in PEP 751 can describe the same environment constraints as above. Because they are all optional, the specificity of the constraints can be changed easily by adding or removing some entries from the list. For example:
[[file-locks]]
name = "wide-env-lock"
[file-locks.marker-values]
python_version = ">=3.8"
[[file-locks]]
name = "narrow-env-lock"
[file-locks.marker-values]
python_full_version = "=='3.12.4'"
platform_system = "=='Darwin'"
We need to change file-locks.markers-values to allowing a marker expression instead of marker value.
In this way, file locking is just another package locking with more restrict environment constraints, I don’t think it’s necessary to keep two lock formats anymore.
Yeah, this sounds totally fine to me. From a security POV, you’re in trouble with an sdist already, so being able to pin the hash of the sdist but no further is fine.
No reason an installer couldn’t take a second lockfile as input for build environments, if that’s what they choose to do. I’d actually really like that as a feature! But it doesn’t have to specified here - we should be able to define the hash checking as “at the point of download” and going no further in this spec.
I’m not sure if my support on this point counts for enough to calm the crowds, but you’ve got my support to go this way
Not in the case of per-file locking, but for per-package then yes.
I’m going to be a bit blunt here: what are you after? Separate keys for file paths or URLs? I didn’t do that as I don’t expect most lock files to specify file paths in packages.files.origin. Or do you not want relative file paths supported after asking about them?
You got a good answer from @flyinghyrax , but the one thing I would add is simply listing files doesn’t guarantee they don’t interact in a way that’s a problem; being “safe” in isolation isn’t enough.
Because it’s not. My initial expectation was separate lock files would be used for any extras you wanted to lock for.
Unfortunately you, Poetry, and now uv have done a good job convincing folks it’s possible.
If I’m reading what you’re suggesting correctly then is you’re advocating for always having some way to specify what requirements a lock file was generated under, whether they are overly strict or very broad?
From a pure, “get me this stuff installed”, you’re right. But as I said in PEP 751: lock files (again) - #99 by brettcannon there’s an aspect to the explicit lock file scenario that makes it fit the security aspect a bit better than asking folks to do a fake install in their head or run some tool to know what would be installed.
I would definitely make it a MUST that lockers have to make it an opt-in to use source-based installs.
Over the weekend I thought about this from the perspective of a lock file of package versions and their files and having some way to express explicit/tight constraints (i.e. take something like what PDM, Poetry, and uv.lock offer and build an optional, security-focused aspect on top of it; per-package to per-file instead of the other way around). To me this feels more unified than my previous proposals.
Now I want to remind people this needs to work for both library developers (i.e. pyproject.toml) and app developers (i.e. requirements.in, maybe PEP 735 in the future).
Also remember that I don’t want to have to run a resolver to figure out what to install.
version = "1.0"
hash-algorithm = "sha256"
dependency-groups = {
'' = ["..."], # Or name "default" and have it manually sorted to the top? Or have a top-level `dependencies` that is ignored if a dependency group is specified?
# Extras are to have their names surrounded by `"[...]"` to differentiate from PEP 735 dependency groups. All names must be normalized (including all extra names). Extra names within a single dependency group should lexicographically sorted.
}
[[resolved-environments]] # Optional
dependency-group = ""
requires-python = "..." # Optional
marker = "..." # Optional
wheel-tags = ["..."] # Optional
packages = [
{package = "...", version = "...", wheel = "..."},
{package = "...", version = "...", source = "sdist|VCS|directory"},
]
unresolved-environments-allowed = false
[[packages]]
name = "..."
version = "..."
dependencies = ["..."] # Optional
extras = [ # Optional
"...": ["..."],
]
dependents = ["..."] # Optional
package-index-project-detail-url = "..." # Optional
requires-python = "..." # Optional
direct = false
files = [ # Optional
{
name = "...",
dependencies = ["..."], # Optional
# Optionally has `extras`, but its use disallows inline table usage so not shown in this example.
origin = "...",
package-index-project-detail-url = "...", # ?
hash = "..."
},
] # Optional
[[packages.build-requires]] # Optional
# ...
[packages.vcs] # Optional
type = "..."
origin = "..."
commit = "..."
[packages.directory] #Optional
path = "..."
editable = false
[packages.tool] # Optional
# ...
[tool] # Optional
# ...
The way to think of this is [[packages]] is a simple graph where the top entry into the graph is listed in [dependency-groups] and then you determine what to install by following where the graph takes you (I will cover this in detail shortly). The security side of things is [[resolved-environments]] which basically is a graph traversal that’s flattened and written down ahead of time.
I also put in support for extras. Probably the most controversial thing is how extras at the top-level of the lock file are normalized into dependency groups.
Lockers
[[resolved-environments]] gets written out in priority order. This resolves the question of what to do if you lock to a pure Python solution on top of OS-specific solutions.
Installers
Here’s an outline of how to decide what to install along w/ some open issues …
Iterate through each entry in [[resolved-environments]] based on the dependency group selected
[[resolved-environments]] is listed in priority order
If an entry passes requires-python, marker, and wheel-tags, install the listed files/packages as specified; DONE
Else if unresolved-environments-allowed is false, raise an error
Get the list of package dependencies based on the dependency group selected (defaults to '')
For each dependency, filter based on markers
Find the matching [[packages]] entry for remaining dependencies
Filter based on requires-python
Too “resolve-y”?
This is to support Paul’s “versions vary across extras” case in a single file for all potential extra uses
Choose the newest package version if a version specifier that supports multiple entries is used
If entries are sorted by packages.version from newest to oldest then it should be the first entry that passes muster
Too “resolve-y”?
This is to support Paul’s “versions vary across extras” case in a single file for all potential extra uses
Could require that version specifiers can only use == to avoid this while allowing varying versions
What if a compatible version lacks the specified extra(s)?
Error out?
Continue search?
Too “resolve-y”?
This is to support Paul’s “versions vary across extras” case in a single file for all potential extra uses
Look for a supported wheel
If no match is found in package version:
Raise an error?
Try next version?
Could require all package version specifiers only use == for simplicity and clarity
Use source if the user said using source was okay, go back and look again for source:
Install from source in order of most reliability and simplicity that the source code has not changed
Install from sdist if allowed
Install from VCS if allowed
Install from directory if allowed
Check source first before looking at wheels of all other versions?
Expensive in the face of source building
Also less secure
Too “resolve-y”?
Notice there is NO backtracking in any of these scenarios
Find the dependencies for the package (and extras)
Specified at the packages.files level
[[packages]] level
Whatever is specified from the source build
Repeat installation of dependencies based on algorithm specified for top-level dependencies
One thing I would be interested in is if [[resolved-environments]] meets the needs for security folks, so if the “security” label applies to you and your job/expertise, please leave a comment whether [[resolved-environments]] meets your needs, isn’t enough, or is unnecessary.
) then at least it’s somewhat contained. Allow the same environment tags as we already have, maybe define a new “custom” one that can be provided by a user at install time, and let the lockers figure out how to calculate it.
(From a security standpoint that would the best approach, but considering what happened to me with PEP 665, I’m a little scared to propose that).
). But since I just came up with this idea while writing this reply I will have to mull it over on the weekend to see if it solidifies into something worth mentioning here.