I think it’s actually the other way around, that on an unlocked script it’s likely to break quicker, as newer versions of packages make breaking changes your script doesn’t handle. In my experience libraries make breaking changes faster than CPython does these days.
And for locked scripts this is an easily solvable problem, put a check at the top of the script to exit early running against a Python version not verified, with instructions on how to relock for new Python versions.
Whereas once the dependencies break, unless you mark down the exact time it worked or every version of every library (e.g. locked) you might struggle figuring out what were the working versions.
That check would be too late if it’s an install time error (failing to compile binary dependencies is generally the most common Python version transition breakage in my experience).
I don’t really want to get sucked into a to lock or not to lock debate but I do find that the people who aren’t familiar enough with packaging to build a wheel are also not familiar enough with packaging to know the consequences of locking.
And I find that the same crowd have issues with any dependency issues, and if the dependencies aren’t locked they run into issues much quicker because transative dependencies can update in incompatible ways with other transative dependencies.
Much less than “a couple of years”, typically. To use your example, if I were to lock a script with a numpy dependency today, it would pin to v2.3.1, which doesn’t include wheels for Python 3.14. Three months from now, if anyone tries to run that locked script on the then-latest Python version, they would have to build from source, which would likely fail. In fact, with CPython being on an annual release cycle, the longest a script is likely to work before it breaks is 1 year, unless it has no dependencies with binary wheels whatsoever.
So locking dependency versions without also locking the Python version is going to be a recipe for disaster. If this discussion turns into a PEP, that PEP should strongly recommend locking a specific Python version in requires-python, even though upper bounds are usually discouraged.
Anyone locking the dependencies for a script should also be locking the interpreter version the script runs on, surely? Then it should run correctly for as long as that Python version is available.
Sorry. I didn’t finish reading your post. Exactly this, although I don’t think this is something that should be part of the PEP as such, it should be part of what we mean by locking. Maybe the PEP needs to reiterate that fact, but that just suggests that people haven’t yet really appreciated the point about what locking a published application involves.
I’m enjoying a healthy debate on what the values of locking a script are. We should try and see if we can get to some mutual understanding of it.
I will warn that we should remember that everything is a tradeoff. In this case the primary tradeoff of locking that we’re discussing is “reproducibility/stability at the expense of the benefits of transient upgrades (usually seen as bug fixes)”. Both are valid things to want but I’m not sure we’ll ever find a solution to both at the same time.
The way I see we already have the latter. What I want (for my use-cases) is the former. Perhaps then we just should focus on educating users when to use what (assuming you can lock a script).
One other theme here is the amount of information in the lock file. I kinda took the easy way and just hand waived “let’s embed the lock file”. But perhaps someone (maybe even me, but doesn’t have to be) could take the harder way and enumerate what’s in a lock file and see if it makes sense to keep to omit.
We could potentially end up with an acceptable sunset and have our solution
Specifies the Requires-Python for the minimum Python version compatible for any environment supported by the lock file (i.e. the minimum viable Python version for the lock file).
Side note: The lock file specification is the only place that specifies the semantics of requires-puthon, but in prior DPO discussions people have been very against using the field to pin or add an upper bound on Python.
But let’s take a step back, this criticisms of Python versions and lock files is a criticism of lock files themselves, not of lock files in scripts. They equally apply to the original spec or to zipapps. So I don’t think it’s valid to say this criticism should block people’s new use cases of using lock files, we’ve already agreed (by accepting the original PEP) that lock files are valid. And if someone wants to improve lock files for this issue around Python versions I suggest that make that a separate post / PEP as it’s orthogonal to whether lock files should be in a script or not.
I don’t believe it is accurate to call it orthogonal. People accepted lockfiles in a distribution context, but the acceptance did not cover the implications of locking a script and how it interacts with people’s assumptions of scripts. Nobody is claiming lockfiles are bad, but that the known downsides of locking are worse for users in a script context.
Unless you’re also reproducibly building the python interpreter with the same environment and linked system libraries, you don’t have reproducibility from just locking dependencies. I don’t think this is the right path to reproducibility. Some people handwave this away, but they really shouldn’t. Since reproducibility already goes beyond the scope of a single file with python, I don’t see a good reason to bake locking into a single file.
Could you expand, I don’t understand why any of the criticisms about Python versions are worse in a script sense. What makes a project failing on a newer Python version better than a script failing on a new Python version?
Also, tooling could help here, in both cases, such as lock file generators having a mode that excludes sdists where possible, and installers letting users know that wheels are only available for certain versions of Python and not the users (or in cases like uv and others they could install a version of Python that matched what wheels are available).
Bugfixes aren’t just bugfixes, they are also security fixes. While this one applies to packaging too, unlike packaging, people aren’t used to having to update single file scripts for dependencies. This makes it a change that requires socialization in a way the packaging one did not, and there aren’t the same distribution mechanisms already in place for “just share a script file”.
The common current target audience for scripts aren’t expecting this behavior (discussed above)
Lockfiles can’t specify a python version (as you pointed out). Distribution through an installer without locking to an exact version handles selecting a compatible dependency, and if the dependencies are stable and supported, for even libraries that drop python versions “Relatively quickly”, this likely means at least 3 years of compatability and bugfixes, rather than breaking on the first python version change (at most, 1 year). The example people gave with numpy actually illustrates this, as numpy has a breaking change schedule and python version drop schedule, if you are only trying to be compatible and not reproducible, you can clamp the upper version based on that schedule, rather than lock an exact version.
Strong -1 from me on having a different spec for locking a script vs a project. On the other hand, I’m +1 on lockers omitting everything non-essential when locking a script.
Exact python version pins might be prohibited in lockfiles in general[1] but when building an application, you would typically fix the Python version - maybe not in the lockfile, but somewhere else in the build process (e.g., when you pick an interpreter to embed).
As @thejcannon said, there are different use cases, but the one I think we’re talking about is more like application deployment - and for that, pinning the interpreter is perfectly reasonable. It can be done with the script’srequires-python metadata rather than the one in the lockfile.
Ultimately, though, the point is that people need to think before locking a script. If they don’t, that’s when they will produce something unmaintainable or unusable.
The key question is whether it’s too easy for users to do the wrong thing here.
I don’t recall whether this was discussed in any detail during the lockfile threads, or if it was just “like other interpreter version requirements” ↩︎
No I certainly don’t have complete reproducibility, but I do have more reproducibility. It’s not an all-or-nothing. I’d like more than what the current spec peovides.
As you say this applies both to packaging and single scripts.
I would argue the main security concern with running scripts is that you are running an arbitrary script in the first place. And that locking is far more likely to reduce your attack vector as it prevents you being exposed to packaging hijacking attacks and dependency confusion attacks.
I would therefore say this argument is a much better criticism of having in-line dependency metadata at all rather than locking specifically, which while I agree there are some trade offs I think generally improves the security situation.
The linked post is more a criticism of the concept scripts having in-line dependency metadata, yes passing scripts around without scrutiny can cause significant technical debt. But we aren’t their managers, users will find ways to build tech debt with any set of open interoperable software tools.
IMO lock files help not hurt here, if the script breaks because it was an unexpected Python version that is better than continuing to work even though it now outputs the wrong results because two transitive dependencies no longer interact in the same way.
This is criticism of lock files in general. But without locked files your transitive dependencies might break your script in the next second at any moment, Python versions are far more predictable.
I do agree it would be good to have a solution for specifying Python versions:
That seems reasonable, although would be a new recommendation for requires-python and would go against previous advise on how to use that field.
For the last point I would like to hear the opinion from popular tool authors that would likely support this.
Edit: Apologies, I made a slight edit to this post to remove all the "again"s I wrote, on reflection I didn’t like the tone it created, reasonable minds can disagree and I don’t want to imply otherwise.
It most certainly wasn’t. I’ll highlight the exact contrast I read, and the most important part of it to refuting this:
…
The implication here is not that inline script dependencies massively change the behavior people had for scripts in terms of their dependency updates being managed by some other tool (system package manager or tool runner), but locking stops users from getting those managed updates.
There are inherent problems with people who run untrusted scripts that no amount of design will prevent. In fact, even nagging users about untrusted scripts can have Worse outcomes. How many users just automatically press right through Windows UAC prompts and end up treating them all identically? There’s an actual term for this (“security fatigue”), and it’s something that people are actively researching.
It’s part of why making things work in a predictable way that doesn’t require extra steps people have to learn to continue using their existing tools in a safe manner is such a problem. I was lukewarm on inline-script metadata. It’s fine, but I was never going to use it. I can include instructions on required dependencies or just make a zipapp if the target user isn’t expected to manage that, and I prefer that as it almost invites the user to ask or point out if a dependency is no longer compatible.
It crosses a line on changing behavior when there’s hard locking involved that I don’t think is going to be good for most users, even if it somewhat assists a small handful. I’d rather find ways to help the problems that small handful have that won’t significantly change what people running python scripts need to be aware of.