Pre-PEP: Locking a PEP 723 single-file script

Packaging
81

Thanks for the response.

Great if we agree already! My initial response had been to “I’ll object strongly to any attempt to remove that flexibility”. I think it’s fine to keep flexibility, in the sense of not constraining reasonable usage patterns without any rationale. But for the /// lock sections, there is a clear rationale to curtail that flexibility somewhat, in the sense that “hiding” executable in/after large comments does not become an attack vector.

So I think we agree that flexibility is good (e.g. introducing other types of blocks can be introduced as necessary). We may also agree that if lockfiles get added to scripts, they should be at the end.

Unusually for me in these discussions, I have a very concrete memory of (and to some degree, participation in) this; to me the distinguishing factor in favour of PEP 723 was hewing as closely as possible to the established pyproject.toml style of dependency-specification, rather than inventing another scheme to do so. The former naturally includes a degree of flexibility – tools can leverage existing patterns to add something like

# [tool.uv]
# exclude-newer = "2023-01-10T00:00:00Z"

and no spec or implementation needs to be changed for this to make sense or be compatible –, but that doesn’t mean that flexibility in and of itself was the overriding principle.

I think the key point is avoiding executable code creeping in after large lockfile blocks. Anything that’s just comments could still be allowed. I guess I could live with a “strong recommendation”, though preferably my suggestion would be: “your locked script is the concatenation of foo.py plus its lockfile”.

That might actually be a neat approach to square the circle with the whole sidecar discussion. In that way, users could choose if they want to have

foo.py       # unchanged
foo.py.lock  # sidecar
# or
foo.py  # concatenation of original + its lockfile (wrapped into `/// lock` block)

and both variants would be trivially transformable into the other. Normally we prefer “one and only one way to do it”, but as long as the two instances are kept interconvertible[1], this wouldn’t actually be a large implementation burden IMO, but would provide users with a meaningful choice that’s hard to dictate for everyone (i.e. whether keeping everything in one file is more important, or whether they want to avoid large blobs in their scripts).

  1. something that would clearly favour keeping the /// lock section at the end to simplify this process ↩︎

82

It isn’t, particularly. But I didn’t particularly get the impression that shipping a separate lockfile was “the current approach”, more that it was an alternative way of locking scripts that didn’t need a new standard to achieve. And the reason we’re looking at a new standard is because people don’t like having to ship two files. So all I’m doing here is pointing out that there are still issues even if we move to an embedded lockfile approach.

Maybe for you, but certainly not for me. If I distribute a script, I create it in my editor (perhaps with a manually written # /// script block for dependencies) and then send it to my user(s) with a note to “run it with Python” (or if it has dependencies, “run it with pipx run or uv run”).

Yes, uv could implement the UI you suggest, and that’s fine. But it’s not how everyone sees “building and distributing a Python script” (I’d argue it’s not even how most people see it).

It almost certainly isn’t. If you want to process script metadata, run the script with a tool that processes that metadata. But I don’t think adding more metadata saying what metadata must be processed will help, because then you just have to use a tool that recognises that metadata, and you’re no better off.

My comment was really just an introduction to the question of “what should tools do if a script has both dependency data and lock data?” And that’s an issue because both of those data types essentially describe the same thing (what libraries need to be installed for the script to work) and so we have questions about which section takes precedence, what happens if the two sections don’t agree with each other[1], and how much checking are tools required to do.

At what point do we stop, though? What if there’s malicious code hidden in the middle of a huge block of comment text that isn’t metadata blocks? My argument is that we can’t prohibit all ways of hiding malicious code, so we should concentrate on not implementing features that encourage people to see exploitable patterns as “normal”. And yes, that means I’m against this PEP unless it can successfully argue that the benefits are worth the risks (which might simply mean saying “this is a specialised feature and should only be used by people who understand the trade-offs”).

Ironically, something like uv lock --script mycode.py --inline would increase my reservations, because it makes locking scripts something people will do automatically, without thinking about whether it’s the right choice.

Ah. That’s our point of disagreement, then. I don’t think individual metadata section types should be treated differently. IMO, if lock sections don’t work well with the general rules for section types, then that indicates that lock data isn’t a good fit for the script metadata feature, not that we need to special-case lock data.

I could live with that suggestion, as a way of describing the recommendation. What I don’t think is OK is to somehow make the inline script metadata standard enforce it, because I don’t see a way of doing that without ending up with a set of individual placement and usage requirements for each section type, including any new ones we might add in the future.

I will note that none of this debate fixes the fundamental problem here, which is that I think that mixing human-readable code and an opaque blob of machine-targeted data in the same file is a bad design[2]. It encourages a lack of clarity over the question of who is responsible for ensuring that the script is safe to use, which I think is bad for security - something that’s particularly problematic when we’re suggesting it for lock data, which is presented as a way of improving security.

Anyway, I think we’ve probably covered this question well enough for now. It needs to be addressed in the PEP, and it will ultimately be up to @thejcannon how he chooses to do that. There are a number of other questions that need to be addressed as well (such as what happens when dependency metadata and lock metadata both exist, but they are incompatible), and I don’t want this particular one to overwhelm the discussion.

  1. Either because of a mistake, or due to malicious intent ↩︎

  2. Or less controversially, it’s an approach that has significant risks, and should be used sparingly, with due consideration of the trade-offs, not presented as normal practice. ↩︎

3 Likes
83

I think one way to address this is to recommend the block goes at the end, but also say that tools MAY raise an error if any code is found after the block for security/awareness purposes.

2 Likes
84

I agree with everything Paul has said. One concern I have is normalizing the practice of not reading a script you are about to run past a certain point.

An adversarial scenario to put into people’s heads:
A malicious script which intentionally “typos” the lock header.

# if the canonical name is "lock"
# /// locks

But also, I don’t want to over-focus on these elements.

Setting aside these sort of odd cases, the level of trust you need to install from a lock is much higher than the level of trust for installing from a (short, readable) list of top level dependencies. Think about who you consider trusted to update lockfiles you maintain – it’s probably not the same set of people who would be trusted to modify a dependencies list[1]. Perhaps a malicious script via “weird hacks” is no more likely, dangerous, and troublesome than a malicious script via crafted lock data.

If I invoke pipx run on a script after reading a /// script block, today, I know what will be installed. If lock data is present (today, alongside the script), and I use uv run, I generally don’t know what will be installed.
What is the right level of caution around this feature idea? A reducto ad absurdum version of my reasoning concludes that lockfiles themselves are dangerous! :joy:

  1. presuming that the change is desirable, I mean ↩︎

2 Likes
85

Couldn’t this be avoided by having the installation tools check that the lock data actually matches the dependecy list? E.g. if my script lists packages A and B as dependencies and those transitively depend C then the install tool would first check that the lock data really only contains A, B and C.

Then users can just check that the dependency list matches their expectations and ignore the actual lock data.

86

It seems to me that we might be dismissing a useful feature that would be useful to some users and is already available in the ecosystem. We seem to be dismissing it because of potential for abuse and not inherent insecurity of the feature.

I think we should be considering this feature on merit in non-adversarial contexts, while highlighting that there is always a risk of abuse when code execution is involved.

I disagree with this.

There are pros and cons:

  • Adversarial scenarios: Both an unlocked and locked script can run arbitrary code and dependencies that are not in the top level dependencies (adversarial script, not adversarial supply chain)
  • Non-adversarial: An unlocked script is at greater risk of supply chain attacks in this scenario (non-adversarial script, adversarial supply chain)
  • Non-adversarial: An unlocked script is more difficult to retroactively audit or understand precisely what was installed when the script was run in the past

I dont think that the argument that it’s more difficult to review a script with an inline lock is accurate. If anything, it would highlight the need for more scrutiny for users who are sophisticated enough to run audit tools or review scripts. Some of it can be aided with docs or tooling.

So I’m not sure we should reject an optional feature that has pros and cons.

Pros:

  • More reproducible scripts
  • More secure scripts from supply chain / dependency cool down POV
  • Far easier to build and distribute a single script than multiple files or archives

Cons:

  • More lines of code to review / be suspicious of
  • Risk of stale dependencies if lockfile is never / rarely updated

I think we should let the author and user of a script make their own decisions on these trade-offs based on their own risk tolerance levels for executing code.

1 Like
87

I think the main difference is that it’s significantly easier to audit a dependency list by hand than a lock file/block by hand.

When embedded in a script as proposed, it also becomes a means to fatigue users who attempt to review, eventually sneaking something in by abusing what people gloss over.

If it were required that lockfile blocks were always contiguous, singular, and at the end of the file, and must match a dependency list block that is always contiguous, singular, and at the start of the file, I think it overcomes those problems while getting the positive benefits of locking.

2 Likes
88

I don’t think this accurately states the risk here. I’m not saying “and therefore the feature is bad and should be rejected”, but the danger should be properly understood.

In the presence of lock data, a script which is not overtly malicious in its content, and which has no malicious listed top level dependencies, may still be an attack vector. The lock data may contain packages which shadow or patch other modules. There is no canonical mapping from package names to import names. Installing from a lockfile which you do not trust is as dangerous as installing a package which you do not trust.

It’s not only nontrivial to determine that a lock does not match requirements – “does not match” is ill defined. The lock may have been produced by another resolver (or the same resolver but with different options/parameters). That resolution may have run with different published state on PyPI.

What I’m getting at is that in order to audit an unlocked script, I need to read the content and the dependency data. In order to audit a locked script, I must also audit the lockfile data itself.

The reality is that most users will not do this kind of work to audit a script.

I’m not sure what this means for this feature idea. To me, it is a significant risk and challenge that any proposal needs to address.

3 Likes
89

Perhaps if you want to use an inline lock file you must also encode the index urls and the lock date in the top level metadata. That would allow the installer to verify that the package versions (and shas if included) are actually valid in that context before installing them.

90

I would assume that “matches the requirements” just means that the package/versions specified to be installed in the # /// lock section have to satisfy all of the requirements given in the # /// script section. Or to put it another way, if you build the environment specified by the lock data, that environment must satisfy the requirements in the script metadata. That’s the only definition of matching that I can see which doesn’t require that resolving a set of requirements is exactly reproducible - and if that was the case, we wouldn’t need lockfiles at all.

Of course the problem with this definition of “matching” is that it does nothing to prevent the easiest form of exploit - namely, adding an extra package to the lock data that isn’t required by the script metadata section.

This is the core of the problem. People will scan a set of dependencies in a # /// script block and quite likely spot if anything looks amiss. That’s not a robust audit, but it is a sensible safety check. With a # /// lock block, people will almost certainly just trust it, because even a quick sanity check requires more review than most people will likely bother with.

3 Likes
91

Can’t you check for that by essentially checking if the lock file represents a possible solve of the depency specifiers? So e.g. if the script depends on A you look in the lock data and see that it wants to use A version 1.2.3. Then you check the dependencies of that version and see that it depends on B and C. You again check for versions in the lock data, and so on. At the end if there’s anything in the lock data that you haven’t seen in the dependency traversal, it shouldn’t be there.

That should prevent people from just maliciously adding packages to the lock data. The other problematic lock data I can think of would be pulling from a source you don’t trust. But the installer could similarly keep track of all source domains and ask the user to confirm that they trust those. That way you know that you’re only getting packages you need and every one of those is coming from the people you’re expecting. At that point, I don’t really see how a lock file is any more questionable than the dependency list.

1 Like
92

Yes, that’s another possible definition of “matching”. But as @sirosen pointed out, now the definition isn’t well-defined any more. You need to define precisely what checks are needed to establish that the two sections match, not just give a rough idea. And don’t forget - tools are allowed to use different resolution algorithms, which could have very different ideas of what constitutes a “possible solve”.

That sounds like a pretty complex check. It’s not a resolution, so existing resolvers won’t necessarily be able to do this. But it has many of the complexities of a resolution (if you pick a different wheel for A 1.2.3, you could have different dependencies, if markers are for some reason different, you could get different results, if the lock was done with some constraints that aren’t present in the script dependencies[1], things might change, etc.).

You could add all of these checks, certainly. But for them to be reliable, the PEP would need to require that tools MUST implement all of them.

If this is the route we’re thinking of taking, I’d want to see a prototype implementation in pipx[2] and some evidence that people had stress-tested the checks with adversarial data.

But honestly, if tools have to write extensive checks with an adversarial mindset in order to ensure the safety of the proposed feature, that says to me that we have a deeper problem than “let’s just check for consistency”.

  1. Such as uv’s --resolution=lowest ↩︎

  2. Specifically pipx, not uv, as I think most people here would be better able to appreciate the impact if they are looking at Python code. ↩︎

3 Likes
93

Maybe I’m missing something, but to get to the same level of trust as a dependency specifier list, I don’t think we really need to check for different dependencies, markers, etc. If we say that by installing a script the user has signed off on the dependency list and trusts that, then I’d say that they’re fine with installing any of those packages (from whatever sources they trust) and anything that those package’s authors might have written in their dependency list. So we can check against that level of trust by checking if every item in the lock data (transitively) appears somewhere in the dependency specifiers, completely ignoring versions, markers, etc.

You definitely won’t be able to figure out if the lock data actually is a “reasonable” solve of the dependency specifiers that way, e.g. it might contain packages that shouldn’t be there because they’re for the wrong platform. But the user is presumably fine with those packages since they trust the authors of the packages in the script’s dependency list. To be clear, I’m not saying that this would verify the lock data in the sense that it definitely couldn’t be manipulated in some way. But rather that I think that we can get the level of trust of the lock data to the same as a dependency list.

2 Likes
94

You may be right. As I said, provide a precise definition and a prototype implementation and we’ll see. But personally, I think the cost of such robustness will be a lot higher than people seem to be assuming.

For me, the mere fact that the proposed feature led to a debate over how easy it would be to exploit suggests that there’s a risk here. It’s not about whether we can win the arms race, it’s the fact that there’s an arms race at all.

3 Likes
95

It seems to me that to mitigate the risk and make it easier to review the list of dependencies that will be installed (if a script were to be executed), we can suggest (SHOULD) that launching tools have options to “pretty print” or “dry run” what will be installed before execution.

Depending on the metadata available in the lock, it can be displayed as a dependency graph / tree or a flat list (the same as project.dependencies).

Tools supporting graph based metadata from the pylock (packages.dependencies) also have the mechanism to identify or warn about top-level direct dependencies from the pylock that are not present in the project.dependencies.

Then there is the outstanding item to ensure placement of the inline metadata is singular and contiguous.

1 Like
96

I’ve been reading along (when time permits), but otherwise have been trying to stick to not being on my devices after the kids go to bed (tonight’s an exception since I’m oncall).

Here’s my high-level thoughts for the PEP stemming from the recent discussion:

  • In the PEP, I’ll clarify that locking a script is an advanced workflow, and re-iterate the pros/cons of such a decision (especially as it relates to security)
  • Wasn’t discussed hardcore, but I think the “type” should be “lock” (to match “script”). It isn’t “pyscript” so “pylock” doesn’t feel right.
  • I think the safest way to settle the relationship between “script” and “lock” would be:
    • “lock” gets priority (put another way, if “lock” exists, the script runner can/should ignore “script”), but also allow script runners to offer a mechanism for the user to prefer/use the script block (e.g. using a flag)
    • script runners can give helpful information if the lockfile isn’t usable (e.g. platform issues), but a script block exists (although I’m not sure the spec should have a stance on if they should)
    • although the two blocks are generally assumed to agree, script runners are not forced to verify this, however
      • script runners MAY error (e.g. using a flag) if the lock block is not a valid “solve” of the script dependencies
  • On location of the metadata, suggest that the lock block go at the end (no Python code below it, comments OK)
    • But also suggest that “script runners SHOULD warn, and MAY error (e.g. using a flag) if it is not at the end” for security sake
  • I’m not sure I have a strong opinion on generation outside of “tools like uv are encouraged to have facilities to generate/update/export” them, but also “copying a pylock.toml generated through other means and putting the right markers around it” should also be sufficient (and we can provide reference implementation)
  • Similarly, tools are encouraged to allow for pylock.toml extraction, but we’ll also provide a reference implementation

I think giving folks a way to get a script to/from pylock.toml means that existing other tools that use pylock.toml for auditing should be usable.

5 Likes