PEP 722: Dependency specification for single-file scripts

(I’m skirting being pedantic here, but) I think there does exist a world where we use TOML for more than just the dependencies, and therefore you’d be forced to re-read the TOML for the equivalent subset of information. [1]


  1. I’m pro TOML, but want to be the most precise here ↩︎

2 Likes

This was from a caching perspective. My hope is to be able to cache based on the specification extracted from the block, rather than using the raw text or some other measure that’s not as directly correlated to the environment that’s going to be created.

Obtaining the specification from PEP-722 is relatively straightforward and fast, using a TOML format requires more work by the parser. Either I need to accept that performance hit or find some other element to base the caching on[1]. The difference is relatively small in the context of existing tools like pipx, hatch or even pip-run so perhaps it doesn’t matter.

Anyway this was mostly intended as a way of expressing a preference for the plain list of dependencies proposal and explaining why. Apologies for the performance tangent this lead to.


  1. or write it in another language. ↩︎

3 Likes

OK, I’ve added those two items to the PEP.

Also, I’ve come to the conclusion that it’s unlikely that @ofek and I are likely to reach any sort of compromise on a combined PEP. I think our goals are simply too far apart for us to be able to agree on something that we’ll both be happy with. Also, I think that enough people have commented on this thread in support of a simple comment-based format, that it wouldn’t be fair to simply switch to a TOML-based format - and the only way I can see to avoid that is to submit PEP 722 as it stands for approval. So I’ll leave @ofek to finalise PEP 723, and it’ll then be up to @brettcannon to make the final decision.

As I’m going to be away next week, I don’t expect to make any further changes to PEP 722, or even to be following any further discussions, so this can be considered the final yes-actually-he-means-it-this-time version of PEP 722 :slightly_smiling_face:.

7 Likes

Oh, I thought we were close to agreement. No worries I will update 723 over the weekend!

2 Likes

I felt like I was just conceding more and more, and when I stopped to think about it, I had given up on more than I was actually happy with, between the TOML format, the parallels with pyproject.toml (implied rather than explicit, but still there), not deferring requires-python to a later spec, etc. And none of the objections I raised in PEP 722 had actually been addressed, they’d simply been ignored in the interests of compromise.

And on the level of intention, it felt like I wanted to standardise existing practice, and you wanted to design a new feature, and I think that’s an important distinction which we’d made no progress on resolving.

Ultimately, I guess I felt that I’d moved my position far more than you had, and that didn’t feel right.

But regardless, thanks for being open to the idea of a compromise solution. I hope you can come up with a good final version of 723, which makes your arguments the way you want it to. And then it’s up to Brett.

8 Likes

I am worried that there’s a chance we miss the forest for the trees when “choosing” between PEP 722/723.

I voiced this in PEP 723: Embedding pyproject.toml in single-file scripts which got split to PEP 723: use a new `[run]` table instead of `[project]`?, but it applies here as well.

I can speak personally, that if I had to choose between these two PEPs I would likely settle on PEP 722, yet it might not be as robust or flexible as a solution that finds a way to embed structured metadata into the script (that isn’t labeled as “pyproject.toml”, as I know you and others have concerns). Then we’re forced to either find a way to “extend” the embedded metadata approach of PEP 722, or invent a new one to replace or live alongside PEP 722 [1] once we want to embed more.

I honestly have the capacity and stamina to co-author a PEP to lay the groundwork taking everything said here (and the other 3 or 4 parallel discussions), but I also know that’s somewhat rude and stressful to you (Paul), Ofek, and Brett. So I’ll just voice my concern, and won’t pursue that unless asked by one of y’all.

EDIT: (I’m backpedaling on the “accidental” wording here, and the expectation of this syntax being extended after re-reading the PEP)


Regardless of my concern, I’m personally excited that we’re addressing a gap in support for what I perceive is a decent chunk of Python’s usage. So :tada:
And professionally, I’ll parrot myself from the PEP 723 discussion

Speaking on behalf of Pex and Pantsbuild, we will support whatever decision is accepted, and neither seem technically infeasible.


  1. and then people lob tomatoes and scream fragmentation. ↩︎

This is the crux I think (not picking on you personally but using this message to point this out):
PEP 722’s goal is not to embed metadata into a script. It just wants to allow script authors to write down their dependencies so that script runners can make them available.

3 Likes

Maybe it’s pedantic, but that most certainly is metadata (whether its called that or not on the tin).

From Wikipedia:

Metadata (or metainformation) is “data that provides information about other data”,[1] but not the content of the data

The dependencies of a script is data that provides information about the script.

…Actually, it IS called that on the tin! The PEP itself in the “Rationale” section:

We define the concept of a metadata block that contains information about a script. The only type of metadata defined here is dependency information, but making the concept general allows expansion in the future, should it be needed.

(That’s not even my emphasis, “metadata” is already emphasized in the PEP)


And that’s the entire point of my concern… We accidentally standardized a way to embed metadata.

EDIT: Although, from the PEP I should backpedal the “accidental” part. I’ll own up to that.

3 Likes

(I need to make a dedicated comment for this, apologies for the noise)

I’d like to rephrase my concern (and apologize for my earlier wording), since I see now we’re explicitly standardizing a way of embedding metadata.

My concern is that we standardize this way of embedding metadata through the lens of this use-case and not others. So still the same forest/trees concern, but certainly not accidental (and again, my apologies)

1 Like

I am concerned about that too. However, @brettcannon indicated on a related thread that there will solicitation of user feedback in the process, and that it remains quite possible that that feedback will indicate a solution that differs from both proposals. That alleviates my worries somewhat.[1]


  1. I do still think it would often be useful, not just for this PEP but for many ideas in the past and future, to get such user feedback at an earlier stage and use it to inform the initial drafting of a PEP, but still, getting it and taking account of it at any stage is valuable. ↩︎

2 Likes

This is absolutely correct.

To expand a little, PEP 722’s goal is to standardise existing practice in the area of allowing script authors to write down what distributions they need available in order to run.

It seems to me that PEP 723 is trying more to address the question of “how do we expand the idea of metadata (pyproject.toml style) to cover single Python files”. That’s a much bigger question, and while I think it’s potentially worthwhile to address, I don’t think we yet have sufficient evidence that people need that capability[1], and I definitely don’t think we’re even close to a good idea of what such an expansion should look like.

I understand that some people are uncomfortable with small, incremental improvements, as they are concerned that such changes ignore the longer term. I disagree with this, personally. My view is that incremental change is crucial if we are to progress - endlessly debating the “long term” will simply drain everyone’s energy, and responding to every proposal with “but what about the bigger picture” ignores the reality of how volunteer-driven projects tend to progress, which is in terms of small, focused PRs and closely scoped feature developments.


  1. Outside of the limited situation where it addresses the “running a script with its dependencies” case. ↩︎

5 Likes

You’re reading an out of date version of PEP 722. The current version removed the concept of a “metadata block”. This was all covered earlier in this thread, although I can understand if you missed it - there’s a lot to keep up with (I should know, I’m exhausted!)

I should’ve known better than to atke the link from the very top OP.

So I redact my redaction? This is getting confusing :sweat_smile:

I think everyone gets the gist (I hope) I’ll tap out before I hit strike 3

I’ve edited the original post, to give the correct URL.

1 Like

I think that’s more of a discussion between you and @ofek about if he agrees with you and wants to change PEP 723 to align with what you’re suggesting. Otherwise I’m not sure how your PEP would differ from PEP 723 beyond what TOML data is included or some bikeshed on how the TOML is embedded.

1 Like

I agree with what Brett said but Joshua feel free to message me here, X (formerly Twitter) or Discord

1 Like

I do not like specifying requirements in the comments block. It looks very strange that comments, which should be normally ignored, in this case have a special meaning.
For Python newbies not familiar with packaging, this PEP is not the solution. It takes additional effort to figure out that this kind of syntax is used. Do not forget that if this PEP is accepted (which I would like to avoid), this practice has to spread across the community, which may take a long, long time.

A newbie who would like to package his one-file script would encounter solutions based on requirements.txt, pyproject.toml, etc.) on the web much more frequently because they exist for a long time. It may occur that this solution is one more niche, which for insiders is yet another way to package (but they may prefer more cleaner and standardized ways like requirements.txt, pyproject.toml etc) and for the rest, it may be only an unknown curiosity or thing to ignore (because it looks like usual comments).

To summarize. Please do not accept that and do not introduce “yet another way” to package.

This PEP isn’t meant to be a solution specifically for “newbies
unfamiliar with packaging” nor is it a way of packaging anything at
all (much less another way of anything).

Many of us who write quick Python scripts already put comments in
the beginning of them with a list of the non-stdlib dependencies the
script needs preinstalled. There are even tools which will read such
comments and autocreate suitable ephemeral environments before
running the script. This PEP merely standardizes that practice so
that these tools can have a consistent format to collaborate on
rather than each doing it their own way.

Not approving this PEP isn’t going to stop those of us who already
do what it documents, nor is it going to make the tools which use
this information suddenly disappear, all it will do is increase the
odds that two different runners will each expect a different syntax.

13 Likes

It seems there’s still misunderstanding: This is not a way of packaging at all.

This is a way for someone who writes a script to provide a comment to users (“Your environment will need these dependencies to run”), but structure it so that script runners can build an appropriate environment (or verify that the environment has the dependencies).

Critically, this capability already exists, but in different forms in different tools. By agreeing to a common syntax, the script writer who chooses to use it does not have to say “run this with pipx” or “run this with pip-run”, but “If you use pipx or pip-run (or X other runner), it will just work. Otherwise, read the comment and make sure you have these things installed.” I would probably actually make this explicit in a comment:

# This can be run directly with pipx or in any environment with these installed.
# If you need to edit this list, refer to PEP 722 to ensure it remains runnable.
#
# Script dependencies:
#    pandas
#    matplotlib

There’s no requirement that any tool that is not aiming for compatibility to implement this, and there’s no requirement for a user to use those tools.

3 Likes

I think this is true. But it’s probably still worth considering the outside perspective, even if it’s technically incorrect. If this PEP (or 723) is perceived as complicating the ecosystem, that’s not a good outcome.

Not that one person’s comment is representative of anything, but I hope the user studies that Brett’s group is working on will clarify whether this is happening.

6 Likes