PEP 722: Dependency specification for single-file scripts

[AA-Turner (Adam Turner)]

@sirosen

You could do it very strangely with

"""
Script dependencies:
"""
__doc__ += open("requirements.txt").read()

This wouldn’t match the specification as set out in PEP 722, which requires static data:

The dependency block is identified by reading the script as a text file (i.e., the file is not parsed as Python source code)

(Emphasis in the original)

In general, only string literals (incl. raw strings) as stand-alone expressions are lifted into docstrings (e.g. Expr(value=Constant(...)), though I might be missing a case.

---

I think the fairly straightforward argument in favour of docstrings is that users (e.g. Pradyun) would expect it, though it does make parsing harder (", ', """, and ''' are all valid openers for a docstring, before enumerating all of the string prefix possibilities!).

A

[pf_moore (Paul Moore)]

I’m not going to update the PEP because (a) I’ve said it’s ready for review and I don’t want to keep changing it after that, and (b) the “Rejected Alternatives” section is already huge, and I don’t think there’s a lot of value in extending it further. However, I will respond here, for the record.

First of all, I do actually think this is covered adequately by Why not use (possibly restricted) Python syntax? That section as written is focused more on the __requirements__ = [...] idea, but that’s because that was the commonest suggestion of this form at the time, but it’s not exclusive to that form.

@pradyunsg

You don’t need to parse the rest of the document – docstrings are guarenteed to be the first bit of “code” in the file.

You’re misunderstanding my point here. To find the docstring, you need to identify the “first string literal in the file”. And to do that you either need a Python parser or you define a restricted syntax for the docstrings you are willing to parse. For example,

"""
Script dependencies:
    \N{Latin small letter r}equests
"""

is a valid docstring. Would you accept that as declaring a dependency on requests? Yes, you can find it easily enough, but you can’t parse it with just a regex, for example.

And if you do need a full Python parser, you hit all of the issues raised in the PEP about tying the client to a particular Python version, being hard to write clients in languages other than Python, the Python AST compiler having no easy to use “incremental” mode, etc, etc.

@pradyunsg

Sure, rf strings are weird but those are reasonable to exclude

rf strings aren’t valid docstrings. So you’re safe there. But this emphasises my point that it’s really hard to be precise about the syntax if you define it in terms of Python constructs^[1], and really hard to define something that evaluates to the same thing statically and at runtime if you don’t. And I think having something that has different values statically and at runtime is a bad source of potential bugs and confusion.

Yes, this is to some extent about weird edge cases, but IMO that’s the key job of a PEP, to address weird edge cases. Anyone can write something that “usually” works, but once it hits the reality of multiple implementations and users that each interpret the spec their own way, vagueness is a big problem.

@pradyunsg

I say this in part because I had a docstring in my script already

Sure, it requires some familiarity to know how to specify your dependencies. But “allowing people to write a dependency block without any prior knowledge” was never a goal here. “You write your dependencies in a specially structured comment” isn’t a big thing to expect people to learn.

Also, people use __doc__ at runtime, and expecting those people to have to decide whether putting a dependency block in the docstring, where it will affect the existing usage of __doc__ (which is often user-visible) is a problem. Especially if there’s no alternative - and I hope you aren’t suggesting allowing a dependency block in the docstring as well as in a comment, because having two ways of doing this seems like it’s way too much.

I hope that explains why I don’t think the argument in the PEP is particularly weak, it’s just that it isn’t specifically framed in terms of docstrings.

---

1. I searched for the precise specification of a module docstring in the Python docs, and it’s pretty hard to find - hard enough that I gave up after 10-15 minutes of looking. ↩︎

[ncoghlan (Alyssa Coghlan)]

@pf_moore

I hope you aren’t suggesting allowing a dependency block in the docstring as well as in a comment, because having two ways of doing this seems like it’s way too much.

FWIW, I think it’s possible to allow dependencies in module docstrings without requiring a Python parser and without making the two ways to do it egregiously different: make the leading “#\s+” part of the header search regex optional. String escapes are already disallowed in the comment syntax, so it would be reasonable to disallow them even in docstrings.

The one additional restriction required on such a variation is that the dependency list would need to end at the first empty (or whitespace only) line in the docstring case, since “first subsequent non-comment line” wouldn’t be a viable terminator (a single or double quote character as the first non-whitespace character in a line would likely also need to be accepted as a docstring dependency list terminator).

That said, I don’t think it’s a good idea to do that immediately. The feature is distinct enough that it could be considered as a separate idea that depends on PEP 722, rather than being part of the baseline proposal.

[pf_moore (Paul Moore)]

@pf_moore

As I’m going to be away next week, I don’t expect to make any further changes to PEP 722, or even to be following any further discussions, so this can be considered the final yes-actually-he-means-it-this-time version of PEP 722 .

@brettcannon re-reading this, I realise it may have been unclear, but I do consider this PEP ready for approval. If you hadn’t realised that, sorry and you can consider this my formal request for a pronouncement.

As it’s 2 months since I finalised the PEP, can you give a time frame for when you expect to be able to make a decision? It’s not particularly urgent for me, but I’d like to have an idea if possible, as I plan to submit PRs to implement PEP 722 in pipx and pip-run if it gets accepted, and knowing when to make time for that would help me.

[brettcannon]

@pf_moore

re-reading this, I realise it may have been unclear, but I do consider this PEP ready for approval. If you hadn’t realised that

You were clear and I had realized it.

@pf_moore

As it’s 2 months since I finalised the PEP, can you give a time frame for when you expect to be able to make a decision?

We have user studies scheduled for Friday (which also eats into my open source time, so no one expect PR reviews this week from me )! It unfortunately takes a lot of coordination to make this happen since we have to plan the structure of the user study, get approvals (this isn’t cheap to do), find the participants, schedule them, do the user study with each participant, and then summarize the overall results. I can’t speak for @courtneywebster as to how long it will take to do the summary, nor do I want to put any public pressure on her to rush things, so I will say we will post a summary as soon as a summary is ready.

I’m expecting public discussions about the results, so we will see where the leads us.

My hope is, worst case, a pronouncement by the end of October.

[courtneywebster]

As @brettcannon mentioned, the user studies are scheduled for Friday, 09/22/2023. I will proactively commit to preparing a summary by the following week and encourage discussion based on any findings. Thanks for your patience as we continue to organize everything!

[brettcannon]

I should also mention Courtney may also reach out to some educators to get their feedback, but this is not a certainty and I also don’t know how long that would take. We are trying to be rather thorough on this, hence the long wait.

[courtneywebster]

I have posted the summary of the user studies in the PEP 722 and PEP 723 User Study Discussion thread. I am happy to provide clarity on anything covered there and answer any questions that come up as a result of the study!