Clarify that PEP 723 inline script metadata recognition is textual

I think the current PEP 723 inline script metadata specification leaves an accidental ambiguity about whether block recognition is textual or follows Python syntax.

The prose describes “top-level comment blocks” and says that it takes precedence over the regular expression. The canonical regex and reference implementation scan the source text without tokenizing Python, so they also recognize a matching block inside a string literal.

That difference is now observable across implementations:

  • uv and pipx scan the source text and recognize blocks inside strings.
  • Pex parses the Python AST and skips them.

That makes this more than editorial in effect: the proposed clarification would make Pex’s current behavior nonconforming.

My read of the earlier discussion is that the textual behavior was intended. The original acceptance rationale emphasized keeping parsing simple and consistent across tools written in different languages. When ambiguity between the prose and regex came up, Paul said the regex was intended to be precise, the text should match it, and that this could be clarified later through the specification update process.

I’d like to make that explicit in the canonical specification without changing the block syntax or requiring Python parsing. The core addition would be:

Metadata blocks are recognized purely from source text. Tools MUST recognize a block that matches this syntax regardless of its position within Python syntax, including when it occurs inside a string literal. Tools therefore do not need a Python parser or tokenizer to recognize metadata blocks.

I’d also replace “top-level comment blocks” with “metadata blocks”, say that the canonical regex implements the recognition rules, and add a recommendation showing how generated Python source can indent a block and use textwrap.dedent to avoid accidental recognition. That recommendation is already tracked in pypa/packaging.python.org#1835.

Relevant prior discussion and reports:

Would this qualify as a text-only clarification under the PyPA specification update process, or should it go through a new PEP? I have a small documentation patch ready if the clarification route is appropriate.

1 Like

I’m pretty sure the intention was that you didn’t need a full Python parser to identify script metadata blocks. Having said that, I’m sympathetic to the argument that quoted strings should not be scanned for metadata (the term “top-level comment” clearly excludes such cases, and in any case it doesn’t seem like a reasonable way to include script metadata in a file). And I don’t agree that the regex should be made the canonical implementation - we had that debate at the time the PEP was written, and I’m pretty sure the conclusion was that the spec should be definitive, not a particular regex.

I’d be inclined to leave the specification explicitly saying “top-level comment blocks” but add:

  • Tools MAY choose to do a simple textual scan, rather than doing a full Python parse.
  • As a result, the behaviour of scripts that contain data that looks like metadata within another Python construct such as a multi-line string is tool-dependent and should not be relied on.
  • The provided regex implementation is a textual scan, and as such will not ignore metadata in a multi-line string.

That basically formalises the idea of “don’t do that, then” as a resolution to this issue :slightly_smiling_face:

7 Likes

I’ve created a PR with my proposed change: Clarify handling of script metadata by pfmoore · Pull Request #2089 · pypa/packaging.python.org · GitHub