PEP 723: use a new `[run]` table instead of `[project]`?

@brettcannon I have many work obligations this week but if you were to allow me one more week I can adjust to the new [run] proposal inside a comment. Otherwise, the current state pretty much represents the final proposal for the __pyproject__ variable approach.

2 Likes

Could we do [run] in this proposal and defer [tool] to a subsequent PEP? Because if we could do that I think we’re very close to something that we both can agree on, which in turn would mean that maybe I can help out with putting together the merged proposal?

But to be clear, I’m fine with waiting until you’re ready, so I hope it works for @brettcannon. But as a heads up, I will be away next week, so it sounds like the next time we’ll both be available at more or less the same time will be around 28th August. I’m not sure how much that matters.

Edit: Given that there really isn’t any rush to get a decision on this, maybe (if it suits @brettcannon) it would be better to simply say that we’ll wait until September before looking for a decision?

3 Likes

I’m fine with that, especially since it seems like [run] will not just be for scripts but a standard way to specify the runtime requirements of a “project” inside any pyproject.toml.

3 Likes

Excellent. But I’m concerned we might be misunderstanding each other[1], so to be 100% clear, what I think we’re agreeing on is basically a version of PEP 722 (not 723) which uses TOML in a comment rather than the custom “structured comment” format. The only defined key will be run.dependencies, and it won’t be described in terms of pyproject.toml. But it will be a list of PEP 508 dependencies.

The PEP will acknowledge that the expectation is that when (or if) a further standard is created which adds a run key to pyproject.toml, that key will match the [run] key defined here - but the responsibility is entirely on whoever proposes that new standard to make sure that’s the case, and to extend the “embedded metadata in scripts” standard to include any additional [run] fields that make sense. In other words, we don’t try to anticipate what might happen in the future, as part of this PEP.

We can have a non-normative section that discusses how this relates to any future directions pyproject.toml might take, but it’ll be explicitly just speculation, at this point.

Is that what you intend?

By the way, whether it’s for this unified proposal, or an independent PEP 723, we should confirm if pipx and pip-run support using embedded TOML. If the existing tools that implement this functionality won’t support the standard, then that’s a serious point against it. I didn’t worry too much about this for PEP 722, because (a) the syntax was so close to what they already have, and (b) I was willing to personally do the work of creating PRs for the PEP support. For a TOML approach, I’m not willing to do the implementation work, so we do need to confirm that it will happen somehow.


  1. it’s happened too often before… ↩︎

3 Likes

The pipx code doesn’t look too complicated. I can’t commit to submitting PRs very soon (as in: probably not before a few weeks, as I will be away at the end of the summer), but I’ll be happy to change it to parse TOML if that helps.

1 Like

I’m slightly concerned that there will be some conflict of opinion around the block delimiter, which might be best to get out of the way early.

# ```pyproject.toml
# [run]
# ...
# ```

Might not feel okay to everyone. The tentative proposal here is that the contents are a TOML block with a [run] key which has been setup to line up with a later expansion of pyproject.toml, but doesn’t match the currently specced contents of that file.
So calling it “pyproject.toml data” is false today but hopefully true tomorrow.

Before this soaks up a bunch of people’s time writing a shared proposal, is that okay? Is there a better option?

1 Like

I agree that it shouldn’t say pyproject.toml in the block delimiter if this is the approach we end up with. Aside from the fact that pyproject.toml doesn’t already support a run table, in the future where run tables are supported, we might also have support for TOML files with other names (one of the ideas bounced around in the previous “projects not intended to create a wheel” thread)!

1 Like

I am in agreement if the initial version also defines support for requires-python.

I understand folks seem to be coalescing on a comment, however I feel the need to say I spoke to one of the people that I mentioned in the PEP over the weekend who is already using the variable approach and while their response in support of that is rather lengthy I can post an exerpt:

[…] My first serious iteration, then, was having the module docstring look like this:

"""pyproject.toml
[project]
...
"""

…which pleasantly looks like a code fence. And I thought, you could also allow it as a top-level string literal immediately following the module docstring. But in the end, even though I still think this variant looks nicest, I went with __pyproject__ because it seemed more explicit and unambiguous.

I actually really like that idea.

I think the important thing for people is to have a marker that is easily searchable (“Python + X”), and for tools to have something unambiguously specified. I think anything that ended up including [run.dependencies] would match both of those – I’m not sure we’d need a ‘pyproject’ landmark?

For the record, I’d be opposed to anything that reccomended three backticks (```), as it embeds concepts from another markup language into the proposal. I’d be similarly opposed to suggestions to use reStructuredText and “::”!

A

3 Likes

To be pedantic, it wouldn’t be that long/noticeable of a table header since dependencies is an array so it would look like:

[run]
dependencies = [
  "...",
]

I think I agree but we need to in some way signify that this section controls aspects of runtime behavior. Do you have an alternative suggestion?

It’s not just submitting a new PR, it’s getting the project to agree to the change. Luckily, in the case of pipx, the current code isn’t in a release, so there’s no major backward compatibility issue.

With pip-run, though, there’s already two different formats supported in production, so you’d need to agree with them how to handle the migration, and how to deprecate the old format(s) - presumably the goal would be for them to support the new PEP as the primary approach. (I know you didn’t offer to do the work on pip-run, but someone would need to). As a starting point, it would be worth getting @jaraco’s views here.

-1 on anything referring to pyproject.toml. At best this data will be the same as (part of) the data stored in pyproject.toml. But this is a distinct data store, and should be represented as such.

But I agree, this needs sorting out. We shouldn’t start assuming there’s going to be a unified proposal only to discover that the authors can’t agree on the format. I think my only constraints are that it must be in a comment, and it must not refer to pyproject.toml. It’s possible that I may find I have reservations over other points, but I can’t think what they would be in advance.

I know this may sound demanding on my part, but please remember I’ve already conceded on a significant issue, namely the use of TOML. And that’s in spite of the fact that no-one has actually provided a convincing response to the arguments I put in PEP 722 against it. So I feel I have the right to draw the line somewhere. After all, if we end up not comfortable with a compromise, we can still go back to two proposals. I’m trying to avoid that, but there are limits beyond which I’m not willing to go.

Please clarify what you expect tools to do with it, and confirm that pipx and pip-run are able and willing to support it. I’m not comfortable dictating behaviour (even in the form of a “SHOULD” requirement) that the two existing tools that implement this won’t support.

This discussion originally started as purely an exercise in standardising existing behaviour. In that form, it’s a very different process than defining new behaviour that we expect tools to implement. And I don’t want to do the work of getting buy-in for new behaviours, but nor am I willing to put my name to a proposal that defines behaviours without getting that buy-in.

If you prefer not to go with a comment form, please say so and we can go back to having two PEPs. I don’t want to spend any more energy trying to form a compromise if you aren’t willing to accept the data being stored in a comment.

Maybe you’re right and a variable assignment is a good way to go. But you’ve not convinced me, and I sense there’s a lot of others who don’t like the idea either. But if you want to stick with it, it’ll be Brett’s decision, not mine.

To be extra-pedantic, it needn’t. It could look like

run.dependencies = ["..."]

That was one of my objections to TOML - there’s lots of ways of saying the same thing, and for the target audience (at least the audience I’m targeting) that’s a source of unwanted complexity.

7 Likes

I’m not sure how I can do that other than by inviting the pipx maintainers to comment on this thread.

“Tools MUST error if they cannot provide a Python that satisfies the version specifier.” I don’t think those tools should be the criterion because we are not even implementing what they do but rather are going with a structured data format now. I’m going to implement this joint PEP in Hatch if/when accepted and it would be a bummer if tools were limited by other tools. I cannot think of a good reason to do so, unless you are of the opinion that Python version management is not a useful feature for folks.

I was just offering another idea because the person didn’t have time to comment here themselves. In any case, I think we need to really think about the delimiter that we use to mark this section as Stephen and Adam said.

It would be great if we could receive user feedback somehow like @brettcannon mentioned to find out what is preferable (for example I also dislike triple backticks).

Is this something we could do in a dedicated discussion?

Maybe even a dedicated PEP? The suggested way to embed metadata in a Python file. THEN we can start talking about how to populate the thing we just agreed on

1 Like

How will that work for comments (above and inline) Over in 722 discussion they seemed highly desired.

As a casual user, I’d understand that if I wanted to convert from a comment-based meta-data to formal TOML, I’d need to convert:

# [run.dependencies]
# rich    # comment
# requests=1.0

To the more structured

[run.dependencies] = […

I think the same rules work? The initial # designates the block and additional # designate a comment inside of the metadata block.

That could be using surrounding whitespace as in 722, or pass everything from col 2 onwards to a toml parser. Is there some additional parsing issue with this?

I wouldn’t try for another PEP. 722 is already specifying a format for embedding a very strict and limited kind of metadata in a python file.
So if you phrase it as a PEP, you’ll likely end up rehashing a lot of the same ground from the PEP 722 discussion – and I think you’re less likely to arrive at a conclusion.

But I would be amenable to talking about an alternative thread for how to embed TOML – and only TOML – in comments.

I’m not that interested in discussing TOML in strings because

  • those strings can already be disambiguated via assignment to special names
  • I’m still not convinced that using strings is a good idea (but there hasn’t been a huge amount of time to respond in detail to the concerns raised)

That is invalid TOML, it would be:

# [run]
# dependencies = [
#   "rich",
#   "requests=1.0",
# ]

I’m not personally interested in a PEP that won’t be supported by pipx and pip-run. So yes, it is one of my criteria, even if it’s not one of yours. I’m very much more interested in supporting interoperability for existing tools than in creating a standard that is simply a spec for a capability that hatch wants to implement.

IMO, having a standard that mandates a capability that hatch wants to implement is precisely a matter of “tools limiting other tools” - in this case, hatch’s insistence that a capability it wants to implement must be backed by a standard, limiting other tools’ choice over whether to implement a Python version check.

Conversely, not mandating support of python-requires doesn’t limit any tools. Hatch can implement such support for itself, as a non-standard extension. If you want, we can state that top-level tables named x-* will never be defined by a standard, so hatch can use x-python-requires without any fear of its implementation being broken by future standards.

This may be arguing over nothing, of course. If pipx and pip-run are happy with supporting python-requires, it’s all irrelevant. We won’t know unless someone asks them.

TOML has comments. If you didn’t know that, then that’s a point against the argument that “people won’t have any problem understanding TOML”, I guess :slight_smile:

# This is a normal Python comment.
# ***some as yet undefined marker indicating embedded TOML follows***
# [run]
# # You can put comments in TOML like this
# # Note that I'm assuming the way of extracting the TOML is
# # "remove an initial hash (or maybe hash-space?) from the start of each line".
# dependencies = [
#     "rich", # or this
#     "numpy"
# ]

# Presumably the comment block ends with a non-comment line, just like PEP 722.
6 Likes

@pf_moore If you don’t mind I would be extremely interested in you commenting on how you envision beginners using this PEP in relation to Python version management.

1 Like