@brettcannon I have many work obligations this week but if you were to allow me one more week I can adjust to the new [run] proposal inside a comment. Otherwise, the current state pretty much represents the final proposal for the __pyproject__ variable approach.
2 Likes
Could we do [run] in this proposal and defer [tool] to a subsequent PEP? Because if we could do that I think weāre very close to something that we both can agree on, which in turn would mean that maybe I can help out with putting together the merged proposal?
But to be clear, Iām fine with waiting until youāre ready, so I hope it works for @brettcannon. But as a heads up, I will be away next week, so it sounds like the next time weāll both be available at more or less the same time will be around 28th August. Iām not sure how much that matters.
Edit: Given that there really isnāt any rush to get a decision on this, maybe (if it suits @brettcannon) it would be better to simply say that weāll wait until September before looking for a decision?
3 Likes
Iām fine with that, especially since it seems like [run] will not just be for scripts but a standard way to specify the runtime requirements of a āprojectā inside any pyproject.toml.
3 Likes
Excellent. But Iām concerned we might be misunderstanding each other[1], so to be 100% clear, what I think weāre agreeing on is basically a version of PEP 722 (not 723) which uses TOML in a comment rather than the custom āstructured commentā format. The only defined key will be run.dependencies, and it wonāt be described in terms of pyproject.toml. But it will be a list of PEP 508 dependencies.
The PEP will acknowledge that the expectation is that when (or if) a further standard is created which adds a run key to pyproject.toml, that key will match the [run] key defined here - but the responsibility is entirely on whoever proposes that new standard to make sure thatās the case, and to extend the āembedded metadata in scriptsā standard to include any additional [run] fields that make sense. In other words, we donāt try to anticipate what might happen in the future, as part of this PEP.
We can have a non-normative section that discusses how this relates to any future directions pyproject.toml might take, but itāll be explicitly just speculation, at this point.
Is that what you intend?
By the way, whether itās for this unified proposal, or an independent PEP 723, we should confirm if pipx and pip-run support using embedded TOML. If the existing tools that implement this functionality wonāt support the standard, then thatās a serious point against it. I didnāt worry too much about this for PEP 722, because (a) the syntax was so close to what they already have, and (b) I was willing to personally do the work of creating PRs for the PEP support. For a TOML approach, Iām not willing to do the implementation work, so we do need to confirm that it will happen somehow.
itās happened too often beforeā¦ ā©ļø
3 Likes
The pipx code doesnāt look too complicated. I canāt commit to submitting PRs very soon (as in: probably not before a few weeks, as I will be away at the end of the summer), but Iāll be happy to change it to parse TOML if that helps.
1 Like
Iām slightly concerned that there will be some conflict of opinion around the block delimiter, which might be best to get out of the way early.
# ```pyproject.toml
# [run]
# ...
# ```
Might not feel okay to everyone. The tentative proposal here is that the contents are a TOML block with a [run] key which has been setup to line up with a later expansion of pyproject.toml, but doesnāt match the currently specced contents of that file.
So calling it āpyproject.toml dataā is false today but hopefully true tomorrow.
Before this soaks up a bunch of peopleās time writing a shared proposal, is that okay? Is there a better option?
1 Like
I agree that it shouldnāt say pyproject.toml in the block delimiter if this is the approach we end up with. Aside from the fact that pyproject.toml doesnāt already support a run table, in the future where run tables are supported, we might also have support for TOML files with other names (one of the ideas bounced around in the previous āprojects not intended to create a wheelā thread)!
1 Like
I am in agreement if the initial version also defines support for requires-python.
I understand folks seem to be coalescing on a comment, however I feel the need to say I spoke to one of the people that I mentioned in the PEP over the weekend who is already using the variable approach and while their response in support of that is rather lengthy I can post an exerpt:
[ā¦] My first serious iteration, then, was having the module docstring look like this:
"""pyproject.toml [project] ... """ā¦which pleasantly looks like a code fence. And I thought, you could also allow it as a top-level string literal immediately following the module docstring. But in the end, even though I still think this variant looks nicest, I went with
__pyproject__because it seemed more explicit and unambiguous.
I actually really like that idea.
I think the important thing for people is to have a marker that is easily searchable (āPython + Xā), and for tools to have something unambiguously specified. I think anything that ended up including [run.dependencies] would match both of those ā Iām not sure weād need a āpyprojectā landmark?
For the record, Iād be opposed to anything that reccomended three backticks (```), as it embeds concepts from another markup language into the proposal. Iād be similarly opposed to suggestions to use reStructuredText and ā::ā!
A
3 Likes
To be pedantic, it wouldnāt be that long/noticeable of a table header since dependencies is an array so it would look like:
[run]
dependencies = [
"...",
]
I think I agree but we need to in some way signify that this section controls aspects of runtime behavior. Do you have an alternative suggestion?
Itās not just submitting a new PR, itās getting the project to agree to the change. Luckily, in the case of pipx, the current code isnāt in a release, so thereās no major backward compatibility issue.
With pip-run, though, thereās already two different formats supported in production, so youād need to agree with them how to handle the migration, and how to deprecate the old format(s) - presumably the goal would be for them to support the new PEP as the primary approach. (I know you didnāt offer to do the work on pip-run, but someone would need to). As a starting point, it would be worth getting @jaracoās views here.
-1 on anything referring to pyproject.toml. At best this data will be the same as (part of) the data stored in pyproject.toml. But this is a distinct data store, and should be represented as such.
But I agree, this needs sorting out. We shouldnāt start assuming thereās going to be a unified proposal only to discover that the authors canāt agree on the format. I think my only constraints are that it must be in a comment, and it must not refer to pyproject.toml. Itās possible that I may find I have reservations over other points, but I canāt think what they would be in advance.
I know this may sound demanding on my part, but please remember Iāve already conceded on a significant issue, namely the use of TOML. And thatās in spite of the fact that no-one has actually provided a convincing response to the arguments I put in PEP 722 against it. So I feel I have the right to draw the line somewhere. After all, if we end up not comfortable with a compromise, we can still go back to two proposals. Iām trying to avoid that, but there are limits beyond which Iām not willing to go.
Please clarify what you expect tools to do with it, and confirm that pipx and pip-run are able and willing to support it. Iām not comfortable dictating behaviour (even in the form of a āSHOULDā requirement) that the two existing tools that implement this wonāt support.
This discussion originally started as purely an exercise in standardising existing behaviour. In that form, itās a very different process than defining new behaviour that we expect tools to implement. And I donāt want to do the work of getting buy-in for new behaviours, but nor am I willing to put my name to a proposal that defines behaviours without getting that buy-in.
If you prefer not to go with a comment form, please say so and we can go back to having two PEPs. I donāt want to spend any more energy trying to form a compromise if you arenāt willing to accept the data being stored in a comment.
Maybe youāre right and a variable assignment is a good way to go. But youāve not convinced me, and I sense thereās a lot of others who donāt like the idea either. But if you want to stick with it, itāll be Brettās decision, not mine.
To be extra-pedantic, it neednāt. It could look like
run.dependencies = ["..."]
That was one of my objections to TOML - thereās lots of ways of saying the same thing, and for the target audience (at least the audience Iām targeting) thatās a source of unwanted complexity.
7 Likes
Iām not sure how I can do that other than by inviting the pipx maintainers to comment on this thread.
āTools MUST error if they cannot provide a Python that satisfies the version specifier.ā I donāt think those tools should be the criterion because we are not even implementing what they do but rather are going with a structured data format now. Iām going to implement this joint PEP in Hatch if/when accepted and it would be a bummer if tools were limited by other tools. I cannot think of a good reason to do so, unless you are of the opinion that Python version management is not a useful feature for folks.
I was just offering another idea because the person didnāt have time to comment here themselves. In any case, I think we need to really think about the delimiter that we use to mark this section as Stephen and Adam said.
It would be great if we could receive user feedback somehow like @brettcannon mentioned to find out what is preferable (for example I also dislike triple backticks).
Is this something we could do in a dedicated discussion?
Maybe even a dedicated PEP? The suggested way to embed metadata in a Python file. THEN we can start talking about how to populate the thing we just agreed on
1 Like
How will that work for comments (above and inline) Over in 722 discussion they seemed highly desired.
As a casual user, Iād understand that if I wanted to convert from a comment-based meta-data to formal TOML, Iād need to convert:
# [run.dependencies]
# rich # comment
# requests=1.0
To the more structured
[run.dependencies] = [ā¦
I think the same rules work? The initial # designates the block and additional # designate a comment inside of the metadata block.
That could be using surrounding whitespace as in 722, or pass everything from col 2 onwards to a toml parser. Is there some additional parsing issue with this?
I wouldnāt try for another PEP. 722 is already specifying a format for embedding a very strict and limited kind of metadata in a python file.
So if you phrase it as a PEP, youāll likely end up rehashing a lot of the same ground from the PEP 722 discussion ā and I think youāre less likely to arrive at a conclusion.
But I would be amenable to talking about an alternative thread for how to embed TOML ā and only TOML ā in comments.
Iām not that interested in discussing TOML in strings because
- those strings can already be disambiguated via assignment to special names
- Iām still not convinced that using strings is a good idea (but there hasnāt been a huge amount of time to respond in detail to the concerns raised)
That is invalid TOML, it would be:
# [run]
# dependencies = [
# "rich",
# "requests=1.0",
# ]
Iām not personally interested in a PEP that wonāt be supported by pipx and pip-run. So yes, it is one of my criteria, even if itās not one of yours. Iām very much more interested in supporting interoperability for existing tools than in creating a standard that is simply a spec for a capability that hatch wants to implement.
IMO, having a standard that mandates a capability that hatch wants to implement is precisely a matter of ātools limiting other toolsā - in this case, hatchās insistence that a capability it wants to implement must be backed by a standard, limiting other toolsā choice over whether to implement a Python version check.
Conversely, not mandating support of python-requires doesnāt limit any tools. Hatch can implement such support for itself, as a non-standard extension. If you want, we can state that top-level tables named x-* will never be defined by a standard, so hatch can use x-python-requires without any fear of its implementation being broken by future standards.
This may be arguing over nothing, of course. If pipx and pip-run are happy with supporting python-requires, itās all irrelevant. We wonāt know unless someone asks them.
TOML has comments. If you didnāt know that, then thatās a point against the argument that āpeople wonāt have any problem understanding TOMLā, I guess 
# This is a normal Python comment.
# ***some as yet undefined marker indicating embedded TOML follows***
# [run]
# # You can put comments in TOML like this
# # Note that I'm assuming the way of extracting the TOML is
# # "remove an initial hash (or maybe hash-space?) from the start of each line".
# dependencies = [
# "rich", # or this
# "numpy"
# ]
# Presumably the comment block ends with a non-comment line, just like PEP 722.
6 Likes
@pf_moore If you donāt mind I would be extremely interested in you commenting on how you envision beginners using this PEP in relation to Python version management.
1 Like