No, it argues that we made a mistake not defining some form of versioning for the [project] table. We keep making this mistake, and we really should stop doing so I’ll take the blame for that, as PEP-delegate I should probably be insisting that any new data format addresses versioning when it’s defined, not later.
But the mistake was made, and we need to deal with the consequences, not stop making changes because of it. Maybe “accept that some things will break” is an acceptable way of dealing with the consequences, or maybe we need something more. I’m honestly not sure.
I would hope so, yes. I’m not clear why anyone would think otherwise, but I’m happy for the spec to be explicit, because if we don’t Murphy’s law says that some backend would choose otherwise.
No, it argues that we made a mistake not defining some form of
versioning for the [project] table. We keep making this
mistake, and we really should stop doing so I’ll take the blame for that, as
PEP-delegate I should probably be insisting that any new data
format addresses versioning when it’s defined, not later.
During the discussion about “not referring to packaging PEPs” when I
asked (as a maintainer of a build backend) for a versioning scheme
for pyproject.toml, because we’re being strongly discouraged from
referring to the PEPs as specifications when we document what we’ve
developed compliance with, I was told the pyproject.toml format is
intentionally unversioned and that ever needing to have versions for
it would be a failure of its design. Has that position changed?
I’ll reiterate that it would be nice when projects that try to
comply with a standard like pyproject.toml (I’m trying very hard not
to mention a PEP) add support for things that come in a particular
revision of the standard, that they be able to accurately discuss
and document what they support without having to enumerate every
last feature covered by that specification. I’ve fallen back on
saying the “pyproject.toml standard as of 2023” in order to indicate
that support for features added to pyproject.toml in 2024 aren’t
implemented yet, for example. Doing that is really unwieldy, and it
seems like having an actual version I can refer to would be much
cleaner, but I’m sure there’s a reason the designers were adamant
that should never be needed.
If we had versioning on the table, and we were proposing the introduction of a new version (e.g. 1.0 → 1.1), we might hear some flavor of the same argument. That introducing a new version will break some tool which doesn’t support v1.1. That’s the context in which I read an argument against adding some new syntax simply because an older tool will break upon seeing it.
Perhaps that was an unfairly ungenerous reading by me of @EpicWink’s comment though. Sorry for that!
I’m not convinced – accepting the lack of a version as a simple fact of life for the purposes of this discussion – that we shouldn’t expand the definition of project.dependencies to allow it to refer back to Dependency Groups.
My intention in making that statement was that the dependency list is expanded in PKG-INFO. But I’m not sure that backends are required to populate Requires-Dist fully. Perhaps this would need to be included as an expectation.
I very much don’t like the idea of requiring that backends edit pyproject.toml before including it in an sdist. My overarching goal is to ensure that the Dependency Groups of a package are never considered part of its public interface either when being installed or once installed.
I need to think about the implications here a bit more, in terms of what I would expect from a built sdist (I have very clear expectations about the wheel, less so about the sdist).
This is what I’ve been considering, but I’m a bit uncomfortable with the resulting state for package authors who have some legitimate need for dynamic dependencies.
They’re left unable to integrate with Dependency Groups because of the requirement that the data is static.
If we allow the inclusion to run in the other direction, then there’s no such conflict. So I see it as a better place to end up, even though it might be more difficult to get there.
I generally agree that it would be nice to support inclusion of project.dependencies in a dependency group though.
I don’t think this is a complete argument in its own right. Sometimes you try to do this and find a 2000 LOC setup.py file, after all.
Truthfully, a part of me wants includes to work in both directions, with the constraint that the data has to be static for inclusion in a Dependency Group (but not vice-versa; a dynamic dependency list could include a Dependency Group).
That offers users the choice of how they want to structure these data, allows us to include something today (inclusions of static dependency lists in groups) and lets users potentially restructure to reverse the relationship in the future if they so choose.
Probably this is a sign that my thinking on this subject is not sufficiently mature yet. I’m certainly not ready to make any change to the PEP to mention inclusions from or of [project.dependencies].
I understand that the topic has arisen naturally here, but can we split any discussion of versioning [project] into a separate thread? I think that will be helpful both in order to keep this thread focused on Dependency Groups and to enable more people to engage on the versioning topic.
But equally, you’re right, the new syntax in pyproject.toml will exist in sdists on PyPI, and therefore when that sdist is used to install (because there’s no suitable wheel available, for example) the install process will need to be able to build a wheel from source that uses the new syntax.
However, you would also need to make sure that it’s clear that the dependency group won’t support any runtime dependencies needed to support editable installs (an editable install can “inject” additional dependencies even if the dependencies are declared static). This should be a minor point that will be irrelevant 99.9% of the time, but it’s an illustration of the sorts of edge cases that could be missed if we try to (in effect) write a rule that says “inclusion of runtime dependencies is only supported when it works the way we expect”.
Hardly more private, it’s still in the pyproject.toml that’s available exactly as you describe. Yes, there’s one more step to get to the data, but it’s not hard, and it’s certainly no less accessible than the existing situation.
Do you remember who told you that? I don’t think it would have been me, but it’s possible I got influenced by something in the discussions at the time that I no longer recall. It’s not (as far as I can see) stated in the PEP itself, though.
If it was me, then all I can say is that I’m sorry, and yes, my position has changed.
If it wasn’t me, then my position is that not making it versioned was a mistake, and I missed the fact that this wasn’t discussed in the PEP when I approved it. If this is the situation, I can’t comment for whoever did make that statement (presumably one of the PEP authors?) but I’d be interested to know what the thinking was behind that statement (if they can remember).
But ultimately, there’s not much point in re-hashing the past. We need to learn the lessons, fix things if they need fixing, and move on. I don’t plan on proposing a versioning scheme for pyproject.toml myself, simply because I don’t have the time, but if someone wants to do so, that’s fine.
Oh, absolutely. We’d get a cleaner break (a clear error “this project uses pyproject.toml version 1.1, but we only support version 1.0 and earlier”, for example) but nothing’s going to magically make older tools able to process newer features. So it may be that we can park the question of versioning for this discussion - but we do still need to at least consider how transition would work (and document the transition process in the PEP).
I’d go further, and object to the idea of backends rewriting pyproject.toml in a sdist at all. I view the sdist as being an unchanged copy of files from the original source tree, with additional metadata to allow tools to process the sdist easily.
It was discussed as part of the deliberations on the PEP. At the time we pointed out that the versioning of the core metadata never really did anything since it’s been backwards-compatible. We figured that we would have to be cognizant of any change.
And why versioning is “fun”: it would probably be 2.0 since it’s backwards-incompatible.
OK, but @sirosen needs to be aware that some replies mixed non-version discussions.
Yep! Should I add this thread to the Dependency Groups PEP links, or maybe post back a short summary there to keep it all in one thread?
Unless someone asks for it, I don’t plan to do either.
I don’t think I have any useful opinions to share about versioning the pyproject.toml contents, other than “that’s not something I want to try to tackle in PEP 735, and I doubt anyone else wants that either.”
Isn’t one way to handle that in fact to stop making changes, temporarily, so that the current state of affairs can be documented and frozen as “version 0”? And then stipulate that any pyproject.toml that doesn’t specify any version requirement will always use version 0 and never anything else, and any projects that want to use the fancy new stuff in subsequently-adopted revisions must declare the appropriate version?
So this is one of the topics I had on my master list (well, tree), and it seems like perhaps this one can actually be discussed independently (and act as a loose end to pull on, in order to unravel the larger problem).
What I’m wondering is: can’t we just start doing that? Say we add a top-level version specifier and define version 1.0 of the pyproject.toml spec as what we have now (but with the version specifier), and mandate that tools treat pyproject.tomlwithout a specified version as being a version 0 (that happens to match 1.0 in every way except for the version specifier). From then on, we do semver normally.
I think the only way this breaks anything is if an existing validator is demanding that an unknown value not be there; but I’m pretty sure they aren’t supposed to do that since PEP 518 claims
Tables not specified in this PEP are reserved for future use by other PEPs.
(If we’re pedantic and require that new top-level names are tables in order to qualify, that could be worked around by defining a [meta] table or something.)
Or else, what am I missing?
(I guess this is what @BrenBarn is saying, but also describing the path forward from there explicitly.)
This seems inconsistent with the above phrasing. The possibility of expanding the format was explicitly provided for, and versioning the changes seems like it should be just the default expectation by now.
So basically you’re saying that the versioning problem is so important that everything must stop while we solve it? That seems rather draconian. Adding a new section, for example, is relatively safe, so we’d be stopping important work that could progress otherwise.
And we don’t even know yet if anyone is willing to do the work of addressing the versioning issue. Are you? Will you commit to writing a PEP for this and getting it accepted? In a timely manner, given that you would be holding up all packaging standards work while you do so? I wouldn’t, personally.
I think it’s important to discuss the versioning issue, and be mindful of it in future proposals, but continue working on making the packaging ecosystem better in the meantime.
PEP 621 is pretty clear about static metadata needing to be preferred. Dynamic dependencies should be used only for things that cannot be expressed statically. E.g., a package needs to run code to check which CUDA version the user has installed on their system. This applies to a pretty small set of packages. Who are unlikely to need to refer to dependency groups. It’s not a compelling reason to break backwards compatibility to me - that bar should be a lot higher than a feature like this.
Agreed, it’s a minor thing and not compelling on its own - it was the least important of a list of reasons, ordered from most to least important.
I don’t know to what extent it affects the versioning question (for pyproject.toml or in general) but I think it’s important to realise that we’re now at a stage with packaging standards where we have a significant amount of standardised behaviour, and much of it has been round for a long time. So we have a big user base, and pretty much any change is likely to affect someone. Packaging tools have been in this situation for some time - pip and setuptools, for example, suffer from a significant maintenance burden as a result of this type of backward compatibility requirement. I think we need to acknowledge that our standards are in a very similar situation.
With that in mind, versioning is a component of having a good framework for making changes and improvements, but it’s not by itself enough. We also need things like a backward compatibility policy, and support policies[1].
Maybe these sorts of policy are something that a packaging council could mandate, but I think that as a community we need to discuss how we see our standards evolving over time.
Personally, I think that we can reasonably take a fairly aggressive policy on what versions of tools we expect people to use[2]. So a standards policy of short transition periods, long enough to give tools time to implement the new standard but not much longer, seems OK to me. And I’m OK with principles like “if you use a feature that’s part of a new standard, you choose to restrict yourself and your users to using tools that support that standard, and that’s acceptable”. But I know others might prefer to take a more conservative (and therefore more slow-moving) approach.
Not in terms of tool support, but in terms of new standards - to what extent do we allow new standards to make breaking changes to older ones? At what point do we say we no longer consider supporting users of an older version X of a spec to be a priority? Etc. ↩︎
The pip maintainers only support the current version, for example, and build backends are typically installed based on build dependencies that don’t have upper caps, so they will normally be the latest version available. ↩︎
We could, although it the Simple API acts as prior art than no version is v1 so you can explicitly opt into specifying the version w/o any semantics changing on you.
I was thinking about this last night and I’m currently -1 on adding a version to pyproject.toml.
First, people will forget the key. It’s a UX sticking point we are asking users to do which may not be clear to them as to what to set it to when they do remember. And if you say, “people will set it to what their tools tell them to in the docs”, then who’s benefiting from the versioning if you’re just doing what the docs say, which can include saying what standards are supported (and before you say “Dependabot”, I address that point below)?
Two, it may be too strict of a thing for all tools to have to follow. There’s a reason why prevailing practice w/ packages is to not set an upper-bound: just because there’s possible breakage doesn’t mean there will be. To me, having tools refuse to work w/ the contents of pyproject.toml feels overly restrictive. Plus, when talking about [project] specifically, we have [build-system] acting like an implicit version restriction (I’m addressing Dependabot next, I promise ).
Third, I don’t think it solves the problem people seem concerned about with external tools. Consider the Dependabot scenario. Either Dependabot will work or it won’t in either scenario; the version number is something it might not support (see above as to why that’s qualified), or it’s going to error out. But in both situations the outcome is the same! Now, you might be thinking, “but the error message will be better w/ version numbers”, which might be true. But if Dependabot does enough error checking on its inputs (which all tools taking user input typically do), then the error might not be, “I don’t know how to handle metadata version 2”, and instead be, “I don’t know how to handle {include-group = "runtime"} in project.dependencies”. I personally don’t find that hard to comprehend. It’s like us updating requirement specifiers and running an older version of pip: we aren’t versioning those strings because we assume the error message wolld be clear enough.
+100 to what Brett said. There’s virtually no user scenario that gets better with a version number in a user edited file, and most likely the experience gets worse because the tools get more picky.
If the underlying problem is “users might have specified something that isn’t compatible with a new feature” then make that an error. They can’t be using the new thing without changing their code, and they can’t use the new thing without tools that support it, which means those tools can also understand that some existing feature “A” is restricted when using new feature “B”. Users will find out immediately (assuming they test, which we have to assume they will otherwise we can’t do software engineering).
Yes, it’s a bit harder for us to keep things consistent over a long period of time, but that’s what we signed up for. If we aren’t prepared to do that, we just shouldn’t add the feature and instead have an escape hatch for tools to let it be specified in some non-standard way (which we have, so they can already do it).
Thanks. This seems like a very compelling argument (in particular, the fact that pyproject.toml is user-edited, so adding a version is just reiterating what using the new syntax already implies).
So where that would leave us, I believe, is that we simply need to be more careful to fully discuss (both forward and backward) compatibility and transition when writing PEPs. Something we’ve agreed about before as “good practice”, but which we still seem to get distracted from when discussing the far more interesting questions around design of the actual feature we’re proposing
Well, I actually think something even more radical, which is that there are multiple problems that are that important.
That (allowing adding of new sections) sounds plausible and could maybe be adopted as a policy, sure.
Basically I think that if we want to make the packaging ecosystem better, a good way to start is to not make it worse. As I’ve mentioned before, to me that means adopting a more big-picture perspective with more of a focus on the end goal and less focus on the incremental improvements. (The improvements can still be incremental, it’s just that whether they are improvements and not just changes depends on their progress towards a goal and not just their delta from the status quo.)
This may mean slower progress on proposals such as these and personally I’m fine with that. Essentially I just disagree (not just with you, but with many involved in these packaging discussions) about the urgency of these various packaging proposals. With regard to recent ones (e.g., PEP 722/723, PEP 735, probably even PEP 725) I don’t expect they are going to radically improve the situation for end users, so I just don’t see it as so vital that we move forward with them. This doesn’t mean I think they are bad proposals (although I have my own opinions on each one); I just don’t see any urgent need for them.[1]
This doesn’t mean the situation can’t or won’t improve. Currently there are at least two very popular tools (poetry and conda) which are either working out in front of standards or blazing their own parallel trail, along with a number of others that are sometimes ahead of standards to various degrees[2]. People use these tools because they provide useful functionality, in some cases nonstandardized functionality. Maybe those tools will continue to improve without the standards and then the standards can simply codify whatever is judged to be the best approach (which is not usually an approach I’m excited about but in this situation I’m grasping at straws ).
So even if no one is willing to write the PEP for pyproject.toml versioning, and hence nothing happens, I don’t see that as a problem — or at least, not as any bigger problem than the fragmented state of the Python packaging ecosystem that we have and will continue to have regardless of what happens with PEP 735, PEP 723, etc. But then again, if no one writes that PEP and we move ahead with these PEPs anyway (which I think is basically what you’re suggesting), we’ll still have that problem. So maybe it doesn’t matter. All I’m saying is that I personally am not convinced that the pleasantness of Python packaging in 10 years will depend significantly on whether we do or don’t resolve PEP 735 quickly[3], but I am convinced that it will depend significantly on whether we do or don’t resolve some larger questions that keep getting back-burnered in favor of specific proposals. So whenever I hear “maybe we should pause until we think about this bigger issue” it sounds like good news to me.
So where that would leave us, I believe, is that we simply need to be more careful to fully discuss (both forward and backward) compatibility and transition when writing PEPs. Something we’ve agreed about before as “good practice”, but which we still seem to get distracted from when discussing the far more interesting questions around design of the actual feature we’re proposing
That is a great way of saying it and I agree. I do get the sense in these discussions that people get excited about the design of feature XYZ and about the technical details of how to make it happen. That’s all well and good. But sometimes just slowing down and taking time to look past those engrossing matters can be equally valuable.
There are other PEPs that are somewhat more removed from direct user experience, like PEP 658, which I think are very useful, although ironically that PEP then required a fix in PEP 714, again apparently due to parts of the system getting ahead of others, resulting in a tangle. ↩︎
as evidenced by the discussion of tools implementing provisional PEPs ↩︎
which isn’t to say it won’t be better in one year; the question is whether that improvement will still be visible in the rear-view mirror in 10 years ↩︎
Thank you for encapsulating it so well. This is what I was trying to get at by bringing up so many discussion topics, describing multiple future proposals at once etc. I had a goal in mind - not necessarily the right one, but a goal; and I do feel there needs to be one.
IIUC, the underlying problems that having a version on pyproject.toml would solve are:
that we have no clear way to communicate what is functionality is supported by tooling that uses pyproject.toml (as a reader or writer).
This isn’t really a problem of the file format IMO, rather a symptom of the fact that we’ve got a fragmented tooling ecosystem. I don’t think/know if we can change this at this point even if we wanted to.
that there is no easy way to deal with backwards and forwards compatibility.
This is effectively something that we’re not going to be able to enforce (see above re fragmentation) unless we have a whole bunch of coordination toward shared implementations. That is something we’ve been doing and, as many of us are aware of, it is a non-trivial problem to solve.
As an outside observer, it strikes me that this sort of thing-we-should-keep-in-mind-but-we-oftent-forget is well addressed by task-specific checklists. I.e., there could be a step in the change process that says “Consider these important things before moving forward.”
What would be the real benefit of having versions? I’m guessing the only reason is going to be support new features but this will create the chaos of users using defining higher version than needed or users using a lower version of the required by the features using (so it will require the overheard of throwing errors for some scenarios or people unable to use the pyproject.toml because the maintainer is used to always define the same version for all their project)
From my experience of requesting changes of remove/modify in the python std library I think it’s almost impossible we will see a v2 of the pyproject.toml happening at any point.
I’ll take the blame for that, as PEP-delegate I should probably be insisting that any new data format addresses versioning when it’s defined, not later.
