On which note, I should ask - is anyone willing to write a PEP that just picks a version and makes it official? I’ll go on record as saying that I’ll be happy to expedite its approval, as long as it doesn’t get into more controversial areas like requiring tools to validate the version.
If a PEP does get written, I’d hope that it standardises the most recently released version of TOML at the time of writing (i.e. 1.1), and all tools and the Stdlib can update to meet the spec.
My understanding was the inverse: We reference a spec (TOML, JSON, URL, HTTP, Unicode, you name it), so that we aren’t bound to one specific implementation. For practical reasons we can’t e.g. require a compression format not in std, but otherwise the goal is that different implementations, across different Python versions, implementation of the spec and across programming language can communicate with each other.
To me, when we’re referencing living specs (or using them without further qualifying, such as URLs, HTML5, Unicode and HTTP) we’re doing that so we can pull in their evolution and their tools by reference, without specifying and being stuck on a “python fork” of those specs. There’s deviations sometimes, and sometimes updates come at different speeds. This does mean that you need to update your tool to get full support for projects that use newer versions, but I think it’s fine that you need to use a somewhat recent version of a packaging tool. I’m aware that this is a specifically difficult problem with pip being shipped with Python and the [build-system] table exposing TOML from dependencies, but I’d be even more worried if we blocked both python packaging and stdlib from evolving.
On a more practical note, to solve the [build-system] problem, we can do the same trick that Cargo has been successfully using for years (Rewrite Cargo.toml when packaging crates by alexcrichton · Pull Request #4030 · rust-lang/cargo · GitHub - @davidism to answer your question: feat(toml): TOML 1.1 parse support by epage · Pull Request #16415 · rust-lang/cargo · GitHub): Build backends rewrite pyproject.toml into TOML 1.0 format [1] and add a pyproject.toml.orig to the source dist. This way, using TOML 1.1 in a project doesn’t affect other project that use it as a registry dependency.
Disclaimer: I haven’t actually checked what TOML 1.1 implementation currently output ↩︎
4 Likes
Thanks for picking me up on the python-centric nature of my comment. You’re right. What I should have said that was for practical reasons, particularly around bootstrapping, in my head I’ve always linked the version to “whatever the stdlib supplies”. But that’s as a maintainer of a tool written in Python.
From a standards POV, I consider referring to a standard without a version as meaning tools can choose what version to support. That results in practical issues, as we’re seeing, and is generally not a good idea, but when it happens we’re making support of later versions a tool-specific choice, and therefore making “choose the lowest version that’s in common use” as the baseline.
At this point I’ll repeat my original position - I think that people are overstating the risks of leaving this to be a choice for individual tools. If we simply used whatever is most convenient for each tool[1] I expect it would be fine.
Would this work for Python? When building from a local directory, pip, build or uv read pyproject.toml to know which build backend to use. That happens before the build backend does any rewriting, and there’s no way to just read the [build-system] section so that TOML 1.1 in other sections gets ignored. So the implication is that pip, build and uv (and any other build tools) have to always support the latest TOML version.
Yes, this is a rare case. But my point is that I expect all of this to be rare, so if we aren’t willing to cover all of the rare cases, why just cover some?
And to head off the “obvious” question, I’m not comfortable with pip (I can’t speak for uv or build, or for build backends) rewriting pyproject.toml - that’s not something I want pip to get into.
Which I admit is a little weird for pip - TOML 1.1 on all versions of Python except 3.11-3.14 ↩︎
This workaround handles registry-based workflows such as pip install foo, where foo transitively depends on bar and bar has a new release that uses TOML 1.1 syntax. This goal is to reduce the impact of adopting a higher TOML version to only developers on the project and people who manually include the project by path/http/git/etc., which are usually developing on or with the project themselves. We reduce the group of people who need support for the latest TOML version from everyone to only people who use a non-published version of the package.
But we don’t reduce the group of tools that need 1.1 support, do we? Build backends must support 1.1, to do the rewriting. Pip, uv and build must support 1.1, to pick the right backend to build with. We can’t say “use a version of pip that supports 1.1 when building” if 1.1 support is based on the Python version (which it would be for pip, because of stdlib support).
So what tools get to avoid having to add 1.1 support in this scenario? Have I missed something obvious?
Yes, by rewriting we avoid asking all users to upgrade to the lastest (1.1 supporting) version of their tools. But that’s not the problem I’m concerned about here - certainly for pip, we don’t really have an issue getting people to use the latest version. And most build backends aren’t upper-limited in pyproject.toml, so they get the latest version by default.
Sorry if I haven’t explained this clearly, the idea is that while the the major tools need to update to TOML 1.1, frontends and backends alike, it avoids breaking your users who use e.g. an old pip version. When the old pip version builds the source distribution of a project that uses TOML 1.1, it gets the re-written version. That means only the author of the library and their tools need to update, but we don’t apply the same requirement to all users too. Afaik that’s also the scope of that guarantee that Cargo makes, old versions of Cargo should succeed and fail under the same conditions as old versions of pip do (but I can ask the cargo maintainers if we need more details).
I think the distinction is that cargo does this when publishing a package, which is much more centralized in the Rust ecosystem. So your local pip might need to support TOML 1.1 (i.e. you updated to the newest version) but users don’t need to.
The equivalent would probably be for warehouse to do the conversion work when it receives a package.
In the Python ecosystem I think the only time it might be sensible rewrite a pyproject.toml is when a user is building their project locally. I don’t think it makes sense to intercept what is inside an sdist and rewrite it.
For building a local project I think two classes of tools could choose to rewrite the pyproject.toml:
- Project management tools (e.g. uv, Poetry, etc.) that could rewrite before passing to a build backend that doesn’t support 1.1 specific syntax
- Linting tools (e.g. ruff, flake8, etc.) that could alert users they are using TOML 1.1 and possibly auto fix.
Speaking of which, for those who are concerned users will starts accidentally writing TOML 1.1 specific syntax, why not request popular linting tools add a rule for this? I’m sure, with some advocacy, it could become best practice for projects to adopt a linting rule for this.
2 Likes
Is this more than just a parsing strategy? Is the machine-written Toml 1.0 file ever published, or is it for internal use only?
If I understood the changelog correctly: a TOML 1.1 parser will be able to parse a file that was written according to TOML 1.0 without a problem, correct?
So, my suggestion is that for now we keep the standards using 1.0.
Meanwhile, it would make sense to work on upgrading the stdlib implementation of the TOML parser to handle version 1.1[1]
Once that is implemented in the stdlib, and the last version of Python that only supports 1.0 reaches EoS, we could update the specs to use 1.1, according to some pre-agreed deprecation policy.
This means that 1.1 parser will be able to read old published distributions, and new distributions will only be published in a context that they are expected to be used with a certain up-to-date version of Python.
I do understand that the standard not necessarily needs to be coupled with the implementation, but the standards are also a political agreement among the members of the community and the past discussions about adopting
tomllibinto the stdlib seem to show that there is appetite for waiting until Python’s standard library implements the parsing before committing to support it. ↩︎
6 Likes
I was thinking about build backends doing the rewriting, though it’s a roughly auditwheel-shaped problem, so it could also be done as a shared post-processing step by build frontends. But I fear distracting from the main discussion here.
1 Like
Well, for one thing, I don’t want users needing yet another tool, but also I think 1.1 should be adopted. I’d want whatever we go with to allow users to do so as It’s better for both git diffs and human review with the the relaxed rules for inlined tables.
I’m less concerned that users will accidentally start writing it, than the ecosystem not having clear rules for what happens when they do write it, intentional or not.
That is the sort of policy I think we should adopt. A new standard sets the minimum version to 1.0, and specifies how we can increment the minimum version only after the all supported CPython versions support the new minimum.
But I also think we should encourage users to be conservative in how quickly they adopt new versions, since this is all about maximizing compatibility within reason.
And I don’t think setting that kind of policy is exactly the same as specifying what we have today as standard. To me, establishing a process for updating the minimum version is it’s own potentially fraught discussion about how many hours after a Python version goes EOL we need to wait – and who it is that is even doing the waiting!
I’m happy to do it, but I probably won’t have time to start on it for a few days.
I think that it’s got to have a little more meat on it than strictly making TOML 1.0 the official version. Otherwise, it could make things worse by requiring too-strict behavior. We need to make enough room to say
- 1.0 is the minimum which tools MUST support reading
- tools may, at their discretion, support newer versions
- tools may, at their discretion, detect and warn or error on newer versions
- future specifications may raise the floor, requiring new versions as the new minimum (does this even need to be specified? I think so)
- users may use newer versions as they please, but doing so may result in spec-compliant tools failing to parse their data
Does that seem acceptable for a truly minimal update to just make the current tool behaviors – and likely behavior as new Python versions ship – documented and specified?
3 Likes
Other than this line, it seems fine.
I’m fine with the “at their discretion part”, but I think newer versions needs to be constrained to “newer versions for which files written as 1.0 are still read as containing the same content” and
“any file written by a newer version should either parse as containing the same content or fail to parse by older versions back to 1.0”
Effectively, we shouldn’t allow “at their descretion” if toml ever has a breaking 2.0, but lay out the specific properties we care about in our ecosystem.
1 Like
I think I agree with this, but I find it hard to follow in that wording (maybe it’s just too terse for me without an example or two). But at the very least I agree with the ethos of being unambiguous about what should happen with future versions.
I’ll try to incorporate this when I do a first draft, and if it comes out wrongly we can address the wording at that point.
All of this looks fine.
I agree with @mikeshardmind on the question of what newer versions can be supported.
The way I’d put it is “tools may, at their discretion, support newer versions as long as those versions are a superset of 1.0”. With an explicit note that if TOML ever releases a backward-incompatible version, a new PEP will be needed to define how the packaging ecosystem handles the transition.
I think that a PEP like this will still leave some difficult questions for tools to answer[1], but I think it’s right that we explicitly leave those decisions to individual tools.
For example, pip will need to decide what to do over the range of Python versions where the stdlib only supports TOML 1.0 ↩︎
2 Likes
My main thought is that the fact that we have to have this long discussion suggests that we need to be careful to be very explicit in future specifications. (I heard somewhere that explicit is better than implicit.)
I also think the anti-Postel’s law approach is the safest: the specs should strictly limit compliance to a particular version and say that all tools “MUST” error on any input that does not conform to the specified TOML version(s) and “MUST NOT” produce any consumable output[1] that doesn’t conform to the specified TOML version(s). (“Versions” plural because then the way to upgrade in the future is to change “all tools MUST error on anything that isn’t valid TOML 1.0” to “all tools MUST error on anything that isn’t valid TOML 1.0 or 1.1”, etc.) To do otherwise introduces amibiguity and allows tools to diverge in what they support, which defeats the purpose of interop. (I heard somewhere that in the face of ambiguity we should refuse the temptation to guess.) A possible alternative is to use some kind of in-file version declaration that producers are required to write and consumers are required to read.
that is, files that other tools are expected to read ↩︎
2 Likes
The dangers of postel’s law shouldn’t apply in this case with respect to toml version, so long as we agree what a breaking change to how we use it would entail, and that that is the line of what isn’t allowed implicitly on upgrading.
There’s still an agreed upon actual standard here, we’re not saying “Accept garbage not defined by toml”, we’re leaving it to tools to update as they see fit within a well-defined standard so long as it remains unambiguously forward compatible.
1 Like
FWIW, I have made a request to the ruff project for such a linting rule: Lint rule to detect TOML 1.1 specific syntax for `pyproject.toml`, `pylock.toml`, or any other Python packaging TOML file · Issue #22588 · astral-sh/ruff · GitHub
While I not against such a PEP, my main question is, how does this improve compatibility between packaging tools beyond being explicit about something that is de facto true.
Isn’t TOML 1.0 already the effective minimum version packaging tools use today? And isn’t it expected there will never be a backwards incompatible version of TOML?
If the authors find the cost of writing a PEP to be worth it though I’m happy to see more explicit language.
2 Likes