Would doing what I suggested here be helpful, i.e. release Tomli v2.3.x (the branch that parses (and validates) TOML v1.0.0) on PyPI with a distinct distribution and import name (e.g. tomli-toml-v1.0 importable asimport tomli_toml_v1_0)?
Build backends primarily use tomllib, but can validate with tomli_toml_v1_0(if available). This extra validation should no longer be necessary when the last Python with a TOML v1.0.0 parser goes EOL.
In this scenario, as far as I can tell
tomllib still solves the bootstrap cycle issue (the main reason why it’s in the standard library)
Standard library doesn’t need to support multiple TOML versions
Build backends can validate for TOML 1.0.0 compliancy until TOML 1.1.0 is ubiquitous (EOL date of Python 3.14?)
I doubt it would help - it’s easy to pin to tomli <= 2.3 if that’s what tools want to do.
There’s two independent questions here.
Do we want to update the standards to require tools to validate that TOML 1.1 features are not being used? No-one seems sufficiently motivated to go through the work of putting a PEP together to do that.
In the absence of a requirement in the standard, individual tools need to make their own judgement on whether the risk of users “acidentally” using TOML 1.1 features is sufficient to motivate them to use tomli <= 2.3 in Python 3.15+ (when the stdlib switches to 1.1).
A lot of the discussion here seems to be basically around the pip maintainers[1] discussing their differences on (2) in public. I’m not aware of any other tool maintainers having expressed an opinion - @konstin hinted that uv would prefer a standard change to require TOML 1.1 support, but that would require a PEP and is effectively an alternative to (1) above.
Beyond the immediate issue with TOML 1.1, I’m aware that we’re setting a precedent here for what to do in future if a TOML 1.2 or later were released. Thinking in those terms, I actually wonder what the stdlib position on TOML support should be. At the moment, it seems that the stdlib only supports the latest version of TOML that was available at the time that Python version was released - is that correct? If so, I’m not sure what the compatibility implications are there - it’s not just packaging that could hit this, any tool using Python which supports multiple versions will have the same issues to address.
Sure, but for larger environments, Linux distributions and such, it may be helpful having an import name that guarantees a TOML v1.0.0 parser. This also allows keeping Tomli up-to-date without the <= 2.3 restriction.
In the event of changes to the TOML specification, we can treat minor revisions as bug fixes and update the implementation in place. In the event of major breaking changes, we should preserve support for TOML 1.x.
So it allows updating parser spec version for non-breaking changes such as TOML 1.1, but doesn’t require an update.
Is it set in stone that the standard library tomllib parser can’t have a ‘1.0 only’ option in some form?
I agree with the people saying that if we’re going to specify TOML 1.0 and have that mean anything, then at least some prominent projects need to enforce that - i.e. deliberately reject files using new-in-1.1 features. We don’t need every project parsing the files to do that, but it needs to be clear that minor-tool-X is not just outdated in supporting only 1.0 syntax.
TOML support was added to the standard library because we standardised on it, and what we standardised on was TOML 1.0. It would be really nice to keep the TOML 1.0 parser as an option, including rejecting things that became valid in 1.1.
Playing devil’s advocate here, we didn’t actually specify TOML 1.0. As of this moment, the standards just say “TOML”.
It’s worth also flagging that there are three cases here. Taking them from least impactful to most impactful:
Lockfiles. These are only ever written by tools, so 1.1 features won’t get used unless tools that write lockfiles do so. And there’s no stdlib write support, so I can’t see how that can happen except as a result of a deliberate choice.
Script metadata. These are human-written, but a significant amount of usage is probably unpublished personal scripts. And the user probably doesn’t care that much about portability across tools. So “whatever works for my runner of choice” is likely fine.
pyproject.toml. This is the difficult one. Generally, tools only read their own “section” of the file (pip reads [build-system], build backend reads [project], everyone has their own [tool.xxx]). But 1.1 syntax affects everyone, regardless of what part of the file it’s in. So we need some level of acceptance of “what suits everyone”, which until now has been TOML 1.0 (because 1.1 didn’t exist).
What you’re saying here makes me wonder if even pinning the version down to 1.0 needs a PEP
If we mandate TOML 1.0, and insist on (some or all) tools validating, then those tools will have to pin to tomli <= 2.3 indefinitely (at least until the standard is updated to mandate 1.1, at which point maybe 1.2 has been released…). I’m not comfortable with the community making demands of projects like that outside of the standards framework. To give an example that is less about my views on what pip should do, I don’t feel I can reasonably demand this of uv, as I know nothing about the options for Rust tools to support TOML 1.0 for an extended period.
How does Rust’s Cargo handle TOML 1.1? Presumably they have the same problem, the toml libraries now handle 1.1 syntax, but older versions of cargo will only understand 1.0. Perhaps they have some other packaging feature that helps with the issue?
This is why I’m asking if it’s practical to keep an option for strict 1.0 parsing in current versions of libraries. I don’t want anyone to have to pin an old library version, or to add a dependency back because the standard library doesn’t have that option.
If the option is there and the standard says 1.0, we probably don’t need to actually mandate tools doing anything. I think Postel’s law (be liberal in what you accept…) is out of vogue enough that enough of us would parse with the strict 1.0 option so long as we can. I certainly would.
Does the TOML project have anything to say about how users should transition to the new format? I assume not, but it might be worth checking.
It’s never really been discussed, so not really. The way this worked with past TOML updates is that people just updated their parsers and this sort of just worked out. While TOML is an interoperable standard, realistically most TOML documents are read by one parser only. For example config.toml for dnscrypt-proxy is realistically only ever read by dnscrypt-proxy.
pyproject.toml is the exception here. As well as Rust’s Cargo.toml (to lesser degree, as Rust seems to have less of an ecosystem of tooling). I guess things changed a bit in the five years between TOML 1.0 and 1.1
Perhaps we underestimated the difficulty in updating for some TOML users like Python/PIP. I’m not sure if any hypothetical TOML update/hotfix could do anything to make things easier? e.g. any sort of version declaration in the TOML file itself should have been done in 1.0 to be useful for PIP now (assuming that’s even desirable in TOML, which I’m not sure it is).
For what it’s worth, I fully expect TOML 1.2 to be the final version. Assuming there will be a TOML 1.2 in the first place. Reading this discussion, we probably should have decided on these issues for TOML 1.1 and make that the final version, but that’s too late now…
Using TOML v1.1 features doesn’t affect the MSRV for crates published to crates.io, since Cargo always rewrites the Cargo.toml in a .crate to TOML v0.5 or lower.
However, using those will bump your package’s MSRV for development. We recommend people setting CI to verify package.rust-version.
I’m not familiar enough with current Python tooling to say if rewriting TOML files is an option.
MSRV refers to the ability to set the minimum required Rust version in Cargo.toml (not familiar enough with Rust to expand on the nuance of this).
Another option I don’t think I’ve seen mentioned is setting the version; that is:
import tomllib
tomllib.loads('a=1') # TOML 1.1, or whatever the latest version is.
tomllib.loads('a=1', toml=1.0) # Use TOML 1.0, forbidding newer features.
tomllib.loads('a=1', warn=1.0) # Use latest TOML, but warn if TOML >1.0
# features are used via tomllib.NewerTOML
# exception that can be caught/ignored/printed
# to stderr/etc.
Or something along these lines. This shouldn’t be hard to do in tomli since the changes in 1.1 are fairly minor. I did this in my Go parser for years to provisionally support the then unreleased 1.1 draft – it’s just a few ifs. For 1.2 we might add some new features, but I wouldn’t expect major changes to how TOML works there either.
And then in pyproject.toml, you need to opt-in to TOML 1.1 with something like:
[build-system]
require_toml = 1.1
Then at some point in the future when all supported versions support 1.1 it can default to 1.1 and that line will no longer be needed.
But I don’t really use Python or PIP so maybe I’m talking bollocks here – feel free to ignore if that’s the case.
Yes, I think this would be ideal for us. I don’t know if it’s something the tomli library and/or the stdlib is willing to support, but I hope they’ll consider it.
this could definitely happen non-deliberately. One of the things that changed was relaxing the rules on inline tables. They can span multiple lines now, and in general having them span multiple lines in tool-generated files is better for both git diffs and human review.
If a packaging library is using a toml writer that shares those views about inline tables and changes the way it is emitted they may end up writing data not valid for 1.0
This is especially true when there’s no language specifying 1.0 compatability.
Speaking as Tomli maintainer, yes this can be considered. Why is this more ideal, though, than a separate distribution (e.g. import tomli_toml_v1_0)? The interface is different, but all the same things can be achieved. I ask because I’d much rather maintain a TOML v1.0.0 enforcing fork than a spec version flag.
Or is it more so that the standard library maintaining a TOML v1.0.0 parser would be ideal?
Can this (rewriting as TOML v1.0.0) work for us too? To avoid the bootstrap cycle I think build backends need to either
a) Vendor a TOML v1.0.0 writer
b) Depend on a TOML v1.0.0 writer and only rewrite if it’s available. The writer’s pyproject.toml must not use TOML v1.1.0+ features.
Is there any chance option b) would actually work?
This. I want pip to not have to vendor tomli when the stdlib version is available, and the lack of TOML 1.0 support in Python 3.15+ is all that prevents that. (I personally don’t care if pip allows TOML 1.1 in Python 3.15+, but other pip maintainers do, so having 1.0 support lets both of us get what we want).
I’d like to be able to take that view, but the issue is with the current lack of specification; that framing of “deliberate choice” is pretty loaded. It was deliberate to update[1], but was it deliberate to break compatibility? Our current definition for compatability allows toml 1.1 content.
And there may be other reasons to update, such as bug/security fixes or performance improvements that were marked as non-breaking, because from a toml-centric standpoint, going from supporting 1.0 to 1.1 is nonbreaking ↩︎
First I want to apologize, I think there are separate concerns that have made this discussion a little more confusing and difficult that I should have highlighted in my original post.
The separate, but interconnected, concerns as I see them are:
What does the spec say and does it need clarifying?
What is the advice for tools that read Python packaging TOML files?
What is the advice for tools that write Python packaging TOML files?
And perhaps, what is the plan in the standard library?
So going through each of these:
Specification
I had not intended for this thread to start a specification discussion, I was more thinking What do you think tools/libraries/scripts should do? However, now there has been so much input I have some thoughts:
I think specifying TOML 1.0 does require a PEP, because of the timeline of TOML versions and the PEPs I don’t see any evidence that TOML 1.0 was meant to be the version defined, or implied, by any of the packaging specifications:
I take all of this to mean that no specific version was intended, and I am interpreting that as the authors used term “TOML” as “the widely available version of TOML for Python users”.
So I am -1 on updating the specifications to which TOML version should be used, without a PEP. And my personal take is that any PEP should propose how packaging tools should transition to new versions, not just pin to a single version.
Consequentially, I reject the assertion that if someone writes a pyproject.toml file using TOML 1.1 specific syntax, then it is necessarily not spec compliant.
Reading Python package TOML files
With regards to what might be best practice, I think for tools which have a very limited user base it’s to adopt TOML 1.1 readers as soon as meets their cadence, it prevents raising errors on TOML 1.1 specific syntax in the future, most limited tooling does not have to worry if something is in the standard library or not.
For foundational tools and libraries, a concern brought up is that if they switch to being able to read TOML 1.1 then users that may introduce TOML 1.1 specific syntax and the tool will not tell them that’s an issue, and it will cause newer packages/projects not to work with tools that haven’t switched or are older.
It’s impossible to guess the real world impact of this, but I think there are a some orthogonal choices these tools and libraries should consciously make, first with regards to errors or warnings tools can:
Read TOML 1.1 specific syntax and not worry about it, wait for real world user reports of issues
Read TOML 1.1 specific syntax, but emit a warning if it is not TOML 1.0 compliant
Error on TOML 1.1 specific syntax
I think all three choices are valid, but bear in mind that at some point TOML 1.1+ will be the commonly available TOML format, so even if a tool decides to reject TOML 1.1 specific syntax now, it will eventually become common.
Assuming tools adopt reading TOML 1.1 specific syntax the second choice is related to adoption strategy, two common choices are probably:
Adopt TOML 1.1 for all cases
Adopt TOML 1.1 as it becomes available in the standard library
I think both choices are valid but there are a couple of downsides for the latter, it is ambiguous for tools not based on Python, it creates the concern that was brought up: PopularTool™ would read TOML 1.1 specific syntax on one version of Python but fail on a different version.
As I don’t think this is a specification issue I would advise tools consider the options here and determine what is best for them, and listen to real world user feedback. Which choice seems sensible also might change over time, for example it might make sense to only support standard library TOML version for now but in 5 years it might make sense to vendor a TOML 1.1 reader for Python 3.14 as it might become mandated.
Writing Python package TOML files
I think we are all in agreement here that the preference would be to not use TOML 1.1 specific syntax, at least for the next 5+ years, so that packages and projects continue to work with older tools.
What is the plan for the Standard Library?
It would be great to have more insight on what the plan is in the Standard Library. I think adopting TOML 1.1 is generally a plus, but the wording of PEP 680 is a little concerning regarding the instruction to “treat minor revisions as bug fixes”. As a tool maintainer I find the most frustrating development moments when behavior changes have been implemented as bug fixes in CPython.
Zooming out a bit, I don’t understand the fuss here. Projects may choose to write TOML 1.1 files if they prefer, at the cost of being incompatible with other tools that don’t yet support it. That’s a trade-off the maintainers of a given project are capable of weighing themselves. I do not support hitting them over the head with “you are spec-incompliant”, when clearly, the TOML version in those PEPs is not specified.
If they’re popular enough, presumably they’d get issues raised with “your project doesn’t work with myfavoritetool”, and they may respond with anything from “ah, I wasn’t aware, let me revert that” to “I know; upgrade your tooling”. That’s just one of many trade-offs maintainers need to make.
To me the situation is pretty comparable with Unicode updates, which “just” happen ~every year (1, 2, 3, etc.). We also don’t go around telling projects they’re spec-incompliant for using too-new emojis.
I may have misunderstood your original intention, sorry.
… sounded to me like you were saying using TOML 1.1 is not spec-compliant[1], but upon rereading, I think you meant the opposite (and thus our positions coincide).
I stumbled over the comma before “then”. In hindsight, it’d have been less ambiguous as “Consequentially, I reject the assertion that it’s spec-incompliant if someone writes a pyproject.toml file using TOML 1.1 specific syntax.”, at least to me. ↩︎
Thanks for doing this review of timelines. While I didn’t ever actually say this, I had a vague recollection that when we decided on TOML, we acknowledged that it hadn’t yet had a 1.0 release, but we agreed that it was good enough where it was and we’d adopt 1.0 when it was released.
Your review prompted me to check, and I found this document linked in PEP 518. The significant section is:
I think that what all this means for us is that if we were to go with TOML, we’d just specify that our bootstrap file format is TOML v0.4.0 – which is a stable document, by definition – and then once they finally release a v1.0.0, we can look at the changes and decide whether we want to update. Most likely, it will be tiny compatibility-preserving improvements, in which case all is fine; or if not, then we (and Rust, and others) will stick with the old version, which is exactly the same situation as happened with YAML. (“YAML” to most people means “YAML 1.1”; supposedly YAML 1.2 is the latest version, but ~nobody supports it.)
And in fact, there was a very short discussion when TOML 1.0 was released on Discourse, where we discussed this, and came to essentially the same conclusion as now, that this was a spec change that needed a small PEP. No-one actually raised that PEP, though
So my conclusion is that informally, we currently require 1.0, but we never completed the standardisation work to formalise that. At this point we could write a PEP requiring either 1.0 or 1.1. But without a PEP, we’ll remain in the situation we now have[1].
+1. Even if we accept that the spec intent is 1.0, adopting a 1.1 reader is fine as it still reads 1.0, it just doesn’t validate as 1.0.
Agreed. One other factor I think tools have to consider, at least in practical terms, is what the stdlib supports. Especially for foundational tools, bootstrapping is an important issue, and the value of not needing to manage an extra dependency (even if the tool has a dependency handling solution like vendoring in place) shouldn’t be ignored.
I’m not sure I’d explicitly say “5+ years”, I’d prefer to keep 1.1 adoption under review, but I agree that in practice it’ll likely be that sort of timescale.
+1. There has always been an implied presumption (at least from me) that packaging practice was linked to what the stdlib supported.
Not least because bug fixes should be backported to all currently maintained releases of Python, and at the moment that doesn’t seem to be happening. So a clarification of the policy here is needed.
I don’t think we can claim to “informally require 1.1”, as the complexity we’ve seen here makes that significantly less self-evident than the 1.0 discussion was ↩︎
– and then once they finally release a v1.0.0, we can look at the changes and decide whether we want to update. Most likely, it will be tiny compatibility-preserving improvements, in which case all is fine; or if not, then we (and Rust, and others) will stick with the old version, which is exactly the same situation as happened with YAML. (“YAML” to most people means “YAML 1.1”; supposedly YAML 1.2 is the latest version, but ~nobody supports it.)