It does little to advance compatibility as a practical matter. Absent any guidance, the current crop of maintainers can all be trusted to make reasonable choices, and there are relatively few bad options.
But when you look at the current (lack of) spec language it just feels a bit weird. Arguably, packaging tools ought to be required to parse TOML v0.4 – though I don’t think anyone takes that idea that seriously, based on this thread so far.
I don’t think of the cost as being high. Nor is the benefit great. It just seems like a chore that needs doing.
It seems like all the changes to the spec have been additive or clarifications to the spec. So I think they already can, with the caveat that the older specs had ambiguity (a 0.4 parser might be fine with something that a 0.5 parser rejects, but that doesn’t mean the input was valid TOML).
And that kind comes full circle to the current topic–allowing tools and libraries to support 1.1 shouldn’t cause problems. Packages using 1.1 features in their TOML will cause incompatibility with older version, but that’s also true of e.g. using an assignment expression in your Python.
When you use an assignment expression you can bump your requires-python and then someone on some old system with ancient Python and pip can still pip install foo and just get an older version of foo from the days before assignment expressions. If old pip can’t parse pyproject.toml then it can’t even read requires-python.
This doesn’t help for source directories or archives, but it could be advised that if a project insists on using TOML 1.1 specific syntax it could require Python 3.15+ in the core metadata of is it’s distributions.
IMO the workflow for supporting TOML 1.1 across the packaging toolchain/ecosystem should be something along the lines of:
validate that existing frontends & backends provide sufficiently good error messages on encountering TOML 1.1 – if they don’t, we should make a release that does.
frontends start accepting/parsing with TOML 1.1, starting with pip+uv, then build etc.
build-backends start accepting/parsing TOML 1.1 after such major frontends have added support.
Ideally, we get this done before Python 3.15 is stable (i.e. in 6-9 months). That removes needing to coordinate with CPython on the specific implementation details. And, while I’m in this ideal world, I’d say that frontends and backends should not drop support for a Python version between the releases adding TOML 1.1 support vs the previous release of theirs – to give an on-ramp for those Python versions into TOML 1.1; minimising some disruption aspects for the ecosystem.
Finally, I think the decision on whether a particular package uses TOML 1.1 is one for the package author to make. We should document what the consequences are (tooling incompatibilities, Python version support, etc), and make it easier for package maintainers to make an informed decision on this topic.
I think we should do almost the opposite of this: Communicate explicitly and clearly what the plan is, and make it easy for users to see how it’s progressing. The most important piece of that would be having a single place to direct people to – listing which release of each tool added support, and having guidance as well about how we’re rolling this out / setting expectations.
Ideally, we’d also include/cross-reference this with the error messages from relevant tools (when the TOML 1.0-only versions of those projects encounter TOML 1.1 there as well), so that search engines and, hence humans, would have a better chance of discovering that particular page. Maybe crosslinks from tooling documentation would help too.
Kinda-sorta catching up on the thread, which has moved wayyy faster than I could keep up with… Sorry if I’ve skipped over something/missed some aspect.
With my pip maintainer hat on, I don’t think avoiding vendoring a small TOML parser library in pip is a good enough reason to make rolling this out across the ecosystem more complicated? It’s not a heavy dependency any way I can look at it.
Having a versioned parser in tomllib seems useful, but the right place to discuss that is CPython’s issue tracker. It won’t remove any of the work for us, unless we want to go down the route of delaying complete TOML 1.1 by at least 5 years for the wider ecosystem. If we’re thinking of this, then we should lean in fully - have TOML 1.1 get used unconditionally on Python 3.15 and TOML 1.0 on older Python versions.[1]
I don’t think the specifications should be updated. This is a coordination problem for managing a change, not a specification/clarity problem IMO. Updating the specifications won’t change the need for the underlying coordination problem.
This is my preferred approach TBH, because it means we’re not breaking any users and it also sets a clear deadline. I am not pushing for this because I know this approach is controversial because folks don’t like the idea of coupling packaging tooling behaviours with CPython versions (based in past experience + increased upgrade-Python friction). It still needs us to do a bunch of communication work, although it can be shaped differently because we can couple it with the Python release’s communication. ↩︎
To be honest, I think it would be much easier for everyone if tomllib had an option to select between TOML versions. Unless I’m mistaken, adding a few conditions to the parser would be easier than maintaining two separate parser versions, especially if it would literally mean vendoring two different tomli versions simultaneously (one to provide TOML 1.0 support for newer Python versions, the other to provide TOML 1.1 support for older Python versions).
I’m not clear on why I’d specify 1.0 in tomllib. I can parse 1.0 just fine with the 1.1 parser, and the library doesn’t write TOML, so what is the option for? Just for testing other files?
That was the requested use case, yes. But when you write it out loud, it indeed doesn’t sound like exactly the best use case. Though I dare say the ability to write validators is also useful.
I suppose my inclination is: if the goal is to test a file for compatibility with Python 3.14 (or below) it makes more sense to just use that version. You get the TOML validation alongside every other change in the distribution.
That only works for users using Python packaging tools that are Python based, the second most popular Python packaging installer isn’t Python based and so will use the same version of TOML regardless of the version of Python.
But uv has no need to enforce toml 1.0 at all? Why would they? And the features of tomllib have no relevance to them since they aren’t using it.
I was viewing it from the frame of someone releasing a package. If you want to ensure it can be installed by pip for python 3.14, test that combination when you are making the release.
I’m not really sure what we’re talking about right now, I think we’re discussing different things?
@mgorny brought up the idea of adding a version flag option to tomllib in the standard library. I was asking what the purpose for that would be–the only thing that specifying 1.0 might do is fail on a 1.1 file that uses new syntax. As far as I can tell, the only reason to need that is a) you are preparing to release a package with tools running on Python 3.15+ and b) you want to test if the TOML file works on ≤3.14.
If you aren’t using Python at all, the features of tomllib are not relevant–you will have a different solution. If you have access to an older version of Python, you don’t need this feature because you have its equivalent in the old tomllib. And if you don’t have access to an older version of Python, then how are you testing the package?
The scenario we’re trying to avoid is a package author builds their package with tool using a TOML 1.1 capable parser, uses a 1.1 specific feature without noticing because it builds fine with what they’re using, then someone else tries to install it using an installer that can only read TOML 1.0.
To be sure that will never happen, build tools would have to handicap themselves somehow, refusing to read TOML 1.1 features. The version flag to tomllib would be one way to achieve that handicap (at least for Python – it’s no use to uv).
I get it! I’m saying that one way to test this is to have your CI/CD install the package using the tool/environment that you’re trying to support. That is the most direct way to test the scenario in question. There’s no possibility that e.g. some patch release of tomllib fixed a bug that exists in 3.14 and would preclude people from installing your package even though you thought you tested it with the use_v1 flag.
Clarify one thing for me please. Is the prproject.toml file shipped with wheels and read by the tool installing the wheel? If so, the wheel standard ought to specify the TOML version (though I assume it doesn’t, since apparently a new TOML version wasn’t widely anticipated).
Since the premise of PyPI is that every build system produces standard-compliant wheels and every installer can handle any standard-compliant wheel, as long as we want to support installers that only support TOML 1.0, all build systems should produce wheels containing a pyproject.toml that complies with TOML 1.0.
Unless I’m wrong and the pyproject.toml is not shipped in the wheel or not intended for use by installers when installing wheels.
Separately, there’s the issue of rebuilding a package from source when no suitable wheel exists. At least pip with default flags tries this, although I’ve never seen it succeed (in my case it’s typically failing to build numpy when I’m trying out an alpha release). Since the build system to be used is specified in pyproject.toml (unless it’s in setup.py), that would require the installer tool to read pyproject.toml – and again the TOML version in the source dist should be TOML 1.0 until all installers that do this have been upgraded to 1.1.
Caveat: I definitely don’t have an complete understanding of how all this works or is supposed to work. I do have experience using uv (as well as pip of course), and have moved from setuptools to uv_build for my latest project.
Yes, you’re wrong. pyproject.toml is only used when building a project from source, so it’s manually written by the project author. (Who may or may not be aware of what versions of TOML the standard supports).
Yes. If a user uses TOML 1.1 syntax in their pyproject.toml[1], then installers and build backends that don’t yet support TOML 1.1 will fail when reading pyproject.toml, and the project won’t build.
The question is whether this is a common enough situation to warrant pushing tools to implement 1.1 support rapidly, or requiring tools to uniformly reject 1.1 syntax (so that the user gets a consistent error, rather than tool-dependent behaviour). And there’s a complexity because currently the stdlib implementation is 1.0 only in Python <= 3.14, and 1.1 only in 3.15 and later.
Whether or not it’s in information related to the project build (it could be config for a development tool, for example) ↩︎
