Adopting TOML 1.1?

Packaging Coordination
101

The gentlest solution for build tools supporting 1.1 would be to (optionally?) warn if 1.1 features are used.

1 Like
102

FWIW, it looks like the biggest changes for TOML 1.1 are allowing tables to span multiple lines, and allowing trailing commas. These seem highly likely to creep into hand written files.

Builds tools should probably warn about 1.1 features and potential compatibility issues for the next few (five?) years.[1]

But I don’t see any actual users being happy about a complete ban - that’s just what spec authors like to do :wink:

  1. How to do this is left as an exercise to the tool authors… I guess parsing in 1.0 mode and falling back to 1.1 mode is the way? Which requires library updates and work and etc. etc… ↩︎

3 Likes
103

I think I actually ran into this a month ago with MarkupSafe, but for a feature that was introduced in TOML 1.0. Users on old Raspberry PI Debian installs started reporting they couldn’t install MarkupSafe. It turned out that we had moved our Tox configuration into pyproject.toml, and that introduced some syntax that wasn’t understood by the version of tomli vendored into the version of pip bundled with Python 3.9. As soon as they knew that, they upgraded pip and were able to install fine. sdist of markupsafe 3.0.3 fails to build under Python 3.9 w/ pip 20.3.4 (e.g. Debian Bullseye version of Raspberry Pi OS) · Issue #509 · pallets/markupsafe · GitHub

4 Likes
104

I don’t think any existing library offers the option to accept 1.1 but warn if 1.1 features are used. Current options are “silently accept 1.1” and “accept 1.0, but fail on 1.1”. And no implementation offers a choice between those modes, it’s one or the other.

Ideally, the stdlib implementation (and tomli) could offer 3 modes:

  • 1.0 only
  • 1.1, warning if new features are used
  • full 1.1

Getting that backported to Pythons before 3.15 is probably a big ask (the PEP adding tomllib says that new versions should be treated as bugfixes, which I assume means backporting to supported older versions of Python, but that doesn’t seem to have happened yet, and extra operating modes is a bit more than just supporting a new version). But if it’s in 3.15, tools can use tomli for backports.

There’s also the question of tools (uv) not written in Python, who have their own libraries to consider.

Agreed. What I’m not sure about is how happy users will be at needing to update their tools. I think they should be OK with it, but there’s always someone stuck on an old version. And I suspect many people will only discover the need to update when something breaks. Which sucks.

I have special privilege here - as a tool maintainer, I can blame the spec authors, and as a spec author, I can blame the tools. Which cup is the ball under? :slightly_smiling_face:

IMO, there’s no really good answer here, just a set of trade-offs.

2 Likes
105

Might be easier to parse tracking the highest featureset used, then just make it possible to access information about that, letting those using the library decide if and what to do with that information. I don’t see it as a a toml parsing library’s job to warn on supported features.

3 Likes
106

90% will be fine with it, which just leaves a million or so who aren’t :wink:

The real concern is how many package authors won’t be happy that their users are unhappy, and that’s only a concern because the package authors are here and can shout at us. I personally have no problem pushing responsibility back onto them (“don’t do things that hurt your users if you don’t want your users to hurt”), though I know Paul and others cop the feedback more directly than me.

So announcing that TOML 1.1 is probably going to hurt users in the short term feels like a fine trade off to me, and build tools warning the package developers when they detect it’s an issue is a bonus, but I wouldn’t consider doing anything on the installer side. It’s a one-to-many[1] relationship, and so the “one” end is the chokepoint.

  1. One version of the tool used by the developer; many versions of the tools used by the consumer. ↩︎

2 Likes
107

As a maintainer, as long as a recent version of popular installers support it, I’m not going to go out of my way to avoid TOML 1.1 features. My experience over the past decade of occasional install issues is that this strategy has worked fine, and the vast majority of people are perfectly happy to update pip/setuptools/etc if a newer version addresses an issue.

5 Likes
108

Thanks, that’s useful data.

If a recent version of pip supported TOML 1.1, but only on some versions of Python, how would you feel about that? Because that’s the tricky issue - do we require tools to vendor 3rd party TOML parsers because older Python stdlibs don’t support the latest features?

109

We support all Python versions that are not EOL, so we’d do the same as with other features added in newer Pythons: add a # do X when >= 3.14 comment to start using it once it’s available for us. If that’s not required because pip provides a newer parser, that would make me happy if it was for a feature that is much nicer than the 1.0 way.

I’d be fine treating this as any other Python feature: the newest features are available in the newest versions, and it’s up to a project to decided how many older versions to support. That doesn’t require files or parsers to add any version argument complexity, it just requires waiting.

8 Likes
110

Can we consider wrapping up this thread soonish? Judging by the repeating suggestions, points, and opinions it has become too onerous to fully keep in mind, so it’s no longer serving the purpose I intended.

There are some concrete actions people have suggested:

  • A coordination plan between tools, and update user guide to reflect that
  • Updates to the specification with a PEP
  • Reqyest, or submit PRs to, tomli or the standard library to support specific features to make migration easier
  • Request, or submit PRs, to popular linters to warn when using 1.1 specific syntax in Python packaging toml files

Beyond a notice here they are probably all worthy of their own DPO thread or discussion in where ever is natural for specific feature requests or PRs.

Also I’d previously requested the standard library to hold off on upgrading to TOML 1.1, but as there are no spec clarifications of concrete decisions coming out of this thread I’ve withdrawn my request.

3 Likes
111

Sounds good to me.

Just a comment regarding a previously expressed expectation in this thread that build-backends warn about possible features not supported in 1.0: this is a big ask, specially considering that right now there is no parsing library capable of doing that. And it would involve vendoring - if not directly supported by the stdlib - which is always leave a sour feeling…

2 Likes
112

The whole problem is that common CI/CD workflows won’t catch this. Given that the common workflow for a pure Python package is to:

  • build the wheel once, usually using the most recent Python version (or uv)
  • test the package against all Python versions, usually working on sources directly

…we’re bound to see CI workflows succeeding with TOML 1.1-capable tool and then primarily downstreams (and sometimes users) reporting issues when they happen to use other tools. If the tools (reasonably) use tomllib, then building on Python versions will trigger issues that people using a newer Python never see.

For example, on Gentoo we run the build with all enabled implementations, in order. For pure Python packages, we cache the first result. So if the user has Python 3.11 still enabled, the build will happen on Python 3.11 — and the user will see issues that, say, don’t happen when building on Python 3.15, or installing from prebuilt wheels.

113

I found this claim surprising. I have not encountered this as a common pattern. The most common thing I see is a job matrix that installs the package with [test] extras and calling pytest. That installation should trigger an issue on TOML 1.1 with Python <=3.14.

I may be wrong, but I feel like the dangers of testing without installing have bitten enough projects over the years that I just never see it anymore.

5 Likes
114

The CPython PR for TOML 1.1 is in; it’ll be released in the next alpha.

Is there a feature request for adding a version selection argument?
If I’m reading the discussion right, it doesn’t seem necessary.

2 Likes
115

In this thread, there seems to be some level of expectation that usage of TOML 1.1 is flagged to the user. For example in:

One possible way to achieve that would be via a version selector.

But I don’t think there is full consensus yet about this, for example:

Also relevant: Adopting TOML 1.1? - #104 by pf_moore

116

Has it been backported to 3.14/3.13, as per PEP 680? That was an outstanding question earlier in this discussion.

117

No, it hasn’t.

@hautsaninja said:

We should not backport TOML 1.1 to Python 3.14 and 3.13

I was the author of that sentence [in the PEP]. IIRC at the time it seemed possible that TOML 1.1 would be minor spec clarifications or would maybe ship while Python 3.11 was in a post feature stage (e.g. release candidate), so I wanted to keep the option open. Multi-line inline tables is a big enough and useful enough feature that I think backporting to 3.14 and 3.13 would be a bad idea.

I fully agree with that.

Formally, the PEP uses “can” and “should”, which suggests some leeway. Also, bugfixes don’t require backports. As the core dev merging this I’ve decided not to backport; feel free to challenge my decision on the issue (or with the SC) :‍)

6 Likes
118

No, I have no intention of doing that - your reasoning seems perfectly correct to me. It does have an implication for how packaging tools will support 1.1, but that’s for us to work out now we know what the stdlib is doing.

2 Likes
119

It’s still possible to add a version argument to tomllib, if that’s the consensus and it’s reached it 2 months or so.