This presents an interesting compatibility challenge for Python packaging standards, as I don’t think any of the PEPs that introduce toml based files (pyproject.toml and pylock.toml) specify which version of TOML should be used.
I assume that at some point TOML 1.1 will be added to tomli and then those changes will be upstreamed to the standard library, but probably not back-ported.
My question here is what would be a good strategy for adoption? My preference is to adopt reading it as quickly as possible, but don’t specially call it out, e.g. not having something in the changelog like"now supporting TOML 1.1", and if possible emit a user warning when 1.1 specific features are used, at least for a few years.
But I’m sure others have opinions or perhaps even experiencing in adopting new format versions.
Regarding the adoption strategy, a lot of time and care has been taken over these changes, and it’s a minor version bump. So I would presume there are no breaking changes, but first of all I’d double check that.
IMO we should not adopt 1.1 until it’s available in the standard library (and tools have dropped support for Python versions that don’t support 1.1). None of the new features seem compelling enough to me to require tools to depend on (or vendor) a 3rd party TOML library.
I have no objection to tools supporting 1.1 if they choose to, but the standards should require 1.0 for now, and using 1.1 features should be considered tool-specific, and non-standard.
The problem is that if people use TOML 1.1 in, for example, pyproject.toml, then older tools, or tools which use the stdlib parser, will fail with an error on that file.
For pip that will mean packages that use a TOML 1.1 pyproject.toml will work on some versions of Python but not others, I would much rather pip consistently support 1.0 or 1.1 for all versions of Python it supports. I suspect other tool authors would feel the same way?
AFAICT the 1.1 only relaxes rules, so some code that is invalid on 1.0 no longer raises an exception. I am not sure if this is a problem.
Either way, changes to the stdlib tomlib should be done under strong consideration to the concerns of the packaging community - the pyproject.toml file is after a significant part of the motivation for why it’s in the stdlib.
If the packaging community decides that PIP and other tools should continue to only parse in TOML 1.0 then the stdlib should support that in some way.
If the standards state that 1.0 is required, then tools that support 1.1 still follow the standard on compliant input.
Or are you suggesting that compliant tools must reject invalid syntax, so accepting 1.1 syntax is in violation of a requirement that we accept 1.0 syntax? Because I’m not suggesting that (nor do I think it’s viable in practice to demand it).
My concern is the following: Python 3.15 supports TOML 1.1 and not a 1.0 only reader, then someone writes a pyproject.toml using the new relaxed grammar, they only tests against Python 3.15+, a user of their package on Python 3.14 or below gets an exception trying to parse the pyproject.toml.
If we want to avoid this scenario then tools must either:
Use a library that supports reading TOML 1.1 for older versions of Python
Or, only use a library to read TOML 1.0 for all versions of Python, and stop using the standard library.
Assuming 1, on the basis of the preference to keep using the standard library, then there is no option to stay on 1.0 only, given that tools may as well adopt 1.1 for all versions of Python as soon as there is use a library that supports 1.1.
To be honest, I wouldn’t worry that much. It’s roughly the same as if you made a setup.py file (or a build backend) that is incompatible with a particular Python version — perhaps even less likely. In the end, you need to test against all Python versions.
Use a library that supports reading TOML 1.1 for older versions of Python
I think pip vendors tomli anyway, and will probably vendor it for some time to come, so in the worst case, it could use tomlifor, say, Python <=3.15. Though obviously that goes against all the downstream effort to get rid of tomli now that TOML is natively supported.
We should probably be clear that the standard only allows for 1.0 syntax. That sets expectations properly. Users might accidentally use syntax from 1.1 if all their tools support 1.1, but as soon as someone tries a tool with a 1.0 parser, they’ll get an error and can raise an issue/PR.
Mandating 1.1, on the other hand, requires that we get all tools to update their TOML parser. I don’t think forcing that is the right thing to do, it’ll happen naturally over time, and once 1.1 support is ubiquitous, we can update the standards to allow it.
Sorry, I’m a little confused by the phrasing here, are you suggesting until 1.1 support is ubiquitous (in Python?) all standards compliant tools should reject TOML 1.1 specific syntax?
FYI, I am not suggesting 1.1 be mandated, but I am trying to consider the consequences of Python dropping a 1.0 only parser from it’s standard library for Python based packaging tools.
I don’t think that’s the suggestion–tools that support 1.1 can read it and be happy. They don’t need to police the files they read–the parser already does that. Older tools will fail on that syntax and so projects that use it will get bug reports.
If TOML 1.0 is always valid for a TOML 1.1 parser (and I think it is?) then a 1.1 parser will happily ingest all existing files, plus additional files that use the new features (including files that have nothing to do with packaging). Packages shouldn’t use those new features unless they don’t care about old tools (which is their right, but they’ll probably hear about it unless they have a very specific userbase)
We should definitely clarify the specs to require TOML 1.0 syntax. I don’t think that even needs a PEP; it seems pretty uncontroversial to me. Should I put up a PR on the PyPUG repo to suggest a location and wording?
I was briefly concerned that the new escape chars would decode successfully in TOML v1.0 but to a different result, but a quick test confirms that they’re rejected:
>>> tomllib.loads(r'null = "\x00"\n')
Traceback (most recent call last):
... # snip
tomllib.TOMLDecodeError: Unescaped '\' in a string (at line 1, column 11)
As for writing data out, for tools which generate TOML… that’s sort of a different problem space. If the specs (once updated) say that the data must be in 1.0 format, then
tools which write new TOML pyproject/pylock files should always use 1.0 syntax
tools which update files in place should never produce TOML 1.1 output where the original was TOML 1.0[1]
tools which update files in place should probably preserve TOML 1.1 files as such when they are encountered[2]
Which may be expensive and difficult to detect. I’m not trivializing it, but I think this is the only sensible behavior anyway, regardless of Python packaging needs. ↩︎
i.e. If my pyproject.toml is detectably in 1.1 format, I don’t expect a style-preserving update tool to convert it back to 1.0 . But that would actually be sensible behavior for packaging use cases. ↩︎
A little quirk is that script metadata wasn’t explicitly specified as TOML – it was thoroughly implied but never stated outright. So we have to clarify that too, but that should also be uncontroversial, I hope.
Seems like a good idea in theory, but I think if we’re locking a version into the spec, we should also include language that doesn’t require use of a 1.0 parser or actively rejecting things that weren’t valid in 1.0 but are valid in 1.x
As it stands, saying the spec uses 1.0 without further guidance means either not using a newer parser, or having to implement added checks for what effective feature version was parsed.
I don’t think so. The text in the PR is “the file is written in TOML version 1.0”. That doesn’t say that it must be checked for valid TOML 1.0, or that it can’t be parsed with a TOML 1.1 parser, as long as that parser can parse TOML 1.0 (which it can, because 1.0 is a subset of 1.1).
Yes, we could be more precise and add in a bunch of “consumers MUST…” rules[1], but I don’t think we need to. Let’s keep things simple.
The pyproject.toml file is written in TOML version 1.0.
(added: version 1.0)
If we had sections which describe how tools should behave if the data is found not to be TOML, I would want those sections enhanced with details on how to handle TOML 1.0/1.1 data and parsers. But we currently don’t even specify how tools should behave if the data is outright invalid, so it would be odd to be more specific only on this particular topic.
Perhaps it’s worth adding a separate page in the PyPUG which specifies how TOML parsing should be handled in Python Packaging (unless otherwise specified). Putting it in a dedicated place gives us more freedom to be precise in a way that would be burdensome to include inline in 3 distinct specs.
While the spec doesn’t prescribe this, what people have learned over time with protocol specifications is that except when leniency is an explicit feature, it’s better to reject anything not supported to prevent non-standardized things from becoming ossified into what everyone ends up having to allow.
As such, I read current lack of language about how to handle this as meaning both that anything other than TOML 1.0 should be rejected, and that unless it’s uniformly rejected, anything that looks like TOML 1.0 will eventually be expected to pass silently.
