That’s the point where I think we stop simply making clarifications to the text and start adding extra requirements on conforming tools. I’m not against having that discussion (although personally I’m not sure it will provide enough benefit to be worth the time we’d spend) but IMO it should be a separate thread.
Normally I’d agree that having tight specifications and clear rules on transition and migration as spec versions[1] change is what we should be aiming for. If this had come up while the relevant standards were being developed, I’d be all in favour of tighter wording.
But it’s a lot of work, and I think we probably have better things to spend our time on than retrofitting such language to existing standards. We’ve had problems in the past with tightening up specs without going through the formal PEP process, but conversely writing a PEP is a very heavyweight exercise when the important bits of a standard are already in use and providing the key benefits we need.
As I said, I’m not going to stop anyone proposing such changes, but I will push back hard on any suggestion that they are “just clarifications”, and can avoid the formal processes needed to change a standard.
whether our own or 3rd party ↩︎
3 Likes
Fully agreed. It’s a topic in its own right.
Being pragmatic about it, our current tools for TOML in common use (tomllib, tomli, tomlkit, toml crate) don’t all have good API accordances for indicating what version was parsed. I know that tomlkit could do it on the document object, but tomllib/tomli don’t have such an obvious solution.
I’d be concerned about describing, in a spec, behavior which we know a priori tools could not or would not implement.
1 Like
tomli 2.4.0 is released with TOML 1.1 read support, I’ve raised a PR on pip side to add this in a way which I think conforms with agreed best practice in this discussion: Vendor tomli 2.4.0 (TOML 1.1) by notatallshaw · Pull Request #13750 · pypa/pip · GitHub
3 Likes
That’s a reasonable stance. I hope in this case that the natural social impacts (like breaking users using tools on a 1.0 parser) keeps the issues with this to a minimum. I have no desire to go through the pep process myself on this, and won’t ask anyone else to either.
1 Like
Continuing the discussion from the pip tracker here.
Python 3.15 is adopting reading TOML 1.1 and not going to support reading 1.0 only, so we [pip] either adopt this [re-vendor tomli] or accept inconsistent TOML read capabilities for pip.
It’s a weird situation indeed.
But this inconsistency issue is going to be present in the whole ecosystem because most tools that read TOML depend on tomli like this: tomli ; python < 3.11 and use tomllib otherwise.
So if pip starts accepting TOML 1.1 today, people will start relying on it (accidentally or not) so all packaging tools will have to rush and adapt their tomli dependency?
I don’t know how to resolve that properly, but I kind of feel this needs more discussion, time and coordination.
I’m struggling to understand the downside here.
First of all, no existing standard (as far as I’m aware) requires tools to check TOML files for validity. I don’t know what various TOML parsers currently do, but if one of them silently accepted unknown backslash-escapes, like Python used to do in strings, people could be accidentally relying on that right now. How is relying on TOML 1.1 features so much different?
Remember, we aren’t requiring a 1.1 parser at this point, just requiring a library that will parse valid 1.0. And a 1.1 parser does that.
If we require validation of 1.0 syntax, the python 3.15 stdlib module won’t be usable, and the whole point of getting TOML added to the stdlib will have been lost… (because we’ll have the same question when 1.2 comes out, and so on).
1 Like
My worry is that some TOML 1.1 syntax is rejected by the stdlib tomllib. For instance this from the TOML 1.1 changelog, fails to parse with tomllib:
tbl = {
key = "a string",
moar-tbl = {
key = 1,
},
}
So if pip starts accepting that today, while other tools are not ready, users may start using such syntax in their pyproject.toml, pip will not warn, and other tools which use the stdlib will fail processing such projects. Not to mention rust or javascript based tooling which may not support TOML 1.1 yet.
So we’d end-up with, say, projects that can be built by pip but not build or uv.
Of course we could just say “don’t use that syntax if you want interoperability”, but still I feel popular installers should not be in a leading position with accepting the new syntax, at least until the rest of the ecosystem has had time to catch up, and there has been some coordination. Because the new syntax is subtle (trailing commas), people will not notice if their favorite installer does not warn, and would therefore unknowingly be publishing sdists that do not interoperate.
Can the build environment specifiers require Python 3.15 for example, or is it build dependencies only?
Overall supporting Toml 1.1 in the packaging system seems far more trouble than it’s worth to me. But it’s up to every project to specify how it should be built, and what is supported to achieve that.
OK, that’s a different position.
I can understand that pip, as a specific project with a somewhat unique place in the ecosystem, might want to delay using a TOML 1.1 parser. But I don’t think that’s a general argument.
And given that there will be no TOML 1.0 parser in the stdlib for Python 3.15, it seems that we might be stuck with some very weird dependency requirements:
- Python >=3.11 and < 3.15: Use the stdlib
- Python < 3.11 and > 3.15: Use an older version of tomli
Or we ignore the stdlib and always use tomli, which means that one of the key reasons for including TOML parsing in the stdlib (the bootstrapping issue for packaging tools) hasn’t worked out.
Plus, we’d have to go through the same dance when TOML 1.2 came out.
I understand the risks, and I understand why you take the view that pip needs special consideration here, but I don’t think the benefits justify the cost (for pip - this is a project decision for pip, not a standards compliance question).
As far as standards go, if we accept @sirosen’s PR, the standards say that files are in TOML 1.0 format but that tools aren’t required to validate that. They can, of course, as tool-specific quality of life decision, but it’s not required.
If we want to require tools to reject TOML versions other than 1.0, that’s a much bigger change and as I said before, one that I think probably needs a PEP, or at a minimum, needs all of the key tools - uv, pip, script runners and lockfile managers, and build backends - to confirm that they either already validate or are willing to add validation.
I’m happy to have the debate about whether pip should go beyond the standards and validate pyproject.toml on the pip tracker. I’m not sure what else you’re asking for, so I don’t have anything else to say here.
Does the TOML project have anything to say about how users should transition to the new format? I assume not, but it might be worth checking.
IMO there is a difference between pip using the stdlib parser to parse 1.0 or 1.1, depending on what version is available, and pip going out of its way to support 1.1 even when the stdlib doesn’t.
The latter means that pip is actively going beyond the standard to support 1.1, which will create an expectation in the ecosystem that 1.1 works, no matter what the official stance on this is.
2 Likes
This is a hypothetical I don’t understand.
Would people use tools that don’t follow the standard* and needlessly use the newer constructs?
Or they would not follow guides and tutorials but some TOML 1.1 reference and thus write newer constructs?
(* granted, only recently clarified regarding TOML version, as that wasn’t necessary before)
From my experience, most users won’t check or be aware if what they are writing is valid toml 1.0 or toml 1.1, they write whatever toml their current IDE/tool accepts. Allowing a 1.1 will lead to user confusion where pyproject.toml (or pylock.toml) are considered valid during development, until they error at some other point. For example, if we support toml 1.1 in maturin or uv (which we want because we want to stay up-to-date with the toml crate), users will be using the new syntax, until they arrive at a tool that doesn’t support it, and then either file a bug against that tool or against maturin or uv.
A possible outcome with toml 1.0 in the spec is that we’ll have de-facto support for toml 1.1, because no tool wants to be the one to not support it, even though the spec say toml 1.0. (I see problem a lot less if it’s only old tools not support old toml, then the answer is the regular update to latest version for the latest features, similar to other additions we made to pyproject.toml)
Would it instead be possible to require toml 1.1? Tools can use tomli like they are doing for pre-Python-3.11, this ensure a consistent experience across the ecosystem, and allows using the long awaited indented inline tables.
I may be biased here because in most tool I maintain we can’t use the Python standard library, but I’d rather avoid tying packaging standards to the standard library. Tools that need to support a large range of Python versions can’t select the standard library version they get, and packaging standards can’t change old version of std, but can depend on or vendor specific library versions or version ranges. For what it’s worth, the problem isn’t unique to toml. I once had a py3-none-any wheel built with only static configuration that would include different files depending on the Python version due to changes to Path.glob().
1 Like
But no-one is suggesting that.
What pip does is off-topic for here, but we routinely upgrade all of our vendored dependencies each release. That’s what we’d naturally do for pip 26.0, vendoring the new version of tomli. We only use tomli for Python 3.10 and earlier[1].
IMO, that’s anything but pip “going out of its way” to support TOML 1.1.
When the stdlib upgrades to TOML 1.1 (Python 3.15), we’ll get that support. Not supporting TOML 1.1 in Python 3.15 is what would require “going out of our way”.
That’s for reading - we use
tomli_wfor writing, which is a whole other question we haven’t looked at yet, AFAIK ↩︎
That is not what I read out of the github dissusions:
One drawback of this is that we’ll drag along a vendored dependency for 5 more years where we could have removed it in one year or so when we drop support for Python 3.10.
The alternative is the same version of pip supports TOML 1.1 on some versions of Python but not the others.
So at least to some people over there the understanding was using tomli and the 1.1 parser till python 3.14 so that pip can always accept 1.1 syntax. This is also what is currently implemented in that PR:
- if sys.version_info >= (3, 11):
+ if sys.version_info >= (3, 15):
OK. That’s not what I’m advocating, and never has been. Thanks for pointing that out, I’d not looked at the PR in detail.
That would be my preference, but it does depend on this discussion, too. If PyPA decides that packaging tools are required to reject new TOML features, then tomllib functions (which exist to enable bootstrapping packaging tools) should grow an argument to reject 1.1.
But yeah, I do think validation should be outside the bootstrap cycle, same as writing TOML.
we routinely upgrade all of our vendored dependencies each release
@hukkinj1 said there’ll be a 2.3.1 if needed. IOW, you can treat tomli 2.3 as a “LTS” branch, and tomli<2.4 as up-to-date.