Pyproject.toml is being used by increasingly many, often not build related tools. Each of the tools reads the file using a parser of their choice, implementing an arbitrary TOML spec version. This creates ambiguity when reading pyproject.toml: some tools interpret it as TOML v0.5.0, some as v1.0.0, some will raise an error parsing heterogeneous arrays, some will not, etc.
This is unfortunately already the case: the two most widely used libraries (toml and tomlkit if I’m not wrong) are at TOML versions 0.5.0 and 1.0.0-rc.1. A valid package built with Poetry may cause pip to throw an error due to TOML syntax not valid in v0.5.0. The problem will only get worse as the TOML spec advances and gets new versions.
I think it would be a good idea for PEP 518 to state that pyproject.toml must be parsed, for instance, strictly as TOML v1.0.0.
I agree. Now that TOML has reached 1.0.0, I think that updating PEP 518 (which is still the reference definition, linked from ) to require that version of TOML is the right thing to do.
On a procedural note, this is technically a backward-incompatible change to the definition. And unfortunately, the reference documentation for pyproject.toml is still PEP 518 (linked from the PyPA specifications page). We had a somewhat inconclusive discussion on process that ended up here - but that doesn’t really allow for modifying a PEP after it’s finalised.
I think therefore that this change would need a PEP, which requires pyproject.toml to conform to the TOML 1.0.0 spec, while at the same time moving the canonical spec to packaging.python.org so we can be more flexible in future.
But having said that, I’d also be happy to support a proposal that we find a better workflow for changes like this (which are so small in scope that a PEP seems over the top, but which have non-trivial interoperability implications). I don’t have the time to champion such a proposal myself, though…
While we are at it, shouldn’t pyproject.toml contain a version header, so that if the spec is updated, parser can still load old and new packages instead of getting stuck in the past or having to deal with fuzzy detection?
An issue here could be that TOML doesn’t natively support version headers. How do you read the version header if your parser crashes hard trying to read the file?
Maybe use something like a hashbang like convention where if the first line is a comment and satisfies certain rules, then that’s a version header.
Anyhow, I think this is a lot more problematic than choosing a single spec version. The pyproject.toml is read by many small tools and utilities that probably can not afford having a TOML parser for n number of spec versions. These tools would have to error when noticing an incompatible TOML, and whenever TOML spec updates all your tools and utilities would be out of sync and incompatible for a long while. Also, there’s added complexity to the small tools: in addition to supporting TOML, they’d need to support the version header convention.
Adding to that, I don’t see anything wrong or missing in TOML v1.0.0. I don’t see why it wouldn’t be able to serve us perfectly for years to come. Not a TOML maintainer myself, but still I also find it unlikely that TOML makes back incompatible changes. So even if parsers keep in sync with the updated spec, they’ll still be able to parse TOML v1.0.0 used in pyproject.toml.
