No worries 
1 Like
I’ll start on a proposal covering just toml use in files intended for interoperability in packaging. It’s new enough and limited enough in where it is used for interoperability, while also being the format seemingly preferred for new proposals.
There’s at least one more hiccup on this than I thought of originally, which was pointed out by the effort to document tomllib’s behavior: tomllib can’t be portably used for floats. I don’t think any packaging standards store those, but because tomllib uses the python float, there are systems where float will only be single precision, which deviates from the toml specification, and would result in different outcomes on different machines.
I’ve reviewed the differences in depth as well as the existing places packaging uses toml meant to be read by tools possibly other than the one that wrote it.
The biggest issue is one that in practice may already.
TOML 1.1 added valid representations. This means using a writer implementing 1.1 is incompatible with any tool using a reader implementing 1.0
I could not find any existing packaging specification that restricted to the subset of representations compatible with either an explicit version or the representations that the standard library is capable of parsing.
While either of those restrictions would be fine with me, I’m reminded of the problems people have had in the past of supporting yaml and availability of different parsers. This is obviously less extreme than the differences in yaml versions, but it’s the same underlying problem.
Unlike yaml, toml doesn’t declare the version for use by parsers as part of the document. We could require a comment as the first line of any use of toml indicating the version, but existing files won’t have done this.
The relaxed grammar in 1.1 includes many things that are strictly beneficial if we could guarantee the use of 1.1, so it might be worth setting a date for the time at which tools should be prepared to handle 1.1.
The transition here could be rather smooth, since correct 1.1 parsers will always parse 1.0 correctly. Tooling, except any meant to validate if a document is readable by tools using 1.0 parsers, can adopt a 1.1 parser today.
Adopting a 1.1 writer today could result in breakage. Setting a date when tools are allowed to write using 1.1 allows not needing to pin dependencies that might not want to keep the ability to write older version indefinitely.
The other remaining details not pertaining to evolving to use 1.1 that I think are worth specifying:
- Tools must use a parser that can handle the recommended integer range (In practice, this is all actively developed toml parsers)
- Data specifications must not rely on key ordering of tables. In the case of needing an ordered mapping, The design should use a list of tables or a table of tables with an explicit index value in inner tables.
- Floats should not be used in any data specification for packaging.
- Any use of toml’s dates must be interpreted as UTC
I couldn’t find any existing use of floats in packaging standards. It’s possible I missed one. tomllib does not and cannot reasonably guarantee it stores floats as specified by toml, and I can’t think of any reason packaging standards should need to use floats.
1 Like
We already had a lengthy discussion about whether to specify TOML 1.1 or 1.0: Adopting TOML 1.1?.
1 Like
Okay, nothing came out of that prior discussion. All of the points about compatibility problems people made in that discussion remain true, and this is an open problem that I’d like to solve.
I have bigger goals here than just the 1.0 vs 1.1 portion because the problem is larger than that.
Maybe the mistake was using toml to begin with rather than a format we specify and control the evolution of from what’s needed in packaging, but seeing how the discussion about replacing wheels has gone, maybe that would have been bikeshed to the point of nothing getting done too.
1 Like
To me, the main thing that came out of that discussion was there are a lot different opinions, and some are contradictory, for lots of different reasons.
So any new proposal should be grounded in understanding those reasons, even if it can’t satisfy all of them.
Some of them wouldn’t have been as big of a deal if things were more specified and agreed on in the first place. But not all of them.
Okay, thanks for explain why you linked the prior discussion.
I don’t think it’s a good thing to leave an interoperability standard in a known ambiguous and underspecified state.
I think it’s unfortunate that we’re here, and I’m trying to minimize the work the ecosystem as a whole needs to do to get us out of the ambiguous state.
toml 1.1 vs 1.0 is the largest problem in that, so if I need to do this is two phases to get the less controversial and easier to handle parts progressing, I will, because incremental improvement is better than no improvement here.
Keeping the standards process constrained to the types, ranges of values of those types, and semantic behavior of those types that toml can portably and unambiguously handle is, as far as I can tell, fully compatible with all existing packaging standards using toml. No existing packaging tool or specified file should be broken by the set of restrictions I want to propose that are not part of the 1.0-1.1 split. I also didn’t find any open source toml parser that couldn’t handle the specific limitation proposed for required support for integer ranges. There might be one, but it doesn’t look like that’s a problem for us to preemptively get ahead of that and remove the ambiguity.
The 1.0-1.1 split has some nuance on how tools should handle it, but from the specification side, we really should be able to point at a specific file and be able to determine if we expect the ecosystem to handle it or not.
The only way I can see getting that with the lack of a specified toml version is to require compatability with all toml versions. Doing that without explicitly saying which toml versions we mean by that opens up a future can of worms with a hypothetical toml 1.2 or 2.0 that introduces something we didn’t consider, so rather than spell it that way, we have to be more specific here.
Getting from where we are to ideal specification to implementation by tools is going to take multiple steps. I think even if we want toml 1.1, allowing it to begin with is inherently breaking because toml 1.0 can’t handle toml 1.1 files.
My plan here in (currently) two steps:
- specify the “now” as a subset that “works with what we have now” for both existing specified files and for existing toml parsers. Even in cases where the existing files don’t benefit from additional restrictions, set them for now so that tools implementing packaging standards don’t need to re-evaluate if their parser is appropriate for each new toml file specified in the future.
- Figure out a reasonable way to move the ecosystem to where writing toml 1.1 is either acceptable or not rather than ambiguous based on the features of 1.1 that end up used.
I don’t intend to try and figure out how we get out of the subset problem until the subset is properly defined.
3 Likes
If we have members of this community and that specifying/implementing TOML then maybe some form of specifying the TOML version in the file at least for new versions could be discussed in their TOML forums, with reference to our problems?
I’m not following what you mean by this, I’m not intending on specifying TOML, but specifying what the current practical limitations are on the use of TOML for packaging.
For example, packaging use of TOML shouldn’t use floats. It’s potentially lossy when the input is human written, and even in cases where it isn’t, the standard library parser for TOML cannot guarantee a specification compliant float when parsing.
The “lack of specified toml version” I referred to there also has very little at all to do with the toml specification, and more to do with the current packaging specification status quo, where no specification of files using toml for packaging require a specific toml version, nor is there a meta-requirement on packaging tools that all uses of toml are expected to be a specific version.
toml itself is versioned.
1 Like
Oh, someone mentioned that the version of TOML that a reader needs to be at is not put in TOML files in a standardised way, and yet that may help users in the future. Maybe that could be brought up in TOML standards land?
1 Like
Oh, I see what you mean now, thanks for clarifying.
That wouldn’t get us anywhere before toml 1.2 as a minimum supported version for use in packaging, as toml 1.0 and 1.1 don’t have that in their grammar. I also think it’s unlikely to end up as something toml does, as one of toml’s goals is to be easily written and modified by human authors (also a reason why toml probably shouldn’t reject a float of 1.3 for not being a representable value in the specified float format)
My personal ideal here would have been that files written by tools for other tools to consume were more rigid, even if building off of something pre-existing and standardized, such as using protobuf, but that ship has already sailed until there’s a willingness to have a breaking change (possibly something for wheelnext to consider?)
2 Likes