I’d like to propose a very simple rule for all packaging standards use.
Packaging use of standardized file formats must only rely on the guaranteed parts of the standard.
That’s it, nothing more to it. The goal here is to ensure interoperability remains as good as reasonably possible.
In the case where something is needed beyond what a file format offers, the needed data should be encoded in a way the format can contain reliably, and be handled at the application level. (eg, using a list of tables in toml, rather than a table of tables if the order is material) or another more appropriate file format should be used.
However, the draft of a feature of a yet-to-be posted pep in the series on variant wheels does rely on key order in toml, something explicitly not part of toml as a format. The authors don’t want it to be discussed further in the only thread currently relevant to the series.
Okay, I don’t mind expanding the scope of this slightly here.
Regarding use of standardized formats, all future use should specify the version of the format they use.
Existing toml use until such a time that the existing standards are updated should be reliably parsed under both 1.0 and 1.1 (This is effectively saying write using 1.0, parse using either, as 1.1 only relaxed the formatting)
I think is fine as an aspirational goal for writing new standards, but there are probably already lots of edge cases in exiting standards where we rely on common behaviors of parsing libraries rather than things guaranteed by the standards. I don’t want each new standard to create backwards incompatibilities with existing behavior.
I don’t think deciding on a standard would create backwards compatibility. If code is relying on an implementation detail then a potential bug already exists, and specifying a standard makes that apparent but doesn’t immediately break the behavior.
This is a principle I’d naturally try to follow, as the current PEP delegate for packaging standards. And I expect the Packaging Council would have a similar view. But everything has nuance - that’s why we have individuals deciding on proposals, after all. So I wouldn’t try to make it a formal rule.
As a general principle to follow, though, it seems fine to me. I’m not sure if that’s enough for you, but I imagine it’s as much as you’re likely to get…
It was eventually removed due to massive backwards incompatibility with existing indexes.
My concern is similar, a new spec clarifies an exact serialization standard of an existing packaging standard, and that creates massive backwards incompatibility.
A lot of the debate revolved around what CPython would support. If a packaging standard specifies a specific version of a serialization MUST be used, I think a lot of debate will be around if Python will continue to guarantee parsing that version of that serialization format.
There’s currently a situation where people working on a proposal are knowingly using something out of spec in toml, and rather than fix that before the proposal, they brought up changing what’s guaranteed in the python standard library, even though that wouldn’t be sufficient to ensure interop.
This isn’t a backward compatibility issue that brought it up, but preventing getting into a forseeable problem, so I’m fine with it only applying to anything going forward, and I’m fine with it being worded “If during the design phase, anyone notices such an issue, the design must be changed prior to acceptance”. Any existing spec issues are unfortunate, but I’m more concerned that we have people actively basing future work on something they know isn’t spec compliant.
That’s not fair. Or at the very least, it’s not how I read the situation, assuming good faith from everyone.
They are considering how to include data that logically should be an “ordered dictionary” into a file that’s currently specified as TOML. In doing so, one possibility they considered is to require consumers to use a TOML reader that preserves table order - something they acknowledged the spec doesn’t require. As part of the investigation, they asked if it would be reasonable for tomllib to document that it preserved order, as an implementation-specific extension to the standard.
Nothing has been set in stone, and the proposal is at a sufficiently early stage that it’s not even being discussed on dpo yet. When it does, these sorts of issues will be flagged and addressed at that point - that’s literally the point of having a standards approval process.
You’ve now raised this “problem” across 3 different threads, and got essentially the same answer in all of them. I think it’s probably time for you to drop the issue, and accept that the standards process will handle it. You’ve raised awareness, which is essentially the only useful thing you can do at this point.
The closest thing I got to an answer was that they were intending on proposing a design based on the outcome of the decision on if the standard library would guarantee non-standard behavior.
That seems like directly stating they intend on using nonstandard behavior if they can, and I don’t see another way to interpret that.
In any case, I think this is a real problem that’s worth heading off even if that wasn’t the case; Is there a reason to not go forward with “when we say we’re using a format, we use the format”?
The most obvious reason is that we don’t have a governance process that lets us impose constraints on the upcoming packaging council. Nor do we have a process that lets a community discussion dictate how I (as PEP delegate) decides on a PEP.
You got the best answer you’re likely to get. I strongly suggest you stop pushing for more.
Apart from anything else, you’re getting (in my view) perilously close to CoC violation territory, by repeatedly suggesting that the people working on wheel variants are deliberately attempting to subvert the standards process. You need to stop seeing conspiracies where none exist.
And while I don’t want to continue this debate any longer than necessary, I’ll point out that:
It’s entirely legitimate to propose a standard that says tools must use certain libraries (or libraries with certain non-standard extensions). It’s extremely likely that such a proposal will fail, but it’s not wrong in itself to propose such a thing.
Not everyone has a deep understanding of the standards process. If someone is proposing something unreasonable, we should discuss their requirement with them, and help them find a workable solution. Leaping straight in with accusations of bad faith is not the right way to help people to contribute.
The discussion you’re referencing was happening outside of dpo. Many contributors are reluctant to engage with dpo except when they have to, and have quoted the hostile atmosphere as a key reason for that. This sort of thread is precisely what causes that sort of reaction, and we’re not helping the community by driving people away like that.
That’s a good idea. I’d extend it to allow recommended parts, which the packaging standard should explicitly document.
A formal standard should not say “TOML”, but something like “TOML 1.1.0. Values of number fields specified in this standard must be Integer (not Float) in range(-2**64, 2**64).”
So much legalese :(
Relatedly, I filed a suggestion for TOML to standardize some details that might hurt strict interoperability. (Hopefully I’m not rude there; I don’t know their processes.)
I also think installation tools should not be expected to validate file formats by default – that’s extra code and compute that’s not needed to put code on my machine.
Packaging standards tend to be less formal than that. This causes problems for us (the TOML 1.0 vs 1.1 issue being one such), but it’s a (probably necessary) compromise. The PEP process is already seen as off-puttingly heavyweight for many contributors, expecting the level of formality that this would imply would probably be the breaking point.
Until now we’ve relied on me and Donald (as PEP delegates) being sufficiently nit-picking to pick up anything like this in the review/approval process. I’d assume the Packaging Council will continue to do that.
It’s a compromise (there’s always nuance!) If invalid data could cause the wrong thing to be installed, we need to validate as a matter of security. If it’s just to act as a belt-and-braces compliance check, you’re right, that’s not the job of an installer.
In the context of this thread, I think the key thing is that standards should use data structures provided by a file format in the way they are intended to be used - so numbers for numeric data is fine (even if there are edge cases that we don’t explicitly set out). But (to pick up on the example that started this discussion) TOML doesn’t provide a data type for “ordered key/value pairs”[1] - the “table” type isn’t ordered. So we shouldn’t use it to represent ordered key/value data, regardless of what individual implementations might do.
toml 1.1 changed language that made integers less automatically portable, does that make toml numbers something we can’t use in 1.1+, do we just place a restrictions that the parsers used must be able to handle the recommended ranges as @encukou says, or do we handle it per packaging specification? The standard library parses using 1.1, so any issues with portability with 1.0 could pass design unnoticed right now.
The other argument in favor of stronger language here that applies to all packaging standards is that each proposal wouldn’t need to repeat this work unless they were proposing using a format not already used by packaging standards.
I see that you’ve already indicated that there isn’t a process for constraining other packaging processes right now, but standardizing the features allowed/required in each format’s use within packaging once could reduce the work of each proposal that needs to use some format for data interchange, and in exchange for doing that work once, we also can be more certain that if someone picks up a parser that can handle what we say it needs to, they can implement any of the related packaging standards using that parser.
I guess if someone wants to do the work - i.e., write a packaging meta-PEP that defines all the generic rules, or write a series of specific PEPs tightening up the rules in our existing standards - then I don’t have any objection. But unless that happens, I’m not sure what anyone wants here beyond the general “I understand the issue and now that I’m aware of it I’ll keep it in mind for future standards” statement that I’ve already made[1].
Feel free to ask the Packaging Council for a similar statement once they are set up and have a process in place for such requests to be submitted. ↩︎