You’re contradicting yourself here. You never addressed one of the major concerns of this topic: the pain of having to migrate from PEP-508 to the TOML format, and the strain that puts on all the tool maintainers. Furthermore as discussed above the TOML format cannot validate just by itself the format (e.g. version constraints need their own DSL), so in practice offers a lot less benefit than you’d think: user still has to learn a DSL and tools still need custom parser beside a plain old TOML parser to manipulate it.
3 Likes
I was afraid that it could be read like this 
To be more precise: The dependency definition acts as an interface. So it affects all users and they have no they have no other choice. So consistency must be within the specification itself. Having a mixture of exploded tables and PEP508 would violates this.
Tools build on top of this interface can have their own opinion about how the user experience should be. Users can choose which one they like most and use it.
Work has to be done when PEP621 is official. And that’s independent from the choice how the dependencies are described.
I believe this is not discussed now. I would like to see a spec, which defines mandatory and optional fields and a way to integrate custom fields with a defined prefix e.g x- to mark them as custom. Validators should then raise an error if fields are found that doesn’t match these specs
We already need to have custom PEP-508 DSL for version specifiers at the very least, so considering we’re violating this constraints we might go down the path of least resistance, and go down the PEP-508 entirely for it. Has the added benefit is what people already used for the last 10 years. I feel you’d had a better case if TOML could validate in table format the entirity of the PEP-508 domain, but can only do very limited part of it - exploding key-values into a table.
As someone who is maintaining multiple packaging projects let me tell you that your chances of me updating projects are better if the amount of work done is as low as possible. So going down a path of status quo for dependencies specification/parsing frees me to address other outstanding issues. PEP-517 for editable installs e.g. I consider much more important.
6 Likes
100% agree with this point. If you don’t want to depend on the packaging library (“hacky regex” is a personal choice, not a forced requirement), you need to fully explode the definition into TOML. Once you do that, nobody is going to be happy with translating all their version constraints.
This is not exactly true since version specifiers come from PEP-440 and are not tied to PEP-508.
Well then what’s the point of trying to specify another metadata standard in this case?
Not the totality but most of it can be represented as TOML, only more complex cases with non trivial markers – which are rare from my own experience of analyzing a lot of packages when developing Poetry – are not easily represented as a full TOML specification, that’s why we introduced the marker property in Poetry to circumvent that.
Like people won’t be happy to adopt the new metadata standard at first as well but there is no reason that it won’t be done eventually, so we might as well go all-in on this new standard.
We shouldn’t make bad changes just because we’ll win in the end. The point is to make things easier for users, not to rule over them.
Depends on how we define it in the spec. 
We would have to specify whether unrecognized keys were an error or simply passed on. Since a key argument from the exploded table side is validation, I suspect we would want to make it an error.
Whether it’s a bad change or not is subjective, until we have proper data to back it up. I think it would be a good change overall and considering the traction Poetry is gaining I think a lot of people think so too.
We don’t have data to make this claim. Poetry might gain traction for other reasons than how it treats requirement specification, just as we don’t have data on how many people decide to not use Poetry because of it and walk away. Considering we can’t agree within ourselves if TOML table is better or not than PEP-508, I’d guess would we have a survey users would be similarly split.
2 Likes
Maybe but that at least shows that people are willing to make the change to another specification if necessary.
Which we know, but this is a mindset we should not take. If we can’t show through popularity that a particular scheme is better, we need to decide based on other criteria.
The pre-existing spec argument is good enough for me. Reference it and we don’t have to write a new one.
ALTERNATIVELY, and this will be controversial, declare dependencies off-limits for PEP 621 and tell backends to do their own thing. (Which I assume will be PEP 508 as that’s their output format and what they all already use, unless Poetry has become a PEP 517 backend while I wasn’t looking.) If there really are strong enough reasons to prefer the exploded table format, people will gravitate to those backends over time, as the theory goes.
Poetry has been a PEP 517 backend for quite a while. And currently there’s really only one other backend that uses TOML for specification (Flit), so I wouldn’t be that sure what you predict would really happen (although of course it may).
1 Like
It’s only potentially controversial from two points that I can think of.
- Will it confuse users if dependencies are left out?
- In a theoretical future where we reach agreement, how do we handle
dynamicfor the agreed-upon field(s)?
Otherwise if we can answer those two questions I’m okay with leaving this out for now and having the whole PEP be provisional until this topic is settled and another PEP supporting SPDX licenses is accepted.
Can another requirement please be defining a concrete way to settle?
No more than the current situation does
Nobody has been arguing about exchange formats, only user written formats. So I imagine the theoretical immediate future looks like “requirements are always dynamic; consult your backend docs to see how to specify them” and the further out future looks like “in metadata version X.Y you can also specify them statically as XYZ, or add the ‘dynamic’ marker explicitly”.
Or we could just always require the dynamic marking from the start, and make it an error to omit it for now?
If you can figure that out, sure. But that conversation happened in this thread with no real consensus on that question either.
Yeah, that’s an idea if we can agree on the field’s name at least.
Are you suggesting that this is dead in the water until we get universal consensus on the dependency specification issue?
From my look at this, I would say there’s a weak preference among people who have weighed in in favor of using PEP 508. As I’ve said before, I think there’s good reason to consider PEP 508 the default solution, so I think we should just move forward with PEP 508 and add a section to the PEP about why it was a tough choice.
5 Likes
No, not at all. As Steve suggested, we could leave this out to start to get feedback from the community and accept the PEP as provisional. My point with that statement is that we do not have a clear way of finding clear consensus ATM, so either we put this part of the PEP on hold until we get that consensus, or we just choose a solution and see if the PEP will get accepted or not based on that choice.
Another option would be to support both approaches provisionally and see if the community trends towards one or another and then ditch the losing idea and write a tool to transition folks to the winning format.
I agree that seems to be where the thread has ended up, but it’s the usual question of whether we are we a proper representation of the community? 
Basically I think we are stuck either punting on this by accepting the PEP provisionally and seeing if the community expresses a clear opinion or we propose PEP 508 and hope for the best.
Actually - who’s the PEP delegate for this PEP? Normally it would probably be me, as it’s an interoperability standard. But as I’m an author, that may be inappropriate. But we should get a PEP delegate, unless you’re thinking of proposing it to the SC (which would be unusual for a packaging PEP). You may have painted yourself into a corner by having so many authors
Don’t have one yet.
Yeah, I should have left someone out in that case. Donald isn’t a co-author.
As I said, the other option is provisional acceptance with neither/both options and see what gains traction in the community. Or at the next in-person PyCon US we do a keynote on this and poll the audience live to make a decision. 
(BTW, there’s historical precedent for that approach as that’s how Guido solved a logjam on the ternary operator’s syntax.)
1 Like