To mitigate the effort, it might be easier to add a feature to tomlib whereby a toml override file can be used to override the data loaded from a toml file. Thus, for all such projects, they would simply replace:
data = tomllib.load(fp)
with
data = tomllib.load(fp)
data = tomlib.update(data, fp)
So, this feature would boil down to writing one update function, and then tools can opt in to using that function.
Maybe Iām wrong or misunderstanding something but isnāt
data = tomllib.load(fp)
data |= tomllib.load(fp)
equivalent and works already?
Cool! Is that all thatās needed for libraries to implement this proposal? Sorry I donāt have time to investigate, but if so, that would be a huge step.
Dunno about the proposal, but dicts can be updated either with the dict.update() method, using the {**a, **b} pattern, or the | and |= operators since 3.9 I think?
Sorry for the OT.
Oh, a dict update is probably not good enough. It needs to recursively update sub-elements
Well if it is enough, and you donāt have time to implement it, then I guess that demonstrates that even this small amount of work is too much to attract contributionsā¦
Sorry if that sounds snarky, but the reality here is that none of the tools needs someone to point out a simple way of implementing the requested feature - they need someone to actually create a PR that does so. A lot of discussions around packaging tools and features fizzle out at the point where itās been established that a bunch of people want something, and that āit looks easy to doā, but no-one ever actually tries to actually do it. And the reality is that itās only when you try to implement the feature do you find out why itās not as easy as youād hoped.
So the maintainers get a reputation[1] of blocking good ideas, users get frustrated that tools donāt care about their use cases, and everyone gets burned out until the next idea comes along. And we make no progress.
If you (or anyone else) genuinely want to progress this issue, then Iāve described most of the points that I think are relevant in the post that contained this one comment that you quoted:
Again, my apologies if this sounds like Iām dismissing your suggestion. I suppose I sort of am, but given that this thread had been dormant since last March, I think itās likely to need something more than this to have any chance of actually moving forward.
which we donāt like having, to be clear! ā©ļø
Perfectly fair points.
In the spirit of moving this thread forward, I think a first step would be to make a list of all of the requested types of overrides. That way we can ensure that whatever mechanism is implemented satisfies them all.
So far, we have the ones mentioned at the top (adding a section with a requirements; adding a section with an index-url), and the two I mentioned.
Adding sections seems easy. My suggestions require replacing elements, which should also be easy. There may be suggestions that require adding things to lists? Will order matter? What if someone wants to delete something from a list?āor delete an entry? In that case we may need a way to specify deletions.
I think if we can narrow down the scope, we may be able to get something rolling.
Honestly, I think the best way to start would be to simply implement @sinorocās original specification, linked in the first post on this thread. It may not get accepted as it stands, but working code is a far more compelling argument than any sort of debate. And even if the proposal there doesnāt do what you want, implementing it would give you a much better understanding of whatās needed for any other proposal than simply talking about it.
For a somewhat easier starting point, just produce an implementation for one tool. While doing that wonāt give you a good feel for how standardising a cross-tool capability differs from implementing a tool-specific feature, itās still a great start.
I think itās something like this. I havenāt tested it or anything so please donāt take this to be a proof of concept. Itās just an illustration of how I imagine this would work. The idea would be to bake the override process (apply_overrides in the code) right into tomllib and tomlkit, and then there isnāt so much code for libraries like poetry to add to their code.
Couldnāt you just vendor the single function?
Sigh. As Iāve said, prepare a PR and weāll see. Honestly, I donāt know but I doubt it (our vendoring process vendors packages from PyPI, not bits of code from the stdlib). Copy and pasting a function definition is a possibility, but Iām not at all sure I would be happy with taking on that maintenance burden (and I canāt speak for the other pip maintainers).
Anyway, this is exactly the sort of thing I donāt think itās productive to endlessly discuss here. Create a PR and we can see if we can resolve any questions. Otherwise, the reason this proposal hasnāt been implemented is simply āno-one has written any codeā. Everything else is incidental.
I understand what youāre getting at, but from the other side, no one wants to invest weeks writing code only for the answer to be ānoā. I remember when I worked for months on PEP 448ās implementation and the initial reaction of the core developers was to reject it. Thankfully, it was eventually merged, but it was looking for a while like all of that work was going to be wasted.
So I think itās the same thing here. It helps to get an idea of whether that work will be accepted. If we had a clear positive or negative indication of whether something was likely to be accepted, it would motivate the time investment of implementing something. On the other hand, I see your point that you donāt want to answer questions would code that you can look at. Ideally, there would be some middle ground. E.g., working from a design document.
You could vendor the entire tomllib with the changes in place if that makes you more comfortable.
Your best starting point is to look at the issues linked above to see what the various projects think of this idea. And getting a consensus for a multi-project standard is even harder to assess.
What I will say (because your comment is certainly fair) is:
(This discussion has probably taken up most of what was left of my open source time for today, so Iāll say no more for now).
Remember, itāll need docs, tests, etc - just a proof of concept isnāt enough ā©ļø
Fair enough, thanks for the explanation. I was just trying to move this idea forward in a small way.
I do not recall if it is mentioned explicitly, maybe indirectly, but if I understand correctly what you mean, then yes it is definitely what I have in mind with this.
No, not what I had in mind for the scope of this, but maybe it could be in scope. At least not what I would consider a priority.
That is not what this proposal is about. That would not work.
At this point, my attitude towards this idea is:
I encourage people to unsubscribe from this thread if they do not want the noise, which is perfectly fair. I do not think there will be any breakthrough anytime soon. If there is any breakthrough then assuredly there will be a new thread. We can also close this thread and redirect discussion to the gist instead.
Would you mind elaborating on what you need in order to implement this idea?
Not quite. As far as I can tell, this plugs into and requires changes to how the installersā package āconcretisingā logic works (i.e. going from a name to a resolved distribution) as well as dependency determination logic. The recursive merge of a mapping isnāt a thing to worry about in terms of the implementation work involved IMO.
I think this would be useful.
That said, I want to caution that we avoid doing too much design work āup frontā, before (at least) considering implementation complexity in the existing tooling.
Could we? Yes, absolutely. Would we? I donāt know, depends on the function. I wouldnāt call it vendoring at that point tho. 
As mentioned, most of the complexity of this lives in a fairly-coupled-with-core-logic parts of the codebase. A function to do a recursive dict merging isnāt really a chunk that moves the needle on implementation story much IMO.
If Iām reading the room correctly, @pf_moore is more concerned about how the core logic (that has a lot of nuance associated with it) would be updated, rather than the specific function for merging dicts. I share that with him ā how feasible this idea is depends on the use cases, the specific proposal design and whether someone is willing to put in the effort to implement this in an existing tool.
And, @NeilGirdhar was looking to help move this forward a bit by volunteering to do so! I appreciate the interest however this isnāt really a piece that moves forward in small pushes sadly.
Okay, I read through the gist more carefully and I think I see why the naive combination of toml files doesnāt currently work.
My proposal would be to do the work in multiple PEPs:
overrides.toml that simply modifies a pyproject.toml.This has a few benefits:
Yes, this needs to be split, no doubt about it.
I am against this. Abstract dependencies belong in pyproject.toml, concrete dependencies do not. Maybe you meant overrides.toml in which case, yes, that is one of the main drivers for this proposal.
Sure, why not. Not what I have in mind, but weāll see, maybe it comes naturally as an extension of what I have in mind.
That is just one aspect of what this proposal is about (But again, NOT in pyproject.toml! From my point of view Poetry got this wrong.). For me right now it is hard to see if it is a good aspect to use a starting point.
To be fair, I also kind of lost track of all the use cases I considered in this. I would need to find time to get back on this topic.
There were a couple of categories I thought of:
Make it possible to use installation modifiers that are currently global, per dependency instead. For example pipās --pre option (which kind of triggered me reviving this thread). See also how it has been added to PDM. And the big one would be if it were possible to specify one --index-url option per dependency.
Having all these as CLI flags would possible become quite unreadable, so maybe a separate file used as input to the installer would be better.
Override faulty package metadata. Typically offer a way to override eager upper caps on version constraints.