Regarding the ‘editable’ metadata, the main issue I have with it is that it’s a piece of declarative configuration which tells all consumers how to do something (in this case, how to install a package from a path).
Instead, I propose the config takes a step up to a more abstract concept, and rename ‘editable’ to ‘sibling’ (name to be bikeshed), where installers get to define what they want to do with that (right now, we know of “ignore” and “install as (tool-specific definition of) editable”).
This means the developer to still indicate projects in monorepos etc, but allows users to disable editable installs (or enable: up to the tool which is the default).
The latest version of the proposal has both include and path, but they’re mutually exclusive. Is there opportunity to allow them to combine to pull dependency groups from other projects (only directories because groups aren’t in distributions)? Would this make the mental model too complex for users?
I have considered ways of phrasing editable such that it’s very explicitly purely optional and some tools are free to totally ignore it. I think renaming it is more or less an obfuscated version of that, and it would be more harmful than helpful, as it becomes an “editable but not really” flag.
IMO, the decision right now appears to be between
include editable (possibly with language like the current draft, which makes it valid to ignore it if a tool must)
don’t include editable, but have a good plan for how it can be added to a dependency in a tool config
don’t include editable, with no further considerations
I’ve considered ideas like letting tools embed additional metadata in dependency groups, and I don’t think they lead to greater clarity or a better user experience (at least, none of what I’ve thought up is better).
If some tools do an editable install and some do not, users will always have the question “why?”
I don’t think that question is escapable – I just want to have a good, coherent answer.
Since this is a new thing, it could be decided to not have both. It really comes down to you as the PEP author and what you think makes sense. But it’s also fine to say, “you can have either, but you can’t mix them”.
This is when you get into introducing something like * as a valid version specifier.
I don’t think you explicitly state that anywhere, while the discussion here has gone back and forth on that topic. Throw in the section on only-deps talking about it and my brain just assumed that mechanism was there. I think having PEP 735 – Dependency Groups in pyproject.toml | peps.python.org also not lay out exactly what top-level dependencies would be expected to be installed also doesn’t prevent one’s mind from wandering to other conclusions.
But then this raises the question of how does one say I do want [project.dependencies] to be included? Is the expectation that you say that when you invoke the installer (which is totally reasonable)? I feel like the PEP doesn’t walk through some typical scenarios where people might do things with dependency groups like:
Install the project and it’s test dependencies
Install the stuff to build the docs
etc. and show what would be expected of installers to do to handle those situations. Otherwise the assumption is the metadata has to reflect all possible scenarios instead of saying installers have flexibility to add on some functionality that is purposefully left out of dependency groups.
I also now don’t quite get the point of only-deps if it’s restricted to path situations. If that’s restricted to dependencies that I’m explicitly pointing at, then is the assumption that only-deps will be used when you somehow get the code from a directory on to sys.path in some other way? pip install --only-deps makes sense to me as that’s providing a way to skip installing my project’s own code. But using this on other code I’m not as convinced (especially in the face of editable).
It seems the PEP wants to be able to say:
Install this dependency
Install this dependency, but without having to copy the code over (i.e. editable)
Just install the dependencies of this dependency, but not the dependency itself (i.e. only-deps)
I’m just not quite seeing the use case for the latter (and I’m personally still fine with leaving out editable, which I discuss below). I guess I’m not seeing the motivation for only-deps.
That’s not my personal take on this; it all flows the way I would expect. With extras being a public API for the package and dependency groups being private, I wouldn’t expect a public thing to be able to refer to a private thing. With dependency groups most likely getting intermingled with extras in development scenarios, I wouldn’t expect anything oriented towards development to need to be public, while I can see dev stuff needing to install things users can install.
I think a key thing to keep in mind is what must be in the proposal for it to be successful/functional, and what can be in the proposal to facilitate use cases, but are effectively optional and can be added later if need be? For instance, is there anything stopping Poetry and PDM from having an e.g. tool.pdm.dependency-groups.editable array where they list which dependencies should be installed in an editable fashion? While that shifts to side-loading this information, nothing in the proposal would stop that either. As long as the PEP doesn’t dictate how an installer should install something and simply focuses on the what, then anything additive like doing an install in an editable fashion doesn’t fundamentally need to be in this initial PEP.
Same reason some tools used to have dependency groups and others did not: not all tools have the same feature set and that’s okay (else the tools are the same and they are just named differently ).
This coupled with a recent comment from Paul made something click in my mind that did not previously.
Not just with the editable topic, but I think there is a push as of late to standardize/generalize all possible features that someone might use. I am uncomfortable with this and I do not think users desire it either actually.
It looks more complicated than the current proposal though, as a format.
I think that the current PEP would be problematic for this, in that it’s a little unclear how to refer to path dependencies. But introducing mandatory name fields for path dependencies would be one way to tidy that up.
Before removing editable, I would like to make sure we’re creating a situation which is conducive to tool authors applying this kind of additional metadata.
Reading up on npm’s peerDependenciesMeta field has made me more comfortable with this sort of approach, as they’ve already demonstrated that it can work – and I don’t see tons of SO tickets full of confusion about the meaning or purpose of that npm config.
Regarding the push to standardize features, I think all I can say is that I see the same basic problem – non-package dependency data – being solved in many different ways across the software I work on and maintain.
I experience some harms from there being no standardized way to declare, manipulate, and view these data, moving project to project and having to adjust to each one on the fly. It also limits my ability to write update tools and other handy utilities.
These are not colossal issues for me, but this seems like a case in which some unifying vision could do a great deal of good.
Standards serve not only to codify existing practices, but also as a way for us to set direction and lead the community.
I suspect this push is in reaction to the user surveys–the oft-cited “there are too many ways to do things” complaint. Of the various ways one might work on fixing that, designing new standards is probably the most fun and interesting[1]
Not meant as a dig at Stephen or anyone else doing it, I think it’s just a natural response to the situation.
There’s also the simple fact that I want the PEP 722/723 behavior badly enough that I had my own script for it before I learned about pip-run. Some of my colleagues are not comfortable with making pip-run another required part of our developer toolchain (“too many tools”, which I think is a reasonable criticism), so they want pipx support. Now pipx support is blocked on 723, and 723 is blocked on some resolution of this issue / the [run] table.
I’ve tried not to let it influence this PEP too strongly, since I think it’s better to solve something broader, but the relationship with 723 is stated amongst the use cases. (And I have some thoughts about how the specs will integrate and 723 will update, but that risks going pretty far OT.)
So this spec is an attempt to directly address a blocker for concrete improvements.
Like I say, the scope has expanded – but that’s within my prerogative as the author. I think a [run.dependencies] table would be a mistake, as it’s too narrow.
FWIW, I think the discussion here has already established pretty clearly (for me, at least) that we’re never going to have a simple [run.dependencies] item in pyproject.toml of the form PEP 723 describes…
I don’t think the issue (at least, not the one I have) is with the idea of needing a standard. It’s more about making sure the standard is grounded in real-world experience with how current tools are used to solve the problems we’re trying to address. This is why I want more information on how the PDM and Poetry editable install capability is used in practice - we need to know how the current solutions work in the real world before we can design a replacement.
My call for use cases was intended to say, let’s see what people are doing with what the packaging ecosystem already provides, and look at where the pain points are and whether a standard could help. But it seems to have been misinterpreted at times to mean “let’s look at what people are trying to do and design a solution from scratch”. If we take the latter view, we’re ignoring valuable experience with what actually works (and what doesn’t).
So, for example, when I look at Rust’s workspaces (which I assume are the inspiration for hatch) I have to say that they look very like how I imagine a Python monorepo works. So I would argue that we explicitly shouldn’t try to solve the issues with monorepos before we see how hatch workspaces turn out in practice - think of hatch as doing some UI research for the ultimate standard around monorepos, if you like
And if that’s the plan, then ignoring monorepos, what are the compelling use cases for path dependencies, editable installs and “only deps”? If they are only useful in the context of monorepos, can we omit them while leaving the option open for a future standard to add them if workspaces turn out to be a bad fit for Python?
I might be missing some context but how is 723 blocked on this? I thought it was only blocked on the [run] table and no one has had time for that and this PEP just happened to come up around the same time.
Am I wrong? If so, how would a script benefit from dependency groups? I see no use case.
So what do you feel has been established solidly enough by the ecosystem to warrant standardizing so we are not specifying the same thing in a bunch of different ways in the context of this PEP? And then can we design it to be simple enough to use, while being flexible enough to build on top of as we establish more widely accepted practices as time goes on?
We have talked about trying to standardize at least the idea of what requirements files represent as an easy way to write groups of dependencies. But we have also talked about how some of the concepts and features supported by requirements files are very specific to pip and not at all standardized. So what exactly do we consider “widely accepted via convention/use” enough to warrant standardizing how everyone should write the same thing down now instead of waiting longer to standardize or never standardizing?
If you were asking me, I would say we agree as a group that the following features are widely established and used:
A way to reference other groups of dependencies (i.e., -r in requirements files)
I think everything else as specified by Requirements File Format - pip documentation v23.3.1 could be viewed as out of scope for now if we don’t feel there’s enough agreement around ubiquity/use of a feature.
I do think we should design a solution for the two features above that allows for future expansion, e.g. editables if Ofek or someone else doesn’t come up with a workspace solution that’s more powerful than editables, but I also don’t think we have to bake that support in now. This is why my initial proposal at Projects that aren't meant to generate a wheel and `pyproject.toml` - #173 by brettcannon which inspired this PEP was rather simplistic in terms of feature scope as I thought it had a high chance of reaching agreement on the feature set (I’m ignoring the format question).
I don’t think it’s blocked by this PEP as much as potentially informing PEP 723.
Hidden to avoid going OT as it's not critical to this PEP, but left here to explain why this PEP might inform PEP 723 ...
If this PEP gets accepted, then there’s a [dependency-groups] table. To me, that suggests we either open up project.dependencies by dropping the name and version requirements for [project], or we add a top-level dependencies key. There’s a third possibility of allowing folks to mark a dependency group as the default, but that feels like purity over practicality to me as that’s probably more verbosity/complexity than is required for that use case.
If this PEP gets rejected, it just means we still have to decide how to represent the runtime dependencies in pyproject.toml that still works for single-file scripts for PEP 723.
Your two points seem right to me. And I was (and still am) broadly in favour of your initial proposal. I’m mostly unconvinced by the extra features that are being discussed here (including editable installs and path based requirements). I see a lot of discussion, but not much consensus. Designing for future expansion seems prudent, though.
Beyond that, I don’t have clear answers. My personal needs are simple enough that a pretty basic proposal would work fine for me. And I’m not particularly familiar with the complex workflows that need the extra features we’re talking about - my knowledge is mostly based on what’s come up in pip’s tracker (in particular around requirement files), and that’s quite a limited perspective.
Ultimately, though, what goes into the PEP is up to @sirosen as the author, and I don’t want to give the impression that I’m requiring or prohibiting specific details here. If the PEP is mostly solid, but includes a feature I’m not personally happy with, but which has broad approval from the community, I’m not going to reject the PEP just because I, personally, don’t like that feature.
My advice is to keep things simple and uncontroversial if possible. And if it’s not possible, to give strong, objective arguments based on real-world use cases for features that have broad support from the community.
I’m pretty heavily weighing removing path requirements entirely, and merely making sure that we leave room for them in the future.
That pretty neatly removes a wide swath of issues and points of contention. It’s a little disappointing (I put forth the inclusion of paths because I think they have utility, of course!), but the removal feels right now like the right choice in terms of building consensus.
There’s still, IMO, a ton of value in simply having a standard way of writing PEP 508 specifiers.
Regarding the relationship with 723, perhaps “blocking” is too strong. 723 needs a formalized way of writing down dependencies for non-package projects. I find it unlikely that we would want more than one solution for that. Either dependency-groups or run, not both.
I don’t see a way to finalize that spec while this effort is active. Maybe my perception here is wrong.
FYI I’m not going to do a competing PEP, but if Stephen’s PEP gets rejected I will consider writing up a variant of my proposal.
I would try to view it less as disappointing and more as a punting/delay on tackling paths. I think the key point is separating what can move forward now versus what could move forward later with more discussion. As you know, it’s easy to get side-tracked, so keeping it small and focused has value. Plus you can focus on getting your first PEP accepted and less about what specific features land.
Luckily it’s orthogonal (mostly; probably bikeshedding on names and where in the TOML file), so I think we can also tackle that issue separately/after this.
Thanks for giving me the time and space to do this! The current PEP is probably on track back towards something quite like Brett’s original suggestion, so it may feel like we’re getting the same proposal via the scenic route.
I therefore want to make all of the time and care that everyone has shown here pay dividends by continuing to contribute beyond this PEP, into the future!
Thanks for suggesting this reframing! It helped me clear my head and commit some more time and energy to the PEP right now.
I’m narrowing the scope to remove path dependencies. Not only does this have some of the obvious benefits, like dodging debate of editable, but it also better leaves room for path dependencies to be formalized in the future. If that uses a string format like PEP 508 URLs (e.g. foo @ ../foo-dir/), it will fit into this spec seamlessly. Trying to include them now means making decisions about how to represent a path dependency which are mostly independent from the core issues here.
I may – only after this effort is over – want to open up a conversation about formalizing path dependencies.
I will need to edit how the PEP presents itself vis-a-vis PDM and Poetry as part of this. The current text states that the spec is trying to allow declaration of effectively the same data as these tools. The newly refined goal is to allow declaration of a standardized subset of said data.
To call out a particular item, there is a rejected ideas section which covers the rejection of local paths as an inclusion. If anyone feels that this section insufficiently captures the major results of this thread’s discussion vis-a-vis paths and editables, please tell me and we can adjust.
This PEP has now had ~1 week without comment. I hope that the main reason for that is that it has now been pared down to the most essential features, and that it does a good job of explaining why some ideas are excluded.
Some of the text can be refined a bit further before submission, so it will probably be at least another week or two before I submit it. I believe that’s enough time for any final comments or thoughts people might want to share.
There is one bit of guidance I’d like to ask for, ideally from @pf_moore or @brettcannon . I would like to have buy in from packaging (I can improve the reference implementation and submit a PR) and at least one relevant tool (pip if possible) before submission. The proposal is much stronger if it has already been reviewed and co-signed by tool maintainers as realistic and desirable. Is this thread the best place to pursue this topic, or is there a better approach (DMs on this forum? Issues on the appropriate trackers?).
Am not Paul or Brett, but am a maintainer on both the projects mentioned: issues to them, with a link posted to the issues from here would be appreciated!
).