`poetry add`, but for PEP 621?

One of Poetry’s “killer” features is the poetry add subcommand, which can be used to add dependencies to Poetry’s custom pyproject.toml sections. From Poetry’s docs:

Adding a few dependencies at once:

poetry add requests pendulum

Adding dependencies, with constraints:

# Allow >=2.0.5, <3.0.0 versions
poetry add pendulum@^2.0.5

# Allow >=2.0.5, <2.1.0 versions
poetry add pendulum@~2.0.5

# Allow >=2.0.5 versions, without upper bound
poetry add "pendulum>=2.0.5"

# Allow only 2.0.5 version
poetry add pendulum==2.0.5

(Similar art exists in the form of cargo add for Rust and npm install/npm save for Node.)

Many developers find the experience of adding dependencies via the CLI more pleasant and less error-prone, and having a CLI allows the associated tooling to (optionally) do other opinionated things, like keep the dependency list sorted.

So, this got me thinking: does it make sense to offer similar functionality for standard PEP 621 metadata, specifically the project.dependencies and project.optional-dependencies fields?

Here’s a rough sketch of what a CLI for that could look like (pep621tool being a placeholder):

$ # adds requests to project.dependencies, no pin
$ pep621tool add requests 

$ # adds example to project.dependencies, pinned to 1.2.3
$ pep621tool add example==1.2.3 

$ # uses normal extras syntax as well
$ pep621tool add example[dev]==1.2.3

$ # adds build to `project.optional-dependencies.dev`
$ pep621tool add build --extra dev

$ # standalone sorting, or sorting while adding
$ pep621tool sort
$ pep621tool add --sort requests

As of writing, the technical aspects of this should be feasible: PEP 621 itself is accepted and is being increasingly adopted, and packages like tomlkit offer TOML modification with edit points (i.e., comments and user formatting).

I’m curious to hear what people think of this! One significant reservation I have around this is contributing to the “toolbag” problem that Python packaging already has by adding yet another distinct tool for package development. Given that, I’m interested to hear thoughts on whether it makes sense to add this to an existing frontend tool.

3 Likes

Forgot to mention: other prior art also exists here in the form of pdm add, which (1) does support PEP 621, and (2) has similar CLI syntax to the idea above.

1 Like

I would hope that as poetry moves towards more standards compliance (a slow process because of backwards compatibility issues, but I believe that’s their intention) poetry add will change to modify the standard metadata rather than poetry’s custom data. And as you say pdm has this already. There’s a good case for requesting it for hatch too.

In other words, I think adding this functionality to existing tools (which we’re hoping will eventually converge on “one unified tool”) is probably a better idea long term than having a separate tool for each feature.

Having said that, a pyproject-tool (general advice is to not use PEP numbers in tool names) utility might be nice as an interim solution. And it’s not as if you need permission from anyone to write it :slightly_smiling_face:

3 Likes

Yeah, that’s the hitch for me – it feels roughly equally silly to (1) have a standalone tool when the packaging tools are converging on the standards anyways, and (2) not have this functionality because my packages generally use a “barebones” setup (i.e. no poetry or pdm or hatch, but just flit as the build-backend).

Indeed :slightly_smiling_face: – I’m going to think on it a bit more, but maybe there’s a happy middle ground here (something like a pyprojectlib that pdm, etc. can rely on as a single implementation of “modifies pyproject.toml in various ways”). Then again, maybe that’s drastically overthinking/over-engineering the problem, given that modifying a TOML file isn’t very difficult.

I am personally very guilty of trying to generalise things as soon as possible, and I agree with your assessment; this is mainly just editing TOML in a CLI-friendly way, it doesn’t need to be shared between projects. I’d be tempted to ping @ofek and see whether he’d be interested in this kind of CLI for Hatch. I know that in the long run, the missing feature of Hatch (and many of these tools) is a standardised lockfile format. However, I could see this being added in a non-locking manner to start with.

Just dumping my thoughts at this point; I wonder what would happen if Hatch adopted PDM’s lockfile as a temporary interrim. Much of the machinery should be re-usable, i.e. the interface for a lockfile resolver. However, I am certain that Ofek doesn’t need any more work to be dropped on him, so perhaps he’d prefer to just wait until we’ve settled on what a lockfile should include.

I hadn’t even considered this as a locking operation, simply a way of adding a dependency (which then gets resolved on install).

If you want a locking tool that isn’t part of a “full workflow” solution, you should probably look at pip-tools, which I understand already fills this niche pretty well.

Sure. I don’t think it is a locking-only feature, but I am of the understanding that the locking aspect is a blocker for Hatch, so I assumed that is why Hatch doesn’t yet have this API. This might be a logical jump too far; perhaps @ofek doesn’t want Hatch to have this kind of API! I’ll let him speak to that.

Yeah, my first thought was just the “naive” functionality of adding a dependency, without any consideration for lockfiles (I’m a happy user of pip-tools :slightly_smiling_face:).

If this is something that Hatch grows in the near future, that’d likely be enough to convince me to move most of my company tools over to it!

So, this is tied to locking in that the operation that is actually performed with add differs depending on the presence of a lock file.

That said I do plan to add this by PyCon (maybe the week of for extra visibility) along with the introduction of locking plug-ins (the first of which will wrap pip-tools) and the concept of workspaces which will be useful especially for monorepos.

I was trying to wait for Brett to write that lock file proposal but it doesn’t seem like that will happen soon and now people are considering turning pip into an everything tool which I view as deleterious, so timelines are forcing my hand lol

3 Likes

Having a command line tool that could manipulate PEP 621 dependencies would be an enormous booster for downstream redistributors such as Fedora.

We often need to patch/sed dependencies (e.g. when we tested that a particular software works with an older version of a library that upstream decided they no longer support but we cannot update it yet or when we remove optional test dependencies). Having the ability o do this in a semantic way instead of hacky sed replacements would be very nice.

%prep
# remove coverage-related test dependencies
pyproject-tool remove --extra test pytest-cov codecov

# allow a lower version of packaging before we can update it to 23 in Fedora
pyproject-tool remove 'packaging>=23'
pyproject-tool add 'packaging>=21'
1 Like

Yeah, don’t wait for me. I am moving forward still, just not at lightning speed (unless someone else wants to take over pushing WebAssembly and specifically WASI in Python forward? :wink:).

1 Like

This is an innocent sentence with a large amount of complexity behind it. That is because TOML is supposed to be human-centered (“A config file format for humans”), and so you absolutely have to be able to maintain things like comments in the toml file correctly.

This is not a trivial thing to do, because when you just .load() the toml file and then .dump() it again with an additional dependency, you’ve lost all those comments, plus other things that people have opinions™ about (order of sections, inlining of tables, etc.).

It’s a much larger space than just reading a toml, and the main reason why the 3.11 stdlib-implementation comes without writing capabilities (see e.g. this thread). This info might be out of date, but a year ago, only tomlkit supported style-preservation (and perhaps unsurprisingly, this is what poetry uses; indeed it’s written by the same people).

Adopting something like this IMO requires either:

  • having tomlkit as a dependency everywhere (yet more bootstrapping problems)
  • getting a style-preserving TOML parser/writer into the stdlib
3 Likes

Hmm, why would this complicate bootstrapping? I don’t see it being part of a build backend or the barebones installer. Note that backends with a frontend counterpart do separate these (compare poetry with poetry-core, hatch with hatchling, flit with flit-core, etc.).

3 Likes

Yeah, I envisioned this as being frontend-only; plus, it looks like tomlkit is pure Python with no dependencies, so vendoring it wouldn’t be the worst if absolutely necessary (and is well-trodden w/r/t breaking cycles in other Python packaging tools).

Oh just FYI when I mentioned I’d be adding commands for adding dependencies and therefore modifying TOML although that implies the use of tomlkit I plan to write my own parser that only supports modifying arrays because I am deeply discontent with the bugs I’ve encountered in that library and I would not want to expose users to that nor implement a postprocessing step to fix style.

2 Likes

I plan to write my own parser that only supports modifying arrays

That’s not as simple as it sounds since you still need to retain the complete document format when you write it back, even if you modify one part of it.

because I am deeply discontent with the bugs I’ve encountered in that library and I would not want to expose users to that nor implement a postprocessing step to fix style.

Good thing tomlkit is open source and is accepting contributions. Feel free to open issues about these bugs and even contribute pull requests to fix them, it will avoid duplicate efforts since we have a common goal in the end :slightly_smiling_face:

2 Likes

On another note, I plan a partial rewrite of tomlkit to solve some known issues around formatting and how nodes are stored internally. I don’t have an ETA since I don’t have a lot of time to dedicate to it at the moment but hopefully it will come soon.

Fyi, I opened a request for this in Hatch a while back. Glad to hear it’s likely to become reality. Feel free to add Hatch-specific contributions there:

1 Like

Oh for sure that will be a multi-day effort :slightly_smiling_face:

If I have time I might do that indeed! In my mind writing logic for a very specific use case sounds like less time investment than getting up to speed on a code base.

FWIW, part of the power of poetry add is that it checks whether the new dependency is compatible with the existing dependencies and will let you know immediately whether it is compatible. A poetry add that only modifies the file and doesn’t check for compatibility is not really very useful.

I do like the idea of a smaller utility that just does the one thing.

1 Like