A PyPI resolver library in Rust

Hi friends,

we’re working on a new package manager that brings together the conda and pip packages more closely. Our package manager is called pixi.

We know that a lot of conda users also use Python depencies from PyPI. That’s why we recently made our resolver (called resolvo) generic and lazy, to work well with Python packages. We bundle that + the metadata handling in a new Rust crate called rip: GitHub - prefix-dev/rip: Solve and install Python packages quickly with rip (pip in Rust)

I also wrote a short blog post about it: Introducing rip - the fast & barebones pip implementation | prefix.dev

I would love to see people try it out or integrate it into their projects. We are going to work on integrating rip into pixi very soon and hopefully have something to show at the end of November. Until then you can try rip standalone with cargo run -- flask (or any other package). Another big topic that we will have to work on for real-world scenarios is sdist support …

13 Likes

Thanks @wolfv. I’m looking forward to seeing pixi soon. Thanks for sharing.

We can’t use it until it’s able to support custom PyPI indexes (and work without conda packages). But otherwise seems nice.

@bernatgabor rip totally supports custom pypi indexes (we tested with the pytorch one). And also does not require any conda package at all :slight_smile: It looks at whatever python it finds on your $PATH and uses that to populate the environment markers.

This is very cool and seems relevant to some recent packaging discussions [1].

It’s cool that the solver works with both indexes. I’m curious, do you think there’s a way to cross the conda-forge / PyPI boundary while resolving? I’ve been thinking about this lately and it seems doable but there are some potential pitfalls. One obvious one is that the package names aren’t exactly 1-to-1, but I don’t know how often.

I’m thinking of the scenario where I want to install A from PyPI and it requires B which is on both PyPI and conda-forge. conda install A doesn’t work because A isn’t there. pip install A will install B from PyPI (let’s assume I don’t want to do that for some reason).

Right now the solution is conda install B; pip install A but it’d be great if a tool could figure that out on its own. There are a lot of recipes on conda-forge that could be removed[2] if conda knew to install from PyPI when needed.


  1. e.g. this one and this one ↩︎

  2. lessening the small burden of updating versions and so on ↩︎

How do I install rip? :laughing:

❯ cargo install rip
    Updating crates.io index
  Downloaded rip v0.0.1
  Downloaded 1 crate (295 B) in 2.34s
error: there is nothing to install in `rip v0.0.1`, because it has no binaries
`cargo install` is only for installing programs, and can't be used with libraries.
To use a library crate, add it as a dependency to a Cargo project with `cargo add`.

Or is there no frontend just yet? I seen no sdist support for now, is that correct? I tried using through pixi, but that does build on conda, based on outputs. (In a parallel world I’d like to replace pip with rip, not sure if that’s on the roadmap).

@jamestwebber yeah, there is quite some “churn” involved with maintaining a bunch of noarch packages on conda-forge, indeed. We’re not at the stage where we want to resolve with both indexes at the same time but we think it’s in the realm of possibilities. conda-forge maintains the following list: https://github.com/regro/cf-graph-countyfair/blob/master/mappings/pypi/name_mapping.yaml

The problem is that – to build a conda-forge package – even one with binary extension, you might need some lower-level noarch package. So even at package-build time the resolver might have to pull some packages from PyPI and not conda-forge. Not sure if we want to do that given the pretty different operating model of PyPI vs conda-forge, but let’s see how these things evolve :slight_smile:

@bernatgabor
I think the rip crate on crates.io is something else, indeed! For the moment you need to pull the repository, and then run cargo run. E.g.:

git clone https://github.com/prefix-dev/rip
cd rip
cargo run --release -- flask

To use a different index, use --index-url.

2 Likes

@wolfv Is there an option to pass the path to a particular Python executable?

I think there is currently no way of specifying the path to the python executable, but the rip binary will look into your PATH variable and find the first one available.
From Rust, it would be very easy for you to give a concrete path to a Python executable (instead of searching on PATH).

I do think this would require a CFEP or something similar to define how it should work, but I think it’d be a major improvement to the ecosystem if it’s possible.

Would it be possible to install it and not run it from source essentially?

cargo run --release -- pyspark
2023-10-23T23:02:59.573715Z  WARN rattler_installs_packages::resolve: Not considering pyspark 3.5.0, 3.4.1, 3.4.0, 3.3.3, 3.3.2, 3.3.1, 3.3.0, 3.2.4, 3.2.3, 3.2.2, 3.2.1, 3.2.0, 3.1.3, 3.1.2, 3.1.1, 3.0.3, 3.0.2, 3.0.1, 3.0.0, 2.4.8, 2.4.7, 2.4.6, 2.4.5, 2.4.4, 2.4.3, 2.4.2, 2.4.1, 2.4.0, 2.3.4, 2.3.3, 2.3.2, 2.3.1, 2.3.0, 2.2.3, 2.2.2, 2.2.1, 2.2.0.post0, 2.1.3, 2.1.2 because there are no wheel artifacts available

After you do cargo build --release you should find a single executable in the target/release folder. You can copy that into a suitable bin folder.

Would be easier to just do, cargo install rip :thinking: not having to clone and copy things around. Then you can also use GitHub - nabijaczleweli/cargo-update: A cargo subcommand for checking and applying updates to installed executables to keep it up to date. From my side the project is unusable until it supports sdists :blush: because, well on any reasonable sized project there will be some dependencies that don’t have wheels. :blush:

1 Like

For sure. We’re mostly shipping the rip binary as a proof-of-concept. The main use-case is to use rip as a library.

1 Like

Understandable, just makes it hard to give feedback on it :blush: