Sharing this tool as it seems relevant to this topic:
5 Likes
Very interesting. And Armin has built lots of cool stuff. The rationale in the README and “Should rye exist?” are interesting reads.
Quoting some bits:
There has been a lot of interest in this over the last 48 hours which I did not expect. I am however not particularly interested in further dividing the community, so in some sense I’m trying to figure out what an appropriate response is at the moment.
6 Likes
Feels like GitHub - cnpryer/huak: 📦 A Python package manager written in Rust inspired by Cargo. would be competing with this tool.
3 Likes
Apologies if posting on the discourse somehow made this problem more frustrating for you. I suspect it had only a minor effect and most of the interest came via Twitter or Github or Hackernews.
Personally, I don’t think you’re further dividing the community. You are scratching an itch, which many in the community also have. If nobody else had this problem, there wouldn’t be so many similar projects and so many conversations and debates.
e.g.
I’m optimistic that things are moving in the right direction. The growing list of similar tools and the shrinking list of gaps gives encouragement that eventually standards and a single tool will emerge as the default.
If a lockfile standard arrives and a dev dependencies setup arrives, then many of the tools become standards compliant and the community just needs to pick one. Pybi and standalone python builds would be a cherry on top in my opinion.
1 Like
With a language as old and widely used as Python it is difficult to quickly generate a consensus.
I was listening to Pete Cannon on the Real Python podcast talking about his PEP for lock file formats.
He actually talks about how difficult it is to do the Rust/cargo process because of the peculiarities of Python and the numerous edge-cases it has that a language like Rust does not.
I think if you are interested in the issue that you check out some of the current PEPs on the topic and see what you can do to help out.
I don’t think it will be as quick a process as you might find elsewhere though 
1 Like
I wanted to post an update here. Over the last 9 months quite a few improvements went into Rye and the latest version (0.21) also tries to push this a bit further. It now ships ruff as linter and formatter and makes these commands available as well. It also has a much improved installation experience which gets a user running much quicker.
I would be super curious to get folk’s feedback on this.
There is also an updated screencast showing a bit how it works including the installation: https://www.youtube.com/watch?v=q99TYA7LnuA
7 Likes
@mitsuhiko Forgive me for asking the question so directly: how does Rye compare to Hatch now? In the past year, it has also added features like Python version management (using the same “python-build-standalone” project IIRC), and easy-to-use installers. As far as I understand, the main feature of Rye that Hatch doesn’t currently have is lock files (this is being implemented, as far as I understand).
Rye (and Hatch, and all other workflow tools) actually doesn’t really enable the workflow I’d like, and I’ve been meaning to write a blog post about that for about a month, but I didn’t find the time. I hope I will soon.
I must admit that I’m not really looking at what hatch is doing. Hatch might very well try to turn into the one tool that does everything, but so far I have mostly encountered it via hatchling. I have however observed other users saying that hatch doesn’t do what Rye does so I’m assuming their overlap does not fully exist.
I have attempted to just wrap the tools I am familiar with in a convenient manner and not actually massively innovate here. It uses pip-tools which still turns out to be what I rely on the most on a daily basis. In addition it now bundles ruff and it bootstraps Python via the indygreg builds.
2 Likes
I’ve been reading through Rye’s docs over the last few days. I think this is a nice alternative philosophy, to have Rye orchestrate things but mostly delegate specific tasks. It’s the convenience of one tool to manage projects, while under the hood it’s just the regular old venv + pip-tools - meaning if something goes awry in my project it’s easy to fix! I noticed in the docs that by default a project’s virtual environment just gets created in .venv in the project folder and that stood out to me as a way that Rye is just doing the stuff I would normally do manually, but streamlined, and with a single consistent cli interface.
Having Python version/installation management in there too is really nice.
Anyway, thank you for sharing your hard work with the rest of us!
No built-in commands like lock yet but this is actually completely supported by this plugin: GitHub - juftin/hatch-pip-compile: hatch plugin to use pip-compile to manage project dependencies and lockfiles
It uses pip-tools under the hood and uses different files based on one’s platform which makes it nice for contributors. I was going to implement some interim solution until lock files are standardized but this plugin is so good that there was no need for me to do so. Perhaps I should advertise it better?
Yeah Hatch has done this since December, speaking to Jean’s point. I don’t mind many tools in this space currently as I’m just heads down working on Hatch in my free time and partially at work. As Jean said I am trying to make it support all use cases and thus become the One True Tool 
and nothing is really going to stop my work on that!
I will admit I am a bit surprised that you are unaware of the feature set of Hatch though. Is there a reason you’re not looking at what other tools are doing/assessing the landscape? If you have observed others saying somewhat similar things then perhaps I need to improve documentation?
If it’s the former case then that makes sense since you do make it pretty explicit that it’s a personal project. Additionally, it makes even more sense to me in particular because I love Rust and so I know you must be having fun with an excuse to program in that 
edit: Oh I just downloaded Rye so that I might try for myself in case the docs are inaccurate and I can confirm that there are two things that Hatch does not and will never do:
pythonshims - Hatch does not act like pyenv and only will ever add Python installations to your PATH if requested. Hatch will never have the concept of shims and such feature requests will be instantly closed. I should put that in the contributing docs…- special behavior with symbolic links - I’m a Windows user and have been forever because historically the best assistive technology has only been supported on Windows (I used to be sad about that but no longer because it’s literally the best developer OS now!). I agree with what Steve said, you simply cannot ever assume support for that and, in my opinion, I will not implement features of marginal impact for a subset of users to the exclusion of others.
3 Likes
I saw your blog post about it as well and I agree with some of what you said there, especially this:
However I would love to see the community converge to a Rye like solution, no matter where it comes from.
I agree with that because I like the idea of a “one-stop shop”; that is what Python sorely needs.
At the same time, I do wonder if some of what Rye offers is really going even beyond that. When I think of the one-stop shop I’m mostly thinking of the stuff that is essential to get off the ground. That includes installing packages, and I think it has to include environment management, because the Python import system means that environments are essentially required in order to be able to work with arbitrary projects that have their own package requirements. These are required in order to work on projects.
Beyond that are the aspects that have to do with building and distributing projects. Wheel-building and publishing fall into that category. I’d say lockfiles do too, as those are another way of allowing someone else to replicate what’s needed to run a certain bit of code. I think these are also good to have.
Things like formatting and linting, though, seem in a different category to me. They’re much less essential. I’d consider formatters and linters as more akin to test frameworks, debuggers, and so on. That’s not to say that it’s bad to include them, but I don’t think they’re as critical for getting off the ground, and I don’t think it’s as critical to have them unified. That is, having multiple competing tools to install packages and manage environments is a problem, because differences in how those tools work will impact people’s ability to even begin working with Python. In contrast, if there are multiple competing linters or debuggers out there, that’s not so big a deal; it’s possible to make functional code without using those at all.
The other kind of functionality Rye offers, of course, is “project management” — keeping your pyproject.toml in sync with what’s installed in the environment, and so on. Rye (like, for instance, Poetry) is “project-bound” — you can’t do anything without having a project, and everything you do is tied to whatever project is “active” at that moment. It’s clear from the popularity of Poetry that a lot of people want this kind of project-management functionality. What’s less clear to me is that it has to be required. Obviously this just comes from my own perspective as someone who does a lot of data tinkering, which usually means having a range of exploratory “proto-projects” that share an environment.
It seems like it would be possible for a tool to manage a project if and when there is one (and when there is one, not several), while still allowing people to create and manage environments that are not project-bound. This might mean having separate commands for mytool install somelib (which just adds it to the current environment) vs. mytool add somelib (which also adds it to the pyproject.toml). Perhaps some people would find that confusing. It also means that the manager configuration has to be separated from pyproject.toml. To me though it’s pretty important to be able to work in this way.
My benchmark when looking at things like rye is, of course, conda. If we take away the bundled linter/formatter (because like I said it seems more optional to me), I think conda already does most of what rye does: it manages the Python interpreter, it manages the environment, and it can build packages. The main gap seems to be lockfiles, which are apparently available with a separate conda-lock package, although I haven’t looked into that much. And then I guess technically uploading packages uses the anaconda command rather than conda, although probably this could be added as a conda subcommand with the new conda plugins thing.
(Incidentally, one thing that is not clear to me from Rye’s documentation is whether installing/removing packages solves for the entire environment state (like conda) or just for the current operation (like pip), although because it uses pip/pip-tools I’m guessing it’s the latter.)
But the other difference is what you talk about a good deal in the blog post, namely that Rye “doesn’t do all that much” and mostly bundles together other existing tools, whereas conda really does have a lot integrated into it. Also, it’s fair to say conda provides substantially more power in the areas where it overlaps with Rye, because it can manage tools that have nothing to with Python (e.g., you can install R or nodejs into a conda environment), and because it has various other supporting features (for instance, environment revisions that allow you to roll back the environment to a previous state).
Perhaps this is just me being once burned, twice shy, but I tend to be apprehensive that a seemingly “one-stop” solution that aggregates multiple independent tools is more likely to wind up being a leaky abstraction somewhere down the line. That’s why I lean more towards the hope that we can get a workflow tool that is unified by design rather than just a layer of unity over disparate pieces. There is no doubt, though, that an advantage of the latter approach (i.e., Rye’s approach) is that it eases migration for many users, since many of them will already be using some of the tools that Rye wraps.
In the end, when I think about a “Rye-like solution” I see it more as a “conda-like solution”. 
It would be easy enough to make some kind of meta-package that pulls in a linter, formatter, and whatever other tools are needed into a conda env, so I don’t see that as a huge stumbling block for getting started with Python. And when it comes to the more heavy-duty tasks of managing the environment, it just seems a lot more likely to me that conda can be extended to do what Rye does than vice versa, and in a more unified way. As you say, though, it’s more important that the solution be a solution than that it come from a particular source.
I do appreciate Rye for what it shows about how we can present a more unified UX to end users. I’m curious if you have used conda and, if so, what you see Rye as providing that conda does not.
AFAIK, hatch does not have a hatch add command to write dependencies directly to pyproject.toml, nor it supports installation of global tools? But I might be wrong on that one.
No commands to modify dependencies yet, that will be coming in the next minor release. As far as global tools I do plan to add wrapper commands around pipx, likely in the second minor release from now.
1 Like
Sick!
Interesting. Do you mean spawning in suprocess?
No I mean similar to Rye you would be able to install, for example, Magic Wormhole such that the executable wormhole would now be on your PATH.
I don’t quite see the benefit of re-implementing pipx but it’s possible if I look through the implementation and it’s simple enough that I may. Very unlikely however.
2 Likes
I want to say that I very much agree with this idea: If there is already a specialized tool that does the job very well, why should I reimplement it in my own tool? But this also precisely eliminated my ambition to turn PDM into that One True Tool . I actually don’t understand why people want that so much. If we centralize the efforts of PyPA maintainers to maintain such a huge package manager, it will mean abandoning smaller specialized tools due to limited resources. This is authoritarian and not in the spirit of open source.
8 Likes
It’s very simple. It is overwhelming and takes a lot of time to figure out what each of the tools does, how to install it, where its documentation lives and how it interacts with other tools. It requires learning concepts like “wheel”, “build backend” and “editable install” which are removed from the user concerns of “upload my package” or “run my tests”. This has little to do with how easy doing things actually is when you’re expert in the tools, and everything to do with communication and how easy it is to get going. Most people don’t want to use tools tailored to their personal preferences, they don’t want to have preferences, they just want to get packaging chore done as quickly as possible. How would you feel if you had to tell whether your broken car has a motor problem, axletree problem, engine oil problem or firmware problem in order to go to a motor, axletree, engine oil or firmware garage? Or if you had to deal with 5 different administrations in order to pay your tax? Or if you came here to visit Paris and you found that urban train, metro, tramway and bus are 4 different networks and there are separate maps, tickets, fares, reductions, desks and apps for each of them? This is exactly how I felt about Python packaging circa 2 years ago.
9 Likes
You can pick your analogies, though. How would you feel if you were a carpenter and you were given a huge single tool that was all of a hammer, screwdriver, saw, drill, plane, chisel, etc? Wouldn’t you prefer a set of individual tools? Or a chef given a microwave and told you didn’t need an oven, grill, hob and fryer, because the microwave could heat anything?
The problem here is that Python’s user base has changed. It used to be that Python was built and used by “craftworkers” - people like the carpenter and the chef above, who use and prefer a variety of specialised tools. But there’s been a huge influx of “non-specialist” users who don’t think like that, and for whom Python (and in particular packaging) is a commodity service like the car mechanic, transport system, etc. from your examples.
The problem is that the people who build packaging tools are craftworkers, in the above sense (they use the sort of tools they build). In non-software contexts, people using commodity services in general pay for them, and that’s how they compensate the people who build the commodity service. But in open source, there’s no payment, and so the same incentives to build commodities don’t exist. And the people who build packaging tools are generally incentivised to build the sort of tools they want (on the “scratch your own itch” principle) rather than the sort of tools non-specialist consumers want.
So there’s a dilemma. Luckily, there are some open source volunteer developers who are incentivised by “making things easier for non-specialist users”, so we do get open source commodity services and tools. But it’s only a small part of the developer community, and the increase in people who are profiting from open source, demanding more and more free support, is making it harder for many open source volunteers to maintain their motivation for developing tools or features that help others but don’t relate to their own needs. And this way leads to burnout, which makes the problem worse, causing a vicious cycle.
15 Likes
@mitsuhiko I hope feedback here is okay. I’m also okay with moving this to a discussion.
I’ve tried Rye again today for an application project (you know “the one not meant to be built into a wheel”) and I very quickly hit a wall:
$ mkdir myproject && cd myproject
$ touch main.py
$ vim pyproject.toml
[project]
name="myproject"
version="0.0.0" # Don't matter
requires-python=">=3.11,<3.12"
$ rye sync
... # errors because my project did not have src/myproject/__init__.py
lerror: could not write production lockfile for project
So, I found Rye is not very application-friendly. It assumes you want to package your application (where src/myproject/__init__.py must exist and implies that [build-system] is specified (if not, uses setuptools by default).
I’ve been testing various project management tool to manage one environment for my folder of Python files (a folder of DAGs for Apache Airflow; a well known blindspot that Python tooling does not manage well). I unfortunately found both Hatch and Rye to not be so friendly for this workflow
1 Like