Mostly I just think it’s hard for a single tool to provide a great UX for installing standalone tools for someone who doesn’t care about Python, and also a great UX for working on Python code, at the same time. You’re right though that a lot of the underlying machinery could be shared. It’s not my focus currently, but if someone wants to figure that out then it’s cool with me :-).
Sure, I get that.
For the Pybi format: I’m not worried about this at all, because there are already literally dozens of ways that people get Python. Python.org download, every Linux package manager, Windows Store, built-in to macOS, homebrew, conda, docker registries, nuget, pyenv and its competitors, Heroku buildpack, preinstalled on GitHub Actions’ VMs, self-built, … and it’s fine. I don’t think adding another option is going to cause any problems, and actually a bunch of those could be simplified by having pybis on pypi.
For workflow management tools: yeah, this is a more direct competitor to those, and the confusion in the community is real. But like, the approach/architecture is different enough that idk how you’d patch it into any of those tools, and this is the tool I want to use so… not sure how to avoid it, really.
I am less worried about the proliferation of workflow tools than some. It’s definitely painful right now, but this is just kind of the natural lifecycle of these things, as people explore the different design tradeoffs and figure out what works? Back in the 2000s, the Python community spent years agonizing and soulsearching over why we had so many web frameworks, and how we needed to get people to stop writing them or else we’d never be able to compete with Ruby on Rails. But telling people to stop writing the frameworks they wanted to use never really accomplished much, and in the end worries just kinda of faded out as mindshare coalesced around a handful of options like Django. A similar thing happened with version control systems around the same time.
I don’t want to derail the thread into conda/pypi discussions, but there’s a common misconception here I want to address.
The packaging ecosystem of PyPI/wheels/sdists/setuptools/etc. is really more of a “meta-packaging” system. Python gets used in tons of different contexts, and packages people upload are PyPI are generally useful in multiple contexts – so e.g. when I upload a Trio release to PyPI, I assume it might later end up in a Debian package, internal tooling inside macOS, a RenPy game, a conda package, a Blender plugin, Google’s monorepo, a Nix snapshot, etc etc. All of these environments make specific choices about how their Python environments work, how to manage dependencies, so on – and the PyPI/wheel/etc. ecosystem is designed to be flexible enough that all these different downstream ecosystems can consume our packages and adapt them to their situation. If I release a conda package, then only conda can use my code; if I release a PyPI package, everyone can use it (including conda).
So I’m going to upload Trio to PyPI. But that means Trio needs to use PyPI-style metadata, and in particular PyPI-style dependency declarations for the packages I want to use (since this is the dependency metadata that all those downstream ecosystems know how to consume and map to their own framework). And that means I need a way to take my PyPI dependencies, and run my tests against them, which means I need a way to build Python environments out of PyPI packages. (And also I really want direct access to the packages that other people are uploading to PyPI, so I can do stuff like push a feature to one package, and then immediately use that feature in another package. The downstream ecosystems will catch up on their own time, but first I need to do the work they want to catch up with!)
So for lots of end-users, conda and PyPI are more-or-less interchangeable – they can pick whichever they prefer based on whatever tradeoffs are important to them. But package maintainers need to work with PyPI, and since they’re the foundation of our whole ecosystem, they deserve good tools!
Currently Posy only supports Windows/macOS/Linux, so not an issue :-). Or more precisely: it only supports platforms that have standardized wheel tags. So if FreeBSD or whoever wants support, there’s a clear way. But from my experience with wheel tag standardization, I doubt it’s going to get ahead of LLVM/Rust.
I mean… you absolutely could make it work in Python. Conda manages it. But
-
When I started, pip was still totally wedded to running inside the environment it was managing
-
A tool whose most important functionality is:
- Running before every Python execution, so startup speed is absolutely critical
- Chewing through exponential-time resolution algorithms
- Moving bulk data around
… is just never going to be Python’s sweet spot. Horses for courses and all that.
-
Then I wouldn’t have an excuse to muck about with Rust, so it wouldn’t exist
Sure, but that’s just out of scope? Use posy to invoke your favorite deployment tool. Or if your deployment procedure is “take your dev environment and stick it in a tarball/docker container”, then posy could do that pretty trivially – all you need to do is unpack the same pybi/wheels into an alternative (simpler!) filesystem layout.
The pybi format is closely modelled on wheels; in posy right now they actually share a lot of code. (There’s even a BinaryArtifact trait to write code that’s generic over both.)
The difference is just, pybi’s have metadata that tells you have to install wheels into them; and wheels have metadata that tells you how to install them into a Python environment. (Remember that one of the mandatory inputs to the wheel unpacking operation is the paths to the Python environment’s purelib/platlib/headers/scripts/data directories… that doesn’t make sense if the thing you’re unpacking is a Python environment!)
Oh and pybi filenames just have platform tags like win32, not full wheel tags like cp310-abi3-win32, for obvious reasons.
I guess we could declare pybis to be a special variant of wheels instead if we wanted, but I think it would be more confusing than helpful?
Honestly, I have no idea, I’ve spent zero thought on what happens if you enter a posy environment and then try to make a venv from it :-). I’m not sure what you’d even want to happen? It’s a weird thing to do.
You can certainly use a pybi as the basis for venvs, though – they’re fully-capable Python installs.
I’m not going to rewrite it in C, no And FWIW the download/verify/extract part is pretty small compared to all the other stuff posy does currently, like parsing package metadata and resolving dependencies.
Is your concern about rust being a bad choice technically, or about having bad optics/marketing?