[announce] Pybi and Posy

[dstufft (Donald Stufft)]

@njs

When I started, pip was still totally wedded to running inside the environment it was managing

This doesn’t feel particularly relevant, given you’re not using pip from posy anyways?

@njs

A tool whose most important functionality is:

• Running before every Python execution, so startup speed is absolutely critical
• Chewing through exponential-time resolution algorithms
• Moving bulk data around

… is just never going to be Python’s sweet spot.

Eh, I’m not sure that I agree, and it’s a bit weird to me to be worried about the performance of startup speed, then create env forest construction that has to happen on every invocation, which AIUI is going to explode the length of sys.path, which is going to create the same perception of slow commands on it’s own.

@njs

Then I wouldn’t have an excuse to muck about with Rust, so it wouldn’t exist

Fair enough

@njs

Sure, but that’s just out of scope? Use posy to invoke your favorite deployment tool. Or if your deployment procedure is “take your dev environment and stick it in a tarball/docker container”, then posy could do that pretty trivially – all you need to do is unpack the same pybi/wheels into an alternative (simpler!) filesystem layout.

Should it be out of scope? If we’re trying to step back from the Elephant, shouldn’t we be looking at the total scope? It feels like the up front idea, or at least the genesis of the idea was how the various packaging tools seeing “one part of the problem” instead of the whole problem. Declaring that out of scope feels like it’s just solving “two parts of the problem” instead.

@njs

I guess we could declare pybis to be a special variant of wheels instead if we wanted, but I think it would be more confusing than helpful?

What I was thinking of to be honest is that there’s no reason that Wheels have to be quite so specialized to installing only Python packages, and could be evolved such that the same packaging format is useful both for installing Python packages, and not Python packages. After all, there’s nothing inherently different about a pybi and a wheel, they’re both just zip files with some files in them that get unpacked onto the filesystem.

@njs

Honestly, I have no idea, I’ve spent zero thought on what happens if you enter a posy environment and then try to make a venv from it :-). I’m not sure what you’d even want to happen? It’s a weird thing to do.

It doesn’t feel like a weird thing to do? A number of tools in Python assume the ability to create virtual environments as part of their workflow. Heck you just told me that in posy, I should use posy to run my deployment tool, what if that uses a virtual environment to create isolated build environments for instance?

@njs

Is your concern about rust being a bad choice technically, or about having bad optics/marketing?

It’s not exactly technical, but I’m not sure that optics/marketing is the right word.

I mentioned it above, but one of the initial ideas you mentioned in your post was that all of the existing packaging tools solve one piece of the problem, which means that there’s no coherent answer to an overall workflow. What I see with posy is that it is repeating that same mistake, except maybe it’s solving two pieces of the problem, instead of one.

So while Rust itself is a perfectly fine language to write a tool in, one of the things it’s achieving is papering over the fact that posy still is making the same fundamental mistake that the other tools do, it’s just maybe done a nicer job at it. That papering over makes it easier to ignore the fact that it’s still missing important parts of the packaging story, which I think is a serious drawback to the approach.

But of course, there’s nothing actively preventing you from writing something that does actually the whole problem, and not just pieces of the problem, in Rust or any other non Python language.

There is also of course an optics side of things, by having tools that work on Python packaging written in something that isn’t Python, it’s an implicit statement that Python isn’t an acceptable language for CLI programs targeting end users.