Python Packaging Strategy Discussion - Part 1

[brettcannon]

@pf_moore

So if I can’t make such a decision and make it stick, and we never achieve consensus, what option do we have other than asking the SC? And of course, they will (entirely reasonably) say that they have far less expertise in this matter than we do, so how can they choose?

Consultation and a lot of talking. Otherwise the SC could help set up a packaging advisory committee or something if you/Paul felt more comfortable with that.

@abessman

If VSCode were to spearhead the rollout of a new, unified packaging toolchain, a significant number of people won’t touch it based only on the (perceived) source; such a move would be interpreted as step two of Embrace, Extend, Extinguish. Whether that interpretation is fair or correct (I’m sure it’s not) is besides the point.

I don’t want to side track on this topic, but:

1. If by “spearhead” you mean “invent”, that’s not going to happen because I personally don’t want that, but I am willing to help push anything this group rallies around.
2. If people don’t already trust me to do the right thing for this community, then I don’t know what more I can do to convince them to trust that I always have the community’s best interest at heart.
3. Haters gonna hate.

@johnthagen

As to what gets bundled, it seems to me like a combination of what people need from Poetry/hatch/PDM + native code building extension/plugin support.

The building is separate from the rest since that comes down to what you put into [build-system] in your pyproject.toml, so there’s nothing to distribute (especially since you have to download your build dependencies anyway).

@pf_moore

@dstufft

However, I now think that that flag had the right idea, we just need to implement it in a better, more sensible way.

For a long time now, I’ve wanted to do something like this. I wasn’t aware that it had previously existed. I certainly think this is a useful direction for pip to take, although I’m not sure how close that gets us to what people mean when they say “environment management” - as usual, the difficulty is likely to be in agreeing what we actually want, in sufficient detail to deliver it.

For me, I think when people say “environment management” it covers:

1. Creating
2. Selecting (as in how to specify which environment to use when there are multiple options)
3. Deleting (if its location is non-obvious)

environments. After that, because pip has historically been installed into environments (and conda/mamba handle environment management already), I don’t think flags on pip to point at specific environments has been what people have thought about.

And I will say that after diving into the topic of virtual environments, I will say there is no 90% answer, so there will be some teeth gnashing regardless of the decision.

@sinoroc

run Python scripts and commands à la py launcher

• this could be the place to include support for __pypackages__ without changes to the Python interpreter itself

That’s already the plan if PEP 582 happens: Support PEP 582: __pypackages__ · brettcannon/python-launcher · Discussion #70 · GitHub

@cgdae

Ok, so packaging does some of what i want. But not all, and the platform tags it generates differ from what cibuildwheel creates.

That’s going to come down to your build tool then since cibuildwheel isn’t directly creating anything, but instead driving the build tools for your project. But at this point, packaging is as close as you get to a standard library for packaging specs.

@cgdae

It seems to me that this situation is crying out for standardisation in the standard library.

I understand why you think this, but this isn’t going to happen to the extent you’re thinking. We are removing distutils from the stdlib for a reason already, so going too far into pushing things into the stdlib would be taking a step back.

Now, having said that, somehow making it so interpreters provide details about their wheel tags directly has been discussed, but that’s off-topic.

@cgdae

Can i ask where this is being worked on? Is it in packaging or somewhere else?

packaging

@cgdae

Thanks for explaining. This shouldn’t preclude providing the basic low-level packaging functionality in the standard library though, and i suspect this would be of enormous benefit.

This is getting off-topic for end user UX, so I’m going to say that I personally do not support moving large chunks of packaging library code into the stdlib for a myriad of reasons and ask that other questions on this topic be done in a new thread.