Python Packaging Strategy Discussion - Part 1

[ofek (Ofek Lev)]

It wouldn’t replicate the logic you have there but rather expose an API build backends can call to execute the logic you have there. Essentially, a way for things to happen before wheels and source distributions are built.

[brettcannon]

@barry

, what I really want is more standardization around pyproject.toml settings and lock file formats.

You already know about where lock files stand (and if I’m wrong and you don’t, message me). As for more standardization, it might be interesting to look at what Poetry, Hatch, pipenv, PDM, and even Flit offer out-of-the-box and see where their feature sets overlap. That’s probably the most telling as to what we might be able to to get the community to rally around standardizing.

@barry

Let’s take test dependencies for example.

This was proposed at Adding a non-metadata installer-only `dev-dependencies` table to pyproject.toml .

@rgommers

I’d imagine this would use test, doc and/or dev groups under [project.optional-dependencies]. The PEP 621 text is a little fuzzy, but the way it describes synonyms like tool.poetry.extras for it, I interpret that as it being for this kind of purpose - standardizing tool-specific dependency groups.

That stems from Core metadata specifications - Python Packaging User Guide which explicitly reserves test and doc for this sort of thing (and which I believe @barry has said he wished were plural ). That makes it the closest we have to a standard around specifying development dependencies

[pf_moore (Paul Moore)]

OK, then it sounds like it might be useful. I agree that the consensus seems to be that something general is only likely to help in simpler cases, so maybe people are simply pointing out that you’re over-selling how broadly applicable it might be. I never expect to write a complex backend, so limited scope would be fine for anything I’d ever need, but I’m not typical. And honestly, the number of people interested in writing a build backend that needs to compile native code, and who don’t have specialist needs, is likely extremely small (most people with simple needs will just use setuptools). So I’m not surprised there’s limited interest. But that’s no reason to not build it - niche libraries are still of use.

[h-vetinari]

@pf_moore

I agree 100%. IMO, PEP 517 was a huge success, and we should build on that, not undo it. And I’d go further - we should do more of the same. Encourage innovation and flexibility, and write standards that allow people to tailor packaging to their needs, not force people into a model that doesn’t work for them.

It’s very tempting to double down like that, because it broadly validates the previous direction and choices made in this space, rather than face the increasingly loud feedback that the outcomes^[1] of those choices have produced a state of affairs that users are very unhappy with.

@pf_moore

Yes, I know, we’re hearing strong demands that packaging is too confusing, and needs to be simplified. Fine, let’s make it less confusing. I’m all for that. But let’s not do that by making it less capable. We should be offering clear, easily accessible defaults that meet the needs of the majority of users.

No-one is talking about making anything less capable. But I argue that overall we do want less tools doing the same thing (like build backends), rather than more. Tools want to be used, and in the case of build backends, those users are library authors^[2], who pass on their choices to all downstream users^[3], which just perpetuates the situation that is being criticised.

I’d say that people with roles like @steve.dower and @barry (absolutely requiring their own backends for internal BigCorp distributions) are an absolutely miniscule portion of users^[4]. Those use cases are also those which will make things work regardless, because of BigCorp paying its employees to solve the (internal) problems at hand, but it’s not a great stand-in for the needs of the average user.

It’s been cited a lot of times in this thread already, but “write standards that allow people to tailor packaging to their needs, not force people into a model that doesn’t work for them” is just going to lead to more of the “XKCD: standards” type outcomes, rather than do any halfway meaningful consolidation^[5].

It means that, despite people’s good intentions, the overwhelmingly dominant inertia^[6] will all-but-ensure that everyone continues in their respective groove / niche / bubble, continues optimising for their own use cases and user base, and leaves the people farthest removed from these discussions with a wild zoo of tools to deal with (at least I strongly doubt that any new default without substantial weight behind it would make dent in this).

If that is the outcome of a 250+ post “strategy discussion” with the explicit aim to reduce this proliferation of tools, I’d be pretty disappointed.

---

1. I’m distinguishing this because while choices are intentional, their outcomes are often not. Also, if we postulate that it’s still the right path to a “promised land” solution, then the speed at which we’re (not) getting there is one of those outcomes. ↩︎

2. the most opinionated inhabitants of the ecosystem, which will have no qualms to go with something that’s not the newly-defined default if it doesn’t solve their problem ↩︎

3. and all roles in between, from authors of reverse dependencies, to distributors, to administrators ↩︎

4. which is not to downplay the outsized role they play for their user base, much less in the ecosystem ↩︎

5. If we accept that we cannot solve 100% of all use-cases with a single tool, then some degree of forcing people to make changes is unavoidable, even just for enforcing a new default. I’d say the case would be pretty clear that this would be a good trade-off if we could cover 99% and have the remaining 1% jump through hoops. It’s less clear at 90%:10%, and even less so at 80%:20%. ↩︎

6. of the “it would be nice to interoperate, but I need to get this to work now” kind ↩︎

[barry (Barry Warsaw)]

@pf_moore

it would be better (IMO) to have libraries that make building your own backend as simple as possible. Then you have PEP 517 as the interface, library code to do all the grunt work of assembling wheels, etc, and you just write your own business logic - just like you’d do with a backend-specific plugin, but without the lock-in

Yes, that’s the “other way 'round” approach I was implying ^[1] and it works for me. You need to get the right division of labor and functionality in those libraries to make it possible to write your business logic efficiently, but the same is effectively true for a single backend with hooks and APIs.

---

1. or at least had in my mind when I wrote my reply ↩︎

[barry (Barry Warsaw)]

@rgommers

I’d imagine this would use test, doc and/or dev groups under [project.optional-dependencies]

Yes, although of course I wouldn’t want to hard code test, doc and dev. Despite my initial cognitive mismatch connecting Hatch’s features and [project.optional-dependencies] settings, the way Hatch does it is exactly what I was thinking about here.

[barry (Barry Warsaw)]

@brettcannon

You already know about where lock files stand

Yes, and I’m confident we’ll ^[1] get there!

---

1. well really, you’ll ↩︎

[barry (Barry Warsaw)]

@brettcannon

This was proposed at Adding a non-metadata installer-only dev-dependencies table to pyproject.toml

Yes, exactly ^[1]. Hatch’s [project.optional-dependencies] + [tool.hatch.envs.<blah>.features] is functionally what I was thinking about, but I agree that pulling these out of optional-dependencies into a dev-dependencies section is is a long term better way to go.

---

1. he says without reading the whole 58 message long thread ↩︎

[barry (Barry Warsaw)]

@h-vetinari

I’d say that people with roles like @steve.dower and @barry (absolutely requiring their own backends for internal BigCorp distributions) are an absolutely miniscule portion of users

Perhaps, and of course I do also wear my “random guy who maintains some open source Python stuffs” hat, for which the current situation (regardless of the specific tool choice) is miles and miles better than the setuptools way. But then, wearing that hat I don’t do a lot of complicated stuff anyway so I’m just trying to make things a little more resilient and standardized.

That said, the needs of BigCorp users are underrepresented here, and probably in public forums because they just deal with what they’ve got, and throw resources ($/time/people) at the problem, grumbling to themselves in Slack/Teams/Zoom/Breakrooms. It’s not made easier by the really diverse choices made by upstreams either, all of which require their own one-off hacks that have to be redone every time something upstream changes. I do think that some of the issues encountered internally by enterprises contribute to the sentiment that “Python packaging is hard” because there can be a lot of Python consumers inside those organizations, and their only encounter with Python is within a corporate environment.

[h-vetinari]

@barry

for which the current situation […] is miles and miles better than the setuptools way.

Just in case it needs to be said: I’m not proposing to go back to that^[1], but forward (in a direction that’s more cohesive than the current state of affairs; however we’d end up shaping that).

@barry

That said, the needs of BigCorp users are underrepresented here, […]

I cut the quote short but I agree with that whole paragraph – I think corporate users (not just Big ones) are both underrepresented in these discussions, and also highly exposed to the many different ways that Python packaging works just within all the internal uses (it’s something I see all the time in my $dayjob). Though in contrast to BigCorp, not everyone has the means to set up their own distribution much less build backend, so the grumbling is very loud also in SMEs.

---

1. at least, people will have a lot of associations with that term ↩︎

[smm (Shamika)]

I want to thank everyone who has participated in this thread. It is quite encouraging to see an engaging discussion such as this one.

I am in the process of writing a blog post summarizing this thread and setting out a way forward. While there has not been a clear consensus on what unification will look like, I do feel there is enough consensus that this should be taken further to flesh out the details.

As a heads-up, the remaining strategy discussions will be about-

• Better support for Packaging users
• Phasing out legacy systems
• Supporting each other in the community
• Encouraging long-term contribution

The second part of the strategy discussion is now live. I would like to invite everyone who has engaged in this thread to continue the discussion with the second part of the strategy discussions.

[BrenBarn (Brendan Barnwell)]

@h-vetinari

No-one is talking about making anything less capable. But I argue that overall we do want less tools doing the same thing (like build backends), rather than more. Tools want to be used, and in the case of build backends, those users are library authors[2], who pass on their choices to all downstream users[3]

I’d say that people with roles like @steve.dower and @barry (absolutely requiring their own backends for internal BigCorp distributions) are an absolutely miniscule portion of users[4]

It’s been cited a lot of times in this thread already, but “write standards that allow people to tailor packaging to their needs, not force people into a model that doesn’t work for them” is just going to lead to more of the “XKCD: standards” type outcomes, rather than do any halfway meaningful consolidation[5].

It means that, despite people’s good intentions, the overwhelmingly dominant inertia[6] will all-but-ensure that everyone continues in their respective groove / niche / bubble, continues optimising for their own use cases and user base, and leaves the people farthest removed from these discussions with a wild zoo of tools to deal with (at least I strongly doubt that any new default without substantial weight behind it would make dent in this).

If that is the outcome of a 250+ post “strategy discussion” with the explicit aim to reduce this proliferation of tools, I’d be pretty disappointed.

Amen to that. I have to say though that after following this thread I have some amount of pessimism. Unless people from all sides of the problem are willing to consider everything on the table — changes to pip, changes to Python, changes to PyPI, everything — I’m doubtful that anything will come out of it that really resolves the issue.