Over the last few years on DPO, I’ve consistently argued that the rift between the packaging paradigms (wheels vs. conda) is something that should ultimately be healed, rather than deepened. Those projects are not directly at the PyPA table, but literally everything that conda, mamba, pixi, conda-forge etc. do is about packaging (and despite being polyglot, the elephant in the room is still Python). While the download numbers are smaller than PyPI, this doesn’t tell the whole story – for example, a lot of libraries that see most of their downloads through wheels have their CI and/or development workflows set up using libraries from conda-forge.
As such, I think it would be a pity to leave out that community from consideration, already during the initial round, because that will have a large influence on the overall direction, which I would prefer to be inclusive of the users whose needs are underserved by the current packaging paradigms (and so end up in conda-land).
And let us mention another elephant standing quietly in the corner … another packaging paradigm is what we at Linux/Mac/Unix/other distributions do. That tends to be ignored even more.
This is a genuine question to both of you, not in any way a dismissal of what you say - are either of these communities involved in the packaging ecosystem governance (assuming such a thing, or its equivalent, exists) for other languages?
More generally, with experience of multi-language packaging, how do other languages involve groups like conda and the Linux distros? Because it would be good if we can learn from them.
I cannot speak for large group of disjoint people, but for my self and from my experience, conda-forge is still dominated by Python packages and their needs. Yes, we have a whole lot of R packages too, but there, less questions come up in the first place, as R has a much more stringent gating mechanism for publishing packages (aside from an overall lower incidence of extremely challenging packages), so we don’t tend to run into as many issues.
Conda grew out of the needs of the Python packaging ecosystem, it just happened that the approach it took was general enough that it can deal with other languages too (not least because python packages themselves use all these languages under the hood in one way or another). As such, it is still intimately connected and related with python packaging overall (in terms of packages, users, and use of our artefacts by python projects in their development), which is a relationship we have with no other language to that extent.
Further along those lines, many of the other languages we provide packages for have their own primary distribution mechanism (rust has cargo, ruby has gems, javascript has npm, java has maven, C++ has vcpkg/conan/…, R has CRAN, Julia has pkg.jl, etc.) where we’re not really a competitor in any serious way.
My core argument is that conda (as a package format) wouldn’t need to exist if python’s primary packaging mechanism was powerful enough to solve a long list of problems that conda does address, and that a long-term unification (or at least reduction of that rift) would be highly beneficial for the python ecosystem. The curation work that conda-forge does would still be useful then, but that’s not intrinsically related to the package format.
In short: the language where we have the biggest stake in – by a long shot – is Python.
One concrete example of a potential benefit to the broader ecosystem: shared resources for building wheels that can be distributed on PyPI as well as conda-forge.
If I understand the current situation correctly, conda-forge has compute resources (via donation) to build wheels for a whole lot of Python packages, but the maintainers of those packages need to upload their own wheel to PyPI[1]. Part of the separation is just due to the external-dependency problem.
in some cases the conda-forge package is just using the PyPI wheel, but not always ↩︎
If you mean “involved in governance” as in sitting on some boards, I have no idea (@vstinner would probably know more, I believe that for example Red Hat was (has been?) very heavily involved with Java for some time), but the answer is most likely “not enough”.
Otherwise, the same as Fedora we are heavily relying on the “upstream first” policy, so we are always heavily involved with sending patches upstream (I think there are plenty of tasks which are sufficiently boring and uninteresting that only people like SUSE/Red Hat employees are willing to do it … removing six or nose comes to mind, we sent hundreds and hundreds of patches just about these two, and I am certain Fedora people did the same).
We at SUSE are much smaller team than what I expect to exist in Red Hat (and I have no internal Red Hat information since 2018 when I left the company): just less than five people for some 3,000 packages on multiple distributions (Tumbleweed, our rolling distro, and multiple versions of our enterprise distributions, and their variants). However, I am trying to free myself from my managing/internal-policy/etc. duties and be more involved in GitHub - python/cpython: The Python programming language world.
Having said that, I still believe that Python team is the biggest language-specific team in SUSE (unless one considers GCC and glibc people as the C language team), actually only other team dealing more with some particular language are YAST people (YAST, SUSE installer, is written partially in Ruby).
The Python community and PSF staff have already made connections to Rust, Ruby, Linux and Perl packaging stakeholders via our conversations around security over the last two years. I think it would be great for our packaging council to intentionally stay in touch with its peers at other open source communities that maintain a package ecosystem.
We have published an update to PEP 772, which should address all feedback we’ve received so far. Thank you!
Highlights:
Clarify the relationship between the Steering Council and the Packaging Council, and the standing delegations proposed from the former to the latter.
Standardize on the term “voting members” rather than “community members”.
Clarify that voting members can nominate anyone to the Packaging Council, including themselves, and that nominees do not need to be voting members. Affiliations of nominees must be disclosed.
Clarify the language around “wider community membership” for the initial voting members. We’ll also add an appendix with the list of initial members once that’s been determined.
Subsequent votes for new members last for one week, shortened from two weeks.
Clarifying language that the PSF will be encouraged to deactivate (not deprecate) the Packaging Workgroup.
Clarifying language around affiliation limits for Council membership.
Explicitly state that PSF staff are not permitted to be members of the Packaging Council.
Since this is a governance document, I have a question regarding
How is the existence of a COI determined, and by whom? Someone with (arguably) a COI might not consider it a COI, or may intentionally not disclose it.
That language mirrors exactly the language in PEP 13, so in both cases it’s essentially trusting the members. I think there are enough checks and balances to ensure that members act in the best interest of Python, or abstain when they must.
A no-confidence vote is triggered when a voting member calls for one publicly on an appropriate public communication channel, and another voting member seconds the call within two weeks.
The vote lasts for two weeks. Each voting member votes for or against. If at least two thirds of voters express a lack of confidence, then the vote succeeds. Quorum for a vote of no confidence is 50%.
A no-confidence vote is triggered when a core team member calls for one publicly on an appropriate project communication channel, and another core team member seconds the proposal within one week.
The vote lasts for two weeks. Core team members vote for or against. If at least two thirds of voters express a lack of confidence, then the vote succeeds.
That’s two weeks for PEP 772 and one week for PEP 13.
In fact, until recently, PEP 13 had no deadline for seconding. We held a vote last year and decided to add a one week period.
I suggest also having a one-week seconding period for similar reasons I gave for PEP 13. The full process for a vote and new election:
First call
Up to two weeks until seconded
Two weeks vote of no confidence
Two weeks for nominations
Two weeks for election
This is 7-8 weeks, plus extra time for admin. Admin is usually 1-2 weeks between phases for SC elections, so the whole thing could be around 10-14 weeks. Any vote of no confidence introduces uncertainty and the council is unlikely to operate as normal (or at all) during this time.
Therefore, like PEP 13, I also suggest only max one week for the seconding period.
Thank you @barry and the other authors for this update and shepherding this PEP towards the finish line.
This version looks to me like it’s in good shape. I like the balance it strikes between what it specifies and what it leaves to the future Council to decide and evolve, and that it aims to follow PEP 13 where it can.
I’m personally very much unconcerned about the exact details of who gets to vote in the initial election. My expectation is that that anyway evolves over time, and that the initial decisions in this PEP will have only a very minor influence on the end result compared to, for example, the quality of the candidate pool. I am confident we’ll be getting a Council with folks that have widespread trust and a diversity of perspectives and experiences across the many aspects of the packaging landscape. And equally confident that having a Council in place that meets regularly and starts facilitating, coordinating and gently steering will have beneficial effects.