Hi Paul,
If we consider the broadest definition of packaging, both the PyPA and conda ecosystems have millions of users. If the Python Packaging Council doesn’t have empathy for and represent all of those users, I think it would be a huge miss for Python.
For technical standards, if the Python Packaging Council is going to represent all packaging users, then broad standards that it creates need to be able to cover both PyPA and conda ecosystems. Instead of trying to shove the full Python ecosystem together to start with, we should divide the standards in two major categories:
- Boundary / Cross-Ecosystem Standards - the things that have to be standardized to make cross-ecosystem interoperability work. For example, things like name-mapping, lockfile and environment formats, hardware / variant detection.
- Internal Standards - things that stay entirely inside one ecosystem’s purview. For example, Python Enhancement Proposals (PEPs) that cover wheel format details or build backend internals, Conda Enhancement Proposals (CEPs) that cover repodata formats, and Conda-Forge Enhancement Proposals (CFEPs) for how packages are pinned.
To answer your question, then conda tools do not need to follow PyPA specs and the PyPA tools do not need to follow conda specs, except where a tool wants to claim cross-ecosystem interop, in which case it needs to meet a Boundary / Cross-Ecosystem standard.
As things get started, there will naturally be some asymmetry here. The Python Packaging Council will have authority over the Boundary and PyPA Internal standards, but it won’t have authority over CEPs and CFEPs. Over time, we can have discussions if the conda steering council and the conda-forge steering council want to also delegate some of their authority to the Python Packaging Council, however, I don’t think that is required to start with or necessary for success.
On the social side, there has already been in person events having broad packaging community attendance, for example at the PyCon US and EuroPython packaging summits. For online discussions, similar to the technical standards above, I don’t think we need to force the whole Python ecosystems together in one place. It is fine if Boundary and PyPA internal standards are discussed here on DPO, and internal CEP/CFEPs are discussed on Zulip. The Python Packaging Council should also hold office hours and seek input in all of those places and the PSF Discord to try to get the broadest input and reduce contentious discussion on DPO.
On the topic of wheel variants and improved binary support, I would prioritize bridging the conda ecosystem. The context is once again we have had the two separate PyPA and conda ecosystems, and then a couple of years ago, Anaconda damaged trust in the conda ecosystems through changes to their Terms of Service. I believe that this caused some of the companies using Python to feel like their corporate interests were at risk due to potential vendor lock-in. Some of these companies then helped fund the WheelNext effort through dedicating resources to it, hiring consulting companies to work on it, and even potentially paying tool vendors to develop solutions. For transparency, I work for Anaconda and both Anaconda and the conda ecosystem are WheelNext members, but I can’t take credit for the amazing work that has gone into the draft standards. Also where any of this touches Anaconda specifically rather than the conda ecosystem broadly, I’d of course recuse myself from any decisions. The acute risk of vendor lock-in has eased since Anaconda updated their Terms of Service to make it more clear and conda-forge has continued to act as a strong community alternative. However, there is still some perception of needing more independence.
The Python Packaging Council needs to understand these corporate interests, but also focus on finding the best solutions for the millions of users in the Python ecosystems. If the major problem with the conda ecosystem is perceived vendor lock-in from Anaconda, I rather we take on helping make the conda ecosystem even more independent. Then we can bring those binary capabilities to all Python users and continue to improve it. This will be a much faster path to providing solutions to users than spending years trying to extend the PyPA standards that are already very powerful for Pure-Python use cases. I also don’t think the effort on wheels variants is wasted, for example, we should improve the conda standards to improve hardware detection like I previously mentioned.