Well, to be even more explicit, to my mind, if “making Python have a good packaging/environment system” is not a goal that takes precedence over logistical concerns like the division of authority between the core devs, steering council, and/or a putative packaging council, then I’m doubtful the problem will really be solved. In some of these threads I get the impression that not everyone even agrees there is a problem that is Python’s to solve, and that is worrisome.
5 Likes
Is there an interest in creating such a council?
I’d like to see this happen. With the recent interest in improving Python packaging, there’s some momentum and willingness to take action, but I find the 100+ discussion threads on here hard to follow. They often get derailed on the most divisive issue or disagreement (a problem inherent in the forum format IMO). The breadth of opinion makes it hard to find consensus or perfect solutions, however, there are plenty of good solutions and directions that we could work in.
Who would be eligible to run for the council? Who would be eligible to vote in electing members to the council? What is the ideal size of the council?
What makes sense to me,
- anyone could run for the council
- PyPA projects (rather than members) vote on councillors. Projects make more sense to me because the projects joined the PyPA, the members come along with it. But if that’s hard to administer, PyPA members would probably work fine.
- Ideal size: 3-5 people. Odd numbers are good to keep decisions flowing.
What powers would this council have? Do we want it to have powers beyond approving/declining PEPs?
PEPs are great for the interoperability stuff because they are precise and stable, but they don’t really help us create great UX. I’d love to see a council that takes responsibility for the Python packaging UX and steers the community towards good user experiences.
Perhaps that’s through
- fuzzy high-level goals (a ‘vision’) so the community can see the direction we’re heading
- a set of recommendations for users e.g. structure your project like this, always use venvs
- opinionated template projects/how-to guides, whose workflows are based on the modern packaging philosophy and form solid foundations
I realise this is hard given the diversity of workflow tools. But we should ask the question - is the goal of the PyPC to serve Python users? Or is it to support the packaging community? I feel that the top priority should be the users.
11 Likes
No one, especially since we can’t. 
Probably the best the hypothetical council can do is outline what they would like to see happen and try to influence whomever they can to follow along.
You’re welcome to your opinion, but you have to realize there are people involved in those groups who have concerns that also transcend or simply do not involve packaging, so telling them they need to set aside their concerns for this other (albeit big) concern that you personally view as tantamount is never going to work. I’m not sure what open source projects you maintain, but in Python’s case it has people and history that have to be taken into consideration, as a project of this size is not driven entirely by technology; people are involved and that means compromise and having to work within people’s needs and constraints, else people walk away and then you’re in an even worse position than when you started.
Or put another way, as the single person in this conversation who crosses every single group you outlined, I can tell you that none of them will be wrenching control of something from the other without discussions and agreeing on how things should be handled, no matter how noble the cause; none of how this entire power structure operates gets trumped by “making Python have a good packaging/environment system” at any/all costs.
5 Likes
I’m not suggesting “wrenching control”, nor “at any/all costs”, nor that anyone “set aside their concerns”, and I’m sorry if I came across as doing so. My point is simply that I see fairly frequent mentions of “how can we slice off part of the problem to get something that doesn’t cross-cut the purviews of existing groups”, and I think that doing that will likely result in “solving” an artificially reduced problem in a way that won’t actually improve the user experience much.
So all I’m saying is that, to me, the purpose of creating a Python Packaging Council would be precisely to have a group whose purview is indeed “making Python have a good packaging system”, in order to cut the Gordian knot of overlaps and gaps in scope among the existing groups. I recognize that that won’t happen without discussion and agreements. That’s totally expected. But from my point of view there’s little use in even having those discussions and agreements unless the end goal is to comprehensively address the actual problem (which as you say, is big).
As various people have mentioned on different threads here, discussions about “how to do packaging” have mostly been had among the people who maintain the tools. That is natural and their work is greatly appreciated. Along the lines that @joerick mentioned, though, what I’m mostly trying to present in these discussions is a user perspective.
3 Likes
I think that creating some sort of formal decision making body is the right thing for us to do, so much in fact that I have a partially done draft PEP from a few months ago that proposes it that I’ve held off on working on until some of my other PEPs have finished.
I think that the core purpose of this decision making body would be to own, maintain, and define the standard toolchain that we expect Python developer to use. They would have broad authority to make really any decision on what that toolchain should be, what use cases it should cover, etc [1].
To take a step back though. I think one of our current biggest problems is that while we have a perfectly reasonable solution for defining interoperability standards, we don’t have a process for deciding anything else. Thus everything we try to do has to be crammed into an interoperability shaped standard or we cannot make any real progress on it. This has been fine for a long time, but it’s obviously showing it’s weaknesses now, and we’re regularly running into questions where there isn’t a standard to define, it’s just a question of what we want the world to look like.
I think part of what this council should be, is the owners of the projects that they operate on, so rather than having the PyPI devs “own” PyPI and the pip devs “own” pip, etc the idea is that these projects now become sub projects under the overall “Python Packaging” project. This means that the council would decide which projects make sense to include under their purview and which don’t [2]. The list of projects that are kept under that umbrella may change over time as well. Maybe one of the decisions is that project X is no longer doing things in the way we want, so we’re planning to deprecate it and instead create/adopt project Y instead, and that may involve deciding to shut down project X and remove it from being an active project of the council [3].
I don’t think that we need to have some sort of controls to make sure that every possible view point or project is represented in this council. Being a member of that council is essentially being trusted to make decisions that are for the good of the Python ecosystem, which includes reaching out and talking to people in other situations or that use other tooling to decide if trying to support those use cases together makes sense or not.
I also don’t think that from a day to day perspective, this would mean changes for the developers of these projects. I think that day to day working on say pip or PyPI or whatever would look pretty much the same as it does now, but when hairy questions like “should we create a lockfile standard or should pip do it’s own thing, and if so what even is a lock file” come up the council can act as the body making those decisions for us. So on a day to day level, I think the biggest impact that the council would have is defining a vision for what Packaging should be and what is and isn’t in scope for it, something that is currently impossible to do because every single person has their own idea of what it should be.
In terms of who should be eligible to stand on the council, I think anyone nominated should be eligible. I don’t think you need to be someone who works on one of the projects to be a good person for the council.
Who should vote on the council is a harder question for me. There’s not a single constrained group of users that I think really captures it correctly.
The Python core devs isn’t right, they are focused on the Python project not the packaging projects and a number of the people working on these things aren’t core devs.
The PyPA members (either projects or people) I don’t think is quite right either. An important part of making this all work I think should be the idea that someone doesn’t need to be a PyPA (or whatever we call it) project to have a say in what packaging looks like and that we don’t need to treat the PyPA as an org that has to “catch em all” for packaging related tooling.
I would be worried about just allowing just anyone to vote… since that feels broad and exploitable… but maybe that’s fine? Maybe we can temper it using something like what the PSF does, just define some basic requirements and let people self certify as meeting those requirements, but tie it to more generic things. Or maybe we don’t and we just let anyone who cares enough to vote, to vote.
-
For instance, the above question about whether the toolchain should care, and if so how much it should care about “downstream” use cases like Conda/Linux distributions, etc is a perfectly wonderful question for such a decision making body to decide. ↩︎
-
Of course as an point of order, this council can’t forcibly take over a project, so if say the pip developers don’t want to give up ownership of the project to this council, then they’d have to effectively set out on their own. ↩︎
-
Of course that doesn’t stop people from deciding they want to continue maintenance on it and evolving it on their own. ↩︎
9 Likes
Another alternative is to do what the PSF used to do (before the membership overhaul 6-ish years ago), and what the core devs pretty much do, which is to let the voting members vote on who to add as voting members. You have pick a starting set of voters based on some involvement with packaging or the PyPA (the PSF’s membership was seeded from core developers, for example), but that set only has to be ‘good enough for now’, so you don’t have to work out exactly where the line of ‘involvement’ is or whether certain areas are getting more or less representation – as long as you can assume good faith efforts from the majority of voters. Then after the packaging council is set up, the membership can vote on adding new members and the council gets a chance to veto.
2 Likes
Yea that would work too. I don’t really personally have strong opinions on who should be the voting body except that I don’t think that just piggybacking off of one of the existing “groups” (core devs, pypa devs, etc) is the right answer (except maybe as a bootstrapping technique) and that we should ideally make it lightweight and easy for someone who cares to participate and vote.
1 Like
I concur with Thomas about not overthinking the solution to the bootstrapping problem for the council, since it feels like an impossible task and demotivating to me to having to pick a group of people that might be the “right” one to constitute a packaging council.
As a group of volunteers, it’s fitting to bootstrap a representative council in a participatory act, focused on good faith and the trust for one another we have right now and enable others to be included that have been outside the various groups (PyPA, Python steering, etc) so far.
So maybe an imperfect but productive bootstrapping could consist of these phases (derived from PEP 13):
- Phase 1: Public call to self-nominate as voter for the initial council election
- Phase 2. Candidates advertise their interest in serving on a 5-person governing body “Python Packaging Council”. Candidates must be part of the initial voters. Self-nominations are allowed.
- Phase 3: Each initial voter can vote for zero or more of the candidates. Voting is performed anonymously. Candidates are ranked by the total number of votes they receive. If a tie occurs, it may be resolved by agreement among the candidates, or else the winner will be chosen at random.
For the initial election, all phases will last two weeks.
The initial election process is managed by a returns officer nominated by the PSF Executive Director.
The council should ideally reflect the diversity of Python contributors and users, and voting members are encouraged to vote accordingly.
I also agree about not overthinking the bootstrapping problem for the council. I assume that however this will be done, the chosen folks are going to be able to be receptive to feedback, and help correct any potential issues in a second voting round after the first term is up.
What is much more important I believe is still the scope of this council. The breath of the scope may determine how the council starts operating, or even who will run for being on it. I think the alignment on that scope is not yet very good. To illustrate, let me quote a few folks:
My impression here is that Shamika’s scope is much broader than Donald’s. I’ll paraphrase the former as “all aspects of Python packaging”, and the latter as “workflow and default toolchain”. Donald’s scope seems to exclude things like interoperability with other packaging ecosystems and also build topics (because there is explicitly not a single default tool). Steve’s point is a bit more targeted, but is also leaning towards the broader side.
Personally, I’d like to see the broadest possible scope, because I suspect that that’ll be more impactful, and coordination and vision is needed also beyond workflows, and the default toolchain and its UX. It’s a harder job though if the scope is broader, and finding a council with in-depth experience/knowledge about everything from supply chain security and workflow tools to distributing Python itself, build systems, and cross-packaging-ecosystem topics may not be so easy. So if the scope is narrowed to what Donald proposes, that is a fine outcome for me too. I won’t be interested in being a candidate at that point - because I think the build system/backend and interop-with-other-package-manager topics that I spend most of my time on will then be out of scope (or of limited interest at best).
4 Likes
It’s sort of buried in a footnote, but to be clear, I don’t think that my scope excludes those things! How these things interopt with other ecosystems, how they handle broader build topics, etc I think is perfectly great things for the proposed council to handle.
What I was trying to do was a few things:
-
Back away from the loosely affiliated project structure of the PyPA, and explicitly say that the projects that are in the PyPA [1] are not just random projects that happen to come together to form a toolchain, but they are in fact owned by the PyPA (and this council is the governing body of the PyPA). This means that individual project maintainers are no longer the “final say” for each project, but rather the council is, and is part of what gives weight to their decisions.
-
To not explicitly define what topics are in or out of scope, and to instead empower the council to decide what is and isn’t out of scope. Do we care about X build topic? Is interoperability with X system something we want to care about? All of these are questions for the council to decide, not us upfront. If we give the council the power to draw the lines, then the council can adapt itself to future needs. If we had defined the council 15 years ago, I suspect the lines drawn would have looked a lot different than it would today, and in 15 years from now would look a lot different.
-
I think there’s two ways that this could end up playing out. One is the idea of an “advisory board”, where the board itself attempts to hold representatives from all the various stakeholders and they exist to mediate changes against that larger backdrop. The other is a more “project ownership board”, where the board owns the project(s) and makes decisions (which can include seeking feedback from the various stakeholders). I’m trying to push things more towards the latter rather than the former. I don’t want various stakeholders to feel like they need to get someone on the board to have their community cared about, nor do I think having a council comprised of people who don’t use the “default” PyPA tools (because they joined in an effort to get their voice heard) to be the best implementation.
By centering the primary purpose around owning/maintaining/defining the “default” toolchain, I hope to push it more towards a project ownership board, who, in service of making those projects the best they can, figure out how things like interoperability, build systems, etc play into that and how to get the relevant stakeholders on board with those changes.
The last of those is the fuzziest one, and the thing I had the hardest time wording (and I’m still not very happy with the wording I used). I do NOT think that the board needs to just be maintainers of the various projects or even direct contributors to them, and I think that diversity of background and topics cared about will make the board stronger overall.
I primarily want to avoid the idea that we need to try and represent every related community on the board (e.g. we don’t need a Conda person and a Linux distro person and a Nix person, etc) if those individuals don’t care about the “default” toolchain and are just trying to get their particular community represented. I don’t think the proposed council will scale to trying to have a representative from all the communities on it, and I think the PEP process will still ultimately be the major way changes get made, so all the various stakeholders still get to be represented.
Hopefully my hamfisted wording isn’t too bad, and the idea I’m trying to get across is coming across!
With of course, time for existing projects to leave if they want. ↩︎
3 Likes
It seems to me during the Packaging Summit people have appetite for both an actual authority that definitely teaches people what to do, and what the current PyPA serves to be but under a better name (Association?). PyPA current sucks for both because it tries to somewhat satisfies both depending on the situation, so perhaps instead of deciding which way to transform PyPA into, we’d better off create a new Council that satisfies the former, and rename PyPA to be the latter—which seems beneficial to me since it effectively keeps PyPA doing what it does right now. Or maybe in that scenario we don’t even need to rename PyPA since we’d have a good response to any misunderstandings: Contact the Council instead.
3 Likes
Thanks @dstufft, that helps quite a bit.
I totally agree that that is not needed - those topics largely overlap, and if you make things better for one other packaging ecosystem with some well-thought-out mechanism(s), it’s almost guaranteed to improve it for many/all of them. I think a council composition like that is quite unlikely to happen though - we’re much more likely to find ourselves in the opposite situation, where folks almost exclusively care about PyPI/wheels/pip/etc. That’s historically what the situation has been mostly (from my perspective at least, and that of many folks in the scientific and data science part of the Python ecosystem).
This kinda is an example of that already. It is a very important problem. Perhaps even the single most important problem. But there’s at least two similarly-sized topics in both importance/impact and in complexity of solution space I’d say:
- The set of problems that https://pypackaging-native.github.io/ covers
- Anything to do with Python packages in other packaging ecosystems, and interop between PyPI-focused tools and those other package managers.
I am still worried that your “primary purpose” framing puts those topics to the side.
4 Likes
I think there is very little value in renaming the PyPA to remove the “authority”.
Having an “association” which really just boils down to a catch all github organization is honestly not particularly useful, and I think overall it hurts rather than helps (even at at technical level it makes it harder for individual projects to add maintainers since only a few people can invite to the org, there is a shared CI budget for GHA, etc). The only real benefit to being a member of the PyPA in it’s current iteration is it imparts some level of “this project is probably okay”.
I also don’t think it will do anything to alleviate the assumption that the people in it are the authorities. I think there is this base assumption that people think the PyPA is an authority because of the name, but I don’t think that’s true at all. We could be called literally anything, and people would look to us as the authority because like it or not, we are generally the people who have the most expertise, and no amount of naming is going to change that.
I’ve seen a few people in various channels suggest that they needed to get someone on the council to get their views represented, like random Linux distros or what have you.
But part of the other reason why I think centering it around the PyPA toolchain makes sense is because realistically that’s what we actually have control over. Like I don’t think anyone assumes that this body would have any direct power over say Conda. That’s a separate project with it’s own governance, and the most we can do is provide integration opportunities, ideally with the input of those projects, that are hopefully useful to them to create a better overall experience.
I don’t think it puts those topics to the side, because those topics are an important part of making a toolchain that doesn’t suck for large portions of the user base. I don’t view those things as anything different than saying… well we need to make sure that the security of packaging is solid, or that it’s fast or whatever. They’re features (some of which are currently lacking) of a good toolchain.
If we get too generic here, we run the risk of overcomitting ourselves. For instance, if Conda and Debian wanted to interopt with each other in ways that aren’t related to regular pip installed packages. I don’t think that this council would have much to do with that, and shouldn’t.
We also run the risk of being too focused on the problems that exist today if we start enumerating specific concerns that the council should solve. If we had enumerated that list 15 years ago it would look very different than what it is today, and I expect in 15 years that list will be different again.
So giving the council a broad directive (make the tooling and ecosystem that they maintain good), they can then set the specific topics that they think are important for us now-- essentially setting the direction (at least to the extent that you can set direction in an OSS project). Hopefully the things you outlined are part of what that council considers to be important topics-- I certainly would think that they are!
3 Likes
I think that’s not all that relevant, “control” is overrated. There are other things that are key to packaging as a whole, like sysconfig or venv, that will fall outside of the scope of this council. I think @brettcannon phrased it really well already:
That sounds about right to me, and neutral enough - I’m happy to leave it at this.
IMHO should stay the name as is. If people don’t agree with the choices we make the ecosystem is better served if they’d join the PyPA and drive changes from within. This was also the opinion I voiced at the summit, so I’m not sure I agree with @uranusjr about his assessment on how things went at the summit 
PS. Yes I wish if conda would join PyPA and we can alter our documentation on what’s the best tool for differing situations. E.g. you want scientific packages, use conda. Otherwise stick to pip and co.
I think the problem is more that people fear that conda won’t have power over the council. I think there’s a hope that a council will somehow solve the fundamental difference in philosophy between conda and the PyPA tools, and people involved in the conda ecosystem see that in terms of “making PyPA more like conda”.
I hope that the council will be able to better understand the two positions, simply through being a smaller group with a common goal and the opportunity to communicate face-to-face. But I don’t think “unifying” the approaches is ever likely to happen, simply because the two ecosystems cater for different sets of user requirements, and as a result make fundamentally incompatible trade-offs. What the council can (and should!) do is ensure that all approaches are treated as equally valid for their own user bases, as well as ensuring that where we can work together, we do.
The sticking point is that the council will be very much the “official voice” of packaging for Python. Making it also be the owner of one of the packaging ecosystems does indeed create a conflict of interest. That’s a flaw in the “council = owner of the PyPA toolset” model. But conversely, a council with no ownership has no accountability in the sense of having to implement their own pronouncements, and as a result no direct checks on them. So both models have their flaws. But by making the council the owner of the PyPA toolset, with an explicit remit to promote interoperability between all packaging toolchains, we can ensure that the council works with those toolchains, not in competition with them. And because the council will be elected by representatives of all toolchains, they will be held accountable for that remit.
One aspect of this that we need to be very careful with, is packaging.python.org. As I understand it, that will be owned by the council and part of the PyPA umbrella. But it will very much be where users go to learn how packaging works in Python. Giving equal space to non-PyPA ecosystems is a very difficult problem - not least because I don’t expect the conda developers to maintain their documentation on packaging.python.org, so they will be fighting a “3rd party” perception from the start. But none of these issues will be any different, no matter what the council has direct control over.
3 Likes
To clarify (since I have a feeling my wording was read differently from what I expected), what I meant is there were people advocating for the two proposals, not that the people in the Summit came to the conclusion these two things should happen.
1 Like
I’m not sure where you’re getting such an impression. Is it worry/conjecture/gut feeling? Or can you point to an example in this thread? Perhaps I’m biased, but I mostly see people excited about a new way to tackle problems in the packaging space as a whole in a more concerted manner.
As such, it shouldn’t be surprising that people expect the Packaging Council to tackle problems in this space, and to care about the different ways in which people deploy & use python. That only requires acknowledging the existence of conda users and taking the issues they are faced with into the picture. It doesn’t require a conda representative on the council, much less conda having “power over the council” or “making PyPA more like conda”.
1 Like
I think that sounds good. Like @rgommers I have a bit of trepidation about the scope being too narrow, but based on your clarifications later I think what you’re proposing is actually pretty broad.
This sounds good and for me “defining” is critical here (i.e., not just narrowing the scope to the status quo). 
Yes that’s what I think the goal should be!
I agree that this is not needed. I do think it is good to get different viewpoints represented. But it’s not so much about representing different “constituencies” or groups using certain tools, but rather to get a range of viewpoints on what tools should do. Ideally, future improvements in the “standard toolchain” would be informed by the experiences of people using other toolchains, and so I think those should be considered. But it doesn’t mean we need people to “represent” conda or poetry or the like on the council.
To me a simple way to say it is: tools like Poetry, conda, etc., have sprung up in large part to meet needs that the standard toolchain does not meet. A Python packaging council should be prepared to examine those unmet needs and think about how and whether they can or should be met by the standard toolchain. In the end, of course, not every use case will be handled by the standard toolchain, but hashing out what needs to meet and how to meet them would be an important part of what the council could do.
I may be among those to whom that concern applies. But, as I’ve mentioned before, I really prefer to think of things in terms of functionality that is provided, rather than the names of tools that provide it or about tools becoming “more like” one another. In some ways, yes, I think pip would benefit from becoming more like conda, but I think putting it that way sort of misses the point. Pip, conda, and all the rest are just points in the space of possible packaging tools. A particular change to pip may appear to be “moving towards” conda, but really the goal is to move towards an ideal point in that space, which is neither pip nor conda.[1]
The bottom line for me is that Python should, as an “included battery”, provide certain functionality which it does not now provide. We can learn from conda, poetry, or other tools about functionality that would be useful to users. It doesn’t matter what the names of those tools are or why they exist or who came up with them. What matters is what they do, why people use them, what users feel the standard toolchain is lacking.
For instance, conda does not itself provide anything like
poetry addto automatically update a pyproject.toml, which seems to be something that poetry users see as a big win of that tool. ↩︎
Here’s my concrete proposal:
- Voters are maintainers of projects currently under the pypa org on GitHub (probably any ballot stuffing attempt would be found out and the harm to one’s reputation is probably enough of a deterrent)
- 5 people on the council, nominated by the voters (which includes self-nomination and nominating non-voters; this is how the SC works)
- Powers are final arbiters on anything under the pypa org and whatever the SC delegates to this council
- Don’t worry about any renaming
Everything else can be changed later just like how PEP 13 can and has been updated since the formation of the SC.
7 Likes