@pf_moore +100!! I think it is a great idea for PyPA to develop a process for making official recommendations. And PyPA does have (as @pradyunsg mentioned) core developers who understand whats going on in the python dev space .
BTW - People are confused about what PyPA is as well. So if you do this clarifying that component would be useful
We (pyopensci) are absolutely going to have a semi-opinionated guide and it will live here - packaging tools to be added this week. We will happily adjust our guide to match whatever PyPA lands on. When you search for python packaging, the PyPA guide emerges first in google! people go there and want direction. People who are more expert in this space enjoy having the flexibility of options. But in the end:
opinionated opinions are easier to teach (and learn)
~ @stefanv ( i stole your quote)
I think that should drive decisions if you want to support those who are less expert / beginner packaging people.
As a position, it’s basically “our users are too diverse to offer a single default position” which seems to be the most common response to this question.
The “do” here I would think is moderation. Maybe put up a few bad suggestions so that the people who’d normally argue in Discord threads endlessly are motivated to post their own alternative Any other process is going to require the same moderation, but also decision making.
At least this way we don’t have to decide which is actually the best, we can just leave a range of published recommendations along with enough context for people to decide for themselves. Throw in some kind of internet points (e.g. emote reactions), and I think authors will be motivated to make their own writeup the best.
A Discourse category would suffice. This doesn’t require overengineering.
Go for it. My instincts are clearly way too much towards over-engineering (I immediately thought about how we’d curate the ideas that got proposed!), so I’ll leave it for someone else to make it work. But I’m +1 on the idea if it delivers results.
Yes - i’m very open to contributing there once there is a process for decision making around content!!
The problem right now is that i think there is debate about how opinionated PyPA can be. but we at pyOpenSci can be opinionated while getting feedback from the community so i figure i’ll continue to contribute here in discourse while working on content.
When i see a path to contribute to PyPA i will! maybe our opinionated pieces stay in pyOpenSci’s domain? Maybe some can be moved to PyPA verbatim or with edits and we direct people that way. i’m super open to how things move forward - i only want to help and support the community!
But doesn’t this response conflate having a default option with having a single option? I bet that the vast majority of users would be fine with any existing packaging workflow as the default. This would remove the huge barrier to entry that the all users currently need to figure out which workflow is adequate for them, even though they probably all are.
If you are building numpy, scipy, or projects of similar magnitude, then you need options, but there you are in the paradoxical situation that your tighter constraints probably make the choice of packaging workflow easier than it is for the average package.
Perhaps, but then, build backend is by far the easiest one here. Almost by definition you are a more advanced developer at this point, and should be quite satisfied with a decision tree that filters out “need native extensions”, “needs to generate/relocate files on build” and then tells everyone else to choose one of the remaining 4-5 options based on style.
It basically boils down to Project Summaries - Python Packaging User Guide but arranged by category and linked more prominently. I don’t have any problems with letting the backends self-promote in this kind of context - they’re the most motivated to commend themselves to potential users.
But I don’t think it’s a good idea for this ad-hoc group of contributors to vote on “who wins”, which is what any “official” recommendation from PyPA would boil down to. Listing the known backends that all fundamentally work the same is better - or possibly a feature matrix to make comparisons easier.
I am worried this would lead to the same issue we have right now of some misguided and ill-informed advice rising quickly to the top for the wrong reasons (example the typical sys.path or PYTHONPATH shenanigans) and then stay at the top forever. Or I am not understanding how it would be different than what we already have now (StackOverflow, and so on). Would that be somewhat moderated / curated by PyPA someone in the know (maybe the owners of said project)?
To repeat, there is no “PyPA” in the sense of a group that can moderate like this. Anyone can create such a project, and moderate or curate it as they wish. I guess it’s similar to the various “Awesome X” projects that exist out there.
The project owners could request membership of the PyPA, which would make it a “PyPA project” - but that wouldn’t (for example) give me (as another PyPA member) any ability to curate the project, unless I asked explicitly to be a maintainer.
No, it would be very similar to what we have now, except it would be “official” in the sense that “official” sites would link to it. That kind of thing matters a lot to some people (and doesn’t matter at all to many others).
The main advantage of having it somewhere PyPA members are already active is that they may learn about issues or difficult aspects to their projects they weren’t aware of, or may be able to correct misconceptions. That can be done anywhere, but it’s hard to find on StackOverflow, and even harder to prove your credibility there.
But whether the idea would work basically comes down to who would want to be involved. Though I think that applies to all the other ideas we’ve had here too. There’s only so much value in Paul and I throwing ideas around - either the people who want to write recommendations need to speak up, or the people who want to maintain the chosen winner need to speak up (or someone with another idea could jump in).
I was talking about package creation at large, but even for the backend, I would have been quite happy to never find out that there even is such a thing, unless I was trying to solve a specific problem with the default. (But I accept that, by contraposition, I might not be an advanced developer.)
If you’re planning to write code, you need to choose what language to use.
If you’re planning to write a web site in Python, you need to choose which framework to use.
If you’re planning to produce a Python package, you need to choose which backend to use.
This aspect is no different from how all development works. About the only difference is that other ecosystems dictated a single option from the start and never allowed the range of diversity we have in Python. So people asking for one option are literally asking us to crush that range of options. Fundamentally, we don’t like that approach to “community development”, which is why we’re resistant.
Thanks, Paul and you have clarified the idea for me now, I understand what you mean.
StackOverflow now has something called “Collectives”. If I understand correctly this would mean PyPA could create its own collective. Honestly I have no idea how exactly it works. I guess the idea would be to put some kind of PyPA stamp on some Q&As. Maybe someone knows more about this and can comment whether or not it could be a good match for our use case here. I also do not know how commited to this they are at StackOverflow. I feel like they have been rather inconsistent lately, adding experimental features then removing them later in a relatively short time span.
Otherwise I guess the GitHub Discussions could work as well, but I do not know how it works from a “moderator/curator” point of view.
This is again the false dichotomy between an informed default option that covers most use cases, and mandating a single option.
I’m labouring the point because it dismays me to see that the current state of things is considered not only acceptable but desirable. How has the packaging UX on average, over the population of all packages, improved from when it was “write a setup.cfg file”, and that was that? I am maintaining a few packages, only to share useful code with others in my field, and having to know anything about and keep up with the packaging meta-development is purely a chore.
It’s only a false dichotomy because you believe everyone else will see that the informed default means that they could equally choose another option and be “correct”.
I don’t believe it’s false, because I regularly deal with people who insist that unittest is the only correct way to run tests, because it’s “official” (as well as many other examples across all fields, not just Python packaging).
So my experience says that not everybody will understand the difference between an official tool and a recommended tool. Hence why I’m opposed to having anything that could be mistaken for an official tool (unless it really is the only one under active development and widespread use - pytest seems to be very close to meeting this criteria).
Working in a large ecosystem that migrated from unittest to
testtools/stestr ages ago, primarily for things like subunit
attachment serialization in test results, I’m happy to provide
counterarguments that pytest is not the only actively developed
unittest alternative in widespread use.
Simply saying so is enough argument Even in that space, there’s no universal one tool.
All that said, within a project or within an organisation, you can have one specified tool. e.g. at work I always recommend (and in a few cases, requi… er… recommend really strongly) using pytest, black, and conda. And that’s fine for my limited scope - we also have a style guide (a few, actually). None of that affects anyone outside of the scope that I understand and to some extent, control.
Recommending something at the level we’re discussing here does affect anyone. Ever had someone use PEP 7 or PEP 8 against you? I have. It’s because they mistook it for a global requirement, and not the style guide for a particular project. And that kind of thing is easy to do - let’s not make it easier.
First time commenting here but I wanted to leave my two cents.
I think simple “vanilla” cookiecutter repos is what many people need. A cookiecutter repo for single file module, another one for a multi-file package, another one with a C extension, another one with entry point or “instalable scripts”, etc.
Then people can choose which one to start with and modify and merge accordingly. I understand, though, that it can grow very quickly. But at least have a few ones for people to start with the most common situations, where 90% of the people are.
Along these lines - what would happen if one of the requirements for being a PyPA project (ie a project repo living in the organization) included a bus factor value (x number of maintainers). is that possible? would that encourage maintainers to work together a bit?
the reason i mention it is it’s a worry of ours that so many of the packaging tools have n=1 maintainers which makes me a bit nervous in terms of sustainability / burn out etc. That then would make the project a bit more community oriented as well. more people can more easily engage with more users, etc.
As for the cookie cutters. i think they are a great idea. but i also think if PyPA recommends ones created by others, they should have some criteria around them being maintained as just like documentation. Otherwise they become dated as tools and approaches evolve.
That’s an even more difficult aspect of PyPA governance than the relatively manageable change I suggested at the start of this thread[1]. The establishment of the current rules (PEP 609) was fairly difficult, as far as I recall, and one of the key factors in getting agreement was an explicit statement that the PyPA will not interfere in the management of individual projects. That would specifically exclude requiring any form of maintainer quota, as it very definitely means that the PyPA has no say in who is given commit rights on any given project. Also, of course, you can’t dictate what people want to work on in any case - you could make me a pipenv maintainer, for example, but it wouldn’t make any practical difference to the sustainability of the pipenv project, as I’ve no interest in contributiing to that project.
We could, of course, change the rules and governance, but I don’t think anyone has the appetite for that (I know I certainly don’t). And in the current climate, with a lot of (entirely reasonable!) outside interest in the PyPA, trying to have a public debate on governance runs the risk of burning people out, and as a result making worse the problems we’re trying to address
And you can see how little progress we’ve made on that one… ↩︎
Any other process is going to require the same moderation, but also decision making.