TL;DR: PyScaffold is a command line utility for bootstrapping Python packages that attempts to follow the Unix philosophy. I would like to propose the project as a candidate for joining the PyPA. I believe PyScaffold is a nice fit, since it closes some gaps not covered by other tools like setuptools and twine (i.e., project generation/initialization)
Recently we have seen a lot of changes in Python packaging, such as the emergence of new packaging solutions such as flit and poetry, culminating with the approval of PEP517/518. These changes are very welcome and foster innovation while making it easier for end users to create and publish Python packages. However, I have also seen people misinterpreting them and disregarding extremely good tools such as setuptools (even referring to them as “not modern”).
I believe that one of the reasons for the misinterpretation is that people used to have a hard time creating setuptools-powered packages manually. The maintainers of PyScaffold have been working hard to make this task easy by providing a project generator that builds upon the strengths of the PyPA ecosystem and complementary tools such as setuptools-scm, tox, pytest and others.
Since PyScaffold also implements the best practices from the PyPA packaging tutorial, it completes the compelling story of the PyPA packaging user guides and tools. Thus, we would love to cooperate more closely with you either by becoming an official part of PyPA or being listed as “Non-PyPA Projects” under your key projects page.
Please let us know if you have any questions about the project or feedback about the proposal.
I see no reason why adding PyScaffold as a “Non-PyPA project” in the packaging guide should be a problem, so for that one I’d suggest just submitting a PR adding yourselves.
As for joining the PyPA, I’d not heard of PyScaffold before now, so I’ve just downloaded it to give it a try, but on first glance it looks very much like the sort of project that would be acceptable under the PyPA banner. The rules on joining are here, basically ask to join and adopt the PyPA CoC, so you’ve done the right thing to get that process started
I’d be slightly against this. Accepting this project would say the PyPA gives blessing that whatever this project decides is the right way. Some of the choices there are personal preference so I’m a bit weary what message this would send.
Can you please give some clear examples of controversial decisions made by this project?
I did not use this project myself but I was aware of its existence for a good number of years
I find the idea extremely useful for people that go into publishing new packages in particular.
PyPa group was known to be a bit secluded in the past and that did not help it grew well. I know at least few python devs that told me that they do not even want to bother engaging with pip/setuptools due to negative previous experience. I would try to avoid actions that may deter others from becoming more close with pypa. My personal take is to welcome new people/projects even when they may have divergent ideas, as long they play nice and are willing to compromise for the good of the community.
I have another example pip-tools project which is under different org because people were afraid that under pypa it would be much harder to maintain than under jazzband org, which is very welcoming by nature.
Many controversial points might be raised here. Use flit or setuptools. Use flake8 or pylint or neither. Use tox or makefile, pre commit? I could keep going.
We already have that situation with sampleproject, and with the advice given in the packaging user guide.
While I agree that we should be very clear that PyPA member projects should not use membership as any sort of “marketing” (we really don’t want another mess like the pipenv episode), I don’t think that we should be refusing to accept projects that make choices for the user. After all, “too many choices” is a well-known criticism of Python packaging, how can we credibly say we don’t want to support a project that’s trying to help users deal with that problem?
The idea behind PyScaffold is to actually be non-controversial by adapting the best practices which are already in the community but making them easier accessible for beginners. This is why we have chosen from the beginning to favor for instance PyPA projects over non-PyPY like setuptools over flit. PyScaffold wants to integrate all the bits and pieces that are already there and doesn’t want to invent something on its own. In this, it differs from projects like flit or poetry.
By cooperating with PyPA we hope to achieve this goal even better for our users and to have some kind of convergence towards a unified way of creating Python packages simple and easy. A bit like what cargo is for Rust. We would love to have your support in this and welcome the discussion.
Just to be clear, “being a PyPA project” does not imply that it’s necessarily best practice. In fact, the PyPA’s goal of promoting standards like PEP 517 is specifically directed at allowing people more choice in what build backend to use - so in that sense, we’re very much against making setuptools any sort of “recommended” or “best practice” backend.
On that note, I hope you understand that being a PyPA member would not convey any sort of “approval” for PyScaffold. Rather the opposite, in fact - I’d personally be happier if you didn’t promote yourselves as the “tool of choice”, because of the fear that we’d have another pipenv-style controversy over the PyPA approving one tool over another. (But that’s purely a personal feeling and not any sort of requirement - one of the PyPA non-goals from PEP 609 is to not micromanage member projects)
Stepping aside from the point of contention here, regarding best practices and PyPA’s role + perception on that front for a moment…
The process of adding a new project to the PyPA, is: A PyPA member (i.e. someone with triage / commit bit on an existing PyPA project) writes an email to pypa-committers (a publicly-archived, invite-only mailing list), where a vote is undetaken.
So… basically, you’ll need to convince someone who’s already a PyPA member to initiate a vote – should you decide to go this route.
A PR for this would be more than welcome right now! We can move that content over to another heading later, should the membership status change.
Thank you very much Paul, I previously had a look on the page and that was the reason why I posted it here. Currently we imply that we follow the PSF CoC (which is the same of PyPA right)? But I will make sure to explicit mention it too.
I completely see were this position is coming from and I am sorry for the misinterpretation that the text might have caused. The original intention was to market PyScaffold as “an obvious choice” or a “very good choice” for developing packages. I am completely OK with changing that text and I will have a look into it.
Thank you very much Bernat from bringing this discussion (which I think it is an important one).
I think that the important part here is to understand that being part of the PyPA is not synonym of being “recommended”. The reason why I requested project membership is because I believe PyScaffold complements the ecosystem that exists around setuptools and setuptools-scm, which objectively exists under the PyPA umbrella. I personally want to see people that choose to use setuptools having a nice/easy time, enjoying sharing packages and contributing to the community.
PyPA currently “contains” setuptools and setuptools-scm but this does not mean that it disregards other tools like flit or poetry. On the contrary, I can see a lot of effort, thought and support coming from members of the PyPA on things like making editable installs or PEPs around packaging that directly contribute to other tools (sometimes even under the risk of making the current status of setuptools look a bit outdated, like PEP 621).
If in the future PyPA decides to include the membership of both flit and setuptools, I don’t see any problems with PyScaffold being part of PyPA. If at some point the community decide that setuptools and setuptools-scm should not be part of PyPA anymore, then it would make sense to move together with them, since I see PyScaffold as being a part of the setuptools ecosystem.
Thank you very much @pradyunsg, I will work on this PR.
Regarding convincing an already member, what I hope to convey is that PyScaffold is our way to contribute to the ecosystem that exists around setuptools / twine / wheel / build, etc. We genuinely believe that PyScaffold can help users to achieve their goals faster and also alleviate a bit the mysticism that using setuptools is complicated. Therefore joining PyPA would be a sensible conclusion.
Thanks for making this clear @pf_moore. I think @abravalheri provided a perfect answer in saying that PyScaffold surely belongs to the setuptools ecosystem in that perspective. Additionally, it incorporates the project structure as stated under PyPA’s Packaging Python Projects tutorial and adds additional, optional features considered “best practices” (which is always open for discussion) like pre-commit, tox etc. Consequently, I believe it’s a suitable addition to PyPA as it integrates the packaging tutorial, setuptools, twine, wheel, trove-classifiers, pip, pipx, etc. in a unified, easy-to-use tool with exhaustive documentation.
That being said, I understand your hesitation to include PyScaffold since PEP 517 is specifically directed at allowing people more choice (not only setuptools) in what build backend to use, as you noted. However, to me, the inclusion of PyScaffold and compliance with PEP 517 does not imply a contradiction. Looking at open source projects like Linux, (e.g. the various init and sound daemons always converged to some “goto” default over time but also always allowed to choose differently if wanted), shows that you can have a choice but also sane defaults at the same time.
These sane defaults are what PyScaffold tries to provide to its users in an easy way.
To be clear, I don’t personally have any particular hesitation here. I don’t have the time to propose/manage the vote myself, but FWIW, I’d vote yes.
I share some of the same hesitations as Bernát. We currently have a precedent to recommend setuptools, which is understandable given its historic position in Python packaging, but I am not sure if it’s the best path forward.
Different backends have different goals, features, and UX. I think the move towards the standardized build backends is a great leap forward in terms of progress as we not longer have to have a single build backend that must fit everyone’s use-cases, and instead we can have multiple backends complementing each other in terms of application, feature parity, target audience, etc.
So, @FlorianWilhelm would extending PyScaffold to have templates for multiple build backends be in scope? I think, ideally, I’d like to see PyScaffold being able to generate flit, poetry, etc. projects and having a documentation page comparing all the supported backends so that users can choose what fits them the best.
I understand that this would be a pretty big change, and it is just my opinion, so please don’t feel pressured to adopt it. I just wanted to raise the question.
Hi @FFY00, in principle PyScaffold is designed to be part of the setuptools ecosystem (same way as setuptools-scm) and I don’t see the focus of the development changing directions.
That said, we do have an extension system (https://pyscaffold.org/en/stable/extensions.html) that can handle such use case, and we are open to support the development of community extensions.
Thanks for sharing your points @FFY00. I also think that having the possibility of different build backends is a huge benefit. It allows for easy experimentation and this spurs more innovation in the sense of evolution but maybe also revolution when one new backend replaces another.
I am not sure though about the fact that the backends complement each other. You can only choose one backend at a time and then you are bound to its features. As in the case of flit, I rather see setuptools adapting cool features of flit and in the end, almost all features are not mutually exclusive, so one backend can integrate them all. This notion is also mentioned in Why use Flit? in a subtle way.
Maybe a bit heretic question: Why do I need as a “default” Python package developer a set of different build backends anyway? Wouldn’t I maybe just be interested in someone telling me: just use X, concentrate on your actual programming task, and be fine? For this reason, I think it makes sense to prefer one approach over the other until the other has overtaken the first.
For me, this is the spirit behind PyScaffold. Do not overwhelm developers with tons of options, rather have sane defaults and a few tuning options for advanced users. For this reason, PyScaffold chose setuptools as it gets the job done really nicely in most cases and evolved well over time. If flit would ever overtake and become better than setuptools, I would rather be inclined to drop setuptools from PyScaffold and replace it with flit than to add it as an additional option.
This reminds me a bit of the discussion of Linus Torvarlds about Con Kolivas’s Staircase Deadline scheduler by the way. Allowing and nurturing experimentation is always a good thing and sometimes a scheduler gets replaced by a better one like in the case of the Completely Fair Scheduler replacing the former one but in the end, there is always only one supported and recommended scheduler in the kernel at a time. This avoids complexity in other components and relieves users from making decisions that need a lot of domain knowledge.
I really like this open discussion and exchange of thoughts about this topic, btw
… if we reach a point were PyScaffold have extensions for different backends specialised in different niches, I will be very happy and would love to have a comparison page for all of them.
I completely understand your positions Filipe and Bernát, and I agree that the community should foster different solutions that are better in different scenarios, and that diversity is a very good thing.
But I also think that “not recommending” or “not giving blessings” to setuptoolsshould not equate to prevent PyPA from hosting/fostering projects that focus on improving the user experience within the setuptools ecosystem or complement the functionalities of the setuptools project itself.
setuptool is in awesome project, that have been serving the community very well, and PyScaffold can help people to have a better/easier time when they decide to use it.
I learned about the existence of PyScaffold also only from this post (I’ve been using cookiecutter for this kind of stuff), but after reading about it I am generally for PyScaffold joining.
I understand and whole-heartedly appreciate the view point of not wanting to promote one specific tool. However, there is a significant population who only want a tool for each job, and providing a coherent solution for people to follow along is, in my opinion, as much important as providing all the build pieces that composite well together. One particularly nice thing I feel about PyScaffold is it is able to provide the Packaging Python Projects tutorial a proper “story arc” to tie all the related pieces together, instead of having them scatter around like individual scenes.[1] And if we need this coherent story line (I think we do), a line must be drawn somewhere for what should be more prominently included (the main arc) than others (side stories), and whether a project is managed by PyPA is about as best a rule one can have (and for those not covered by PyPA project we can choose whatever seems more popular, e.g. Tox and Pytest, and leave room to swap them out when the community landscape changes).
With that said, I do have concern over whether PyScaffold will fulfill the vision. This is not related to the technical aspects of the projects, but more about (hopefully minor) adjustments needed to further tidy up the narrative (some are already discussed), and—most importantly—PyScaffold’s committment to the idea. I want to hear from PyScaffold folks whether they’d be interested in taking up the task of providing the story line, write and edit the content on packaging.python.org, and shepard the effort through community reviews to its completion. This is a very difficult task, but one I feel should be “part of the deal” for such a tool to make sense as a PyPA project.
[1]: A bit more context about this analogy. I am referring to video making, a topic I’ve always found fascinating. Movies are performed by actors and coordinated by the director, and they are generally the people that take most of the credit. But some of the less appreciated people that make the story “come together”, screenwriters and editors, however, can literally make or break how a movie is eventually evaluated by the audience. Python packaging is, right now, mroe like The Phantom Menace than A New Hope because we lack a good screenwriter and editor to create a complete thing instead of all of its parts.
