Python packaging documentation feedback and discussion

Packaging
1

Hi everyone! i just wanted to chime in here. i do agree that understanding user needs is the most important starting place as those requirements will define the requirements for any tooling that supports packaging.

I am not sure if this is helpful but i have been working on a review of the existing packaging ecosystem for pyOpenSci. I’ve been asking people what their questions are about packaging and what would be useful (not formally but consistently). i’ve also taught python and R to people for gosh over 11 years now and understand the pain points of someone new entering an ecosystem like this.

Through our peer review process i see packages come in and often they aren’t sure of what best practices are - they are also often happy to adjust their package structure with guidance (but where is that guidance coming from?!).

The most common questions i get are things like

  1. should we use pyproject.toml for metadata or setup.py or setup.cfg
  2. where should we look for answers in terms of packaging? (there are many guides out there and cookie cutters and blog posts) i can assure you MOST of these are very dated. it’s a confusing user experience.
  3. how do we pick a tool to create a python package?
  4. what does a python package look like (should we use src/ etc?)

These are core common questions. and ones we don’t have to impose but can suggest. most people would appreciate those suggestions - they just want to build a package!

i’m just wondering if a list of requirements is the place to start aligned with an overview of existing tools (i have some summary data on hatch, pdm, flit, setuptools, poetry and some of the build back ends as well to contribute to this overview - to be published next week. i’ve also talked to maintainers in some cases to understand where the tools are headed.

the requirements can look something like a list of features the tool offers

front end features

  • can it support different build backends?
  • can it publish to pypi, do editable installs, setup a structure for your package for you (eg PDM init, hatch init)
  • environment support (yikes!)
    etc…

back end features

  • does it support non pure python builds etc
  • what files are included by default in the SDist /wheels etc
    etc…

i also like to look at what the maintainer team looks like (bus factor) and can that maintenance be supported over time.

And maybe it’s a draft document that people can comment on?

Then perhaps a second survey goes out that is focused on the various tools and needs of the community but also that somehow categorizes the types of users (i think the science community for instance has specific use cases that need to be considered). there are likely others.

So after this work you have a list of what we want a tool (or tools) to do and be.
And you have specific feedback from users about their biggest pain points, what communities they are involved with, what their needs are. (separating front and back end tools out)

i think there are some real pain points in this community but one of them i think is related to good clear documentation with beginning to end examples for new users. we CAN build packages now. but users do NOT know where to go to figure out how. And i think many want to look to PyPA - it would be nice if that was the trusted source but people don’t know what PyPA is in many cases!

And also dealing with complex packaging needs is unclear. many don’t even know about tools like meson-python being available! or what build is.

I also wonder - would existing maintainers be willing to team up and work together (is that a dream that can’t happen given the competitive nature)?

id be happy to help on the survey end of things if that becomes of interest to build on top of the survey work already done. we will continue to evaluate existing tools in hopes of finding a single workflow that we could suggest to users NOW while all of this other important work is happening here too and am happy to contribute what we find if its useful!

i hope this very long post is somewhat helpful. i just want to users to have a bit of clarity and if we can support this effort with the work we’ve been doing (which is a huge effort!) i’d like to !

14 Likes
2

FWIW here in exploring front and back end tools i found that people respond well to something concrete. So i’ve pulled content together knowing i’d make mistakes. Then I ask for review while clearly articulating the end goal. i think a small group needs to start this effort and then let everyone provide input.

In a review, someone needs to be on top of moderation to steer conversation. Comments can become circular but that’s ok. Ask questions to get to the bottom of the issues.

my two cents. I’m so appreciative of the work that everyone here does!

3 Likes
3

I think points 1 & 3 could be answered if we spent a bunch of time on updating packaging.python.org (which answers point 2 :wink:). As for point 4, that’s going to be personal/tool-specific.

Personally, I think we might be reaching a point where we should evaluate the state of packaging.python.org in terms of what documentation it contains, what we want it to have, and making sure it has appropriate support/staffing. That exercise will probably also help illustrate where we have some gaps or experiences that are a bit hard/rough that we would like to see improved.

3 Likes
4

I think one of the big things we clarified in the long threads is the distinction between packaging tools and environment management tools.

packaging.python.org covers the first really quite well, as well as the information needed to create a backend, and I think we’re justified in assuming it’s the more technical audience who are able and likely to find published information.

The second is where things are really hard. There are so many dimensions in how people acquire Python, how they acquire packages, how they isolate projects, and how they launch or deploy apps written using Python, that it’s actually not that surprising we have so many options here. And each option legitimately is best in a specific area.

But environment managers seem to be the biggest problem, in large part because people don’t find out they need one until they’ve taken a few steps, and then don’t realise they need to start over in order to use them. It’s also an area that we don’t inherently get, because so few of us are still new to this.

The kind of things that would be great to study and summarise are:

  • what kind of person is likely to start using Python?
  • how do they know that they want Python?
  • how do they decide how to get it?
  • what do they think “getting Python” means?
  • what are they trying to achieve?
  • how urgently do they need to get it done?
  • what docs/tutorials/blogs/videos/friends do they consult first?
  • what assumptions do they make about what they need to do?
  • what are the common problems they are likely to encounter (importantly, these need to be correlated with all the preceding questions)

(Probably more from there, but I’ve run out of ideas off the top of my head.)

Once we have that documented in a way that we can all agree represents our actual user base, we can start tracing through the ways that people’s assumptions about how Python should work don’t match up with the current realities of how it works. From there, we can start looking at things to change and find the people to change them.

We’ve also got a rough idea of things that we could change (or might change). I’m not going to list them here to avoid distraction, but we need those to make sure that we collect enough info about users to be able to decide whether they’re going to align with user expectations or at least be introducible to them on the common “journeys”.

1 Like
5

I completely agree that working on the packaging website would be great!! and then we can all promote the updates to the community as well?

Things like this i think we could enhance with a more beginning to end example using a single tool (i know that could be contentious but…)

Example:

if you look at this part of the packaging tutorial, immediately users are asked to make a decision about back end tools. But most have no idea what the difference is between them. each back end each have subtle differences for pure python in what ends up in the final SDist etc. But for a new user they are already lost here - cognitive overload.

And PDM, Flit are front ends whereas hatchling is a backend.

Screen Shot 2023-03-09 at 4.52.16 PM
Screen Shot 2023-03-09 at 4.52.16 PM1682×582 67.8 KB

this is the type of feedback i got in our review as well :slight_smile: from amazing folks like @rgommers who pointed out things i wasn’t quite getting right.

it’s confusing to users. many new users (not all) but many just want to know how to create a package from beginning to end. they might customize later (build back ends etc) as they learn more (or not!).

5 Likes
6

I agree with this wholeheartedly. As I mentioned on one of the other threads, I think that is one of the most egregiously horrible things in the docs. It epitomizes all the problems that have been discussed in these packaging threads.

Personally I think it would be better to replace that entire section with “sorry, please look elsewhere for information” than to have that completely useless quadruple non-explanation staring people in the face when they go to find out how to do something.

2 Likes
7

I started something to maybe answer this here, but I have no idea if this rough first take at this is a good way to address this or not, feel free to leave feedback.

I firmly believe PyPA should take a stance here. Decide on a canonical project directory structure and development workflow, document it clearly, advertize it, strongly recommend it, and then let it match the default behavior of PyPA’s one packaging tool (if any).
Maybe it could even go as far as saying: “do use black, nox, ruff, and so on” (a bit like create-react-app does). So I guess it could be a cookiecutter template. So that PyPA can focus their resources on a limited amount of things.

Yes! I am not entirely sure if this refers to the “strategy” and “vision” discussions here, but I definitely feel like it is sorely needed here on Packaging for those longer but critical discussions. These very long discussions are full of repetitions and asides (I am probably guilty of some of them) that feel quite counter-productive. For example I guess if there was someone who could promptly split the discussions into a separate topic as soon as some kind of inner thread of 3+ messages forms, this could already be quite the breath of fresh air for everyone and would give more space to both the original thread and the split one.

Same, and same as above I feel like some authority to keep things on track is needed.

Maybe I interpret intentions wrong (and maybe I will get chewed for saying it) but I also feel like starting a topic asking for contributions and opinions, setting rules and an end date and so on, all this to end up not being active in the discussion, not making sure that the rules are enforced that the discussion stays on track and that the discussion ends up in a conclusion of some sort before the end date is kind of… rude maybe? I do not know how else to say it.

3 Likes
8

I don’t have too much to add, but I do want to take a second to share my appreciation for your contributions here @lwasser. Like other commenters, the number of conversations in which people are talking cross-purposes, with different understandings of the purpose of the discussion, and with no clear “goal” left me feeling that further contribution would be fruitless. This is definitely not how we want these discussions to go, and I’m sure that I’m a part of that problem for someone else.

Maybe we can’t solve the problems in front of us in a few Discourse posts. But, your suggestions to take concrete steps on both improving our existing documentation and gaining a better understanding of the actual concerns and confusions our user base faces are very welcome, and feel like a good investment of our collective resources.

5 Likes
9

Same, I had a quick look yesterday and it seems very valuable. It feels like a lot (if not all) of it would be a nice addition to https://packaging.python.org/ as well.

10

As the person that pushed really hard for over six months to get that in, I’d like to defend my stance:

  1. The packaging.python.org tutorial is completely generic. It is useable for any backend that follows PEP 621. As an author of a PEP 621 backend (scikit-build-core), I want to be able to avoid rewriting the entire tutorial, and just point users to this tutorial, and say “this is how you set up a package”. Seeing that selection screen makes it completely visually clear what you need to change to use scikit-build-core instead of the few backends listed. Same for any other backend not listed. By including more than one backend (even two would have been enough), you get something that is not just a single-tool tutorial.
  2. Single tool tutorials can be placed in that tool’s docs. Packaging.python.org doesn’t need single-tool tutorials, IMO - Setuptools docs can hold setuptools tutorials, Flit docs… You get the point.
  3. Packaging.python.org shouldn’t “pick the one correct tool”, which is exactly how a lot of people would interpret it if it listed just one backend here. Just being “first” already is being used as “the official recommendation”! I even considered randomizing the order when a user visits…
  4. PDM & Hatch are the user facing names, but the backend is the one listed. The PDM backend actually changed names, so using the user-facing name is more stable for the tabs. :slight_smile:
  5. Nothing in the tutorial changes if you use any of these backends. All the “subtleties” are for advanced usage, beyond the scope of the tutorial. It’s literally just a “pick one and go on”. Or just go on and don’t change the tabs at all, which IMO is what 90% of new users do. Have you seen the sorts of questions we get? :wink:

IMO, that’s one the best parts of the packaging.python.org page, and it highlights the beauty of PEP 621. I’ve not seen actual learners complain about that part yet. Only people who already know the difference between backends complain. I think that’s because you know about the various subtleties, and understand the weight of the choice. Someone writing a very basic package doesn’t, and that’s okay, they don’t need to. Once they start adding things that go beyond this tutorial, hopefully they start learning about the backends.

If you want a ‘This is how it should be done’ page, it’s Developer information - Scikit-HEP, GitHub - scikit-hep/cookie: Modern Python Package Cookiecutter, and Repo Review - Scikit-HEP. :slight_smile: And that is still completely generic as far as backend (cookie, specifically, has 12 backends, like scikit-build, meson-python, Maturin, hatching, flit-core, setuptools, and more - one size cannot fit all for backends, and that’s okay, at least for now[1]). But it does make hard recommendations about most other parts, like task runner, style checkers, tests, etc.

  1. I do think two more PEPs could potentially really help here. I’ll be proposing ideas in the future. ↩︎

4 Likes
11

i really appreciate the amazing work that you’ve done in this space @henryiii !! you are, i think, one of the most / more knowledgable people in this space right now around each and every tool as far as i can see !!

I think the point to consider about that guide is that :

  1. it’s geared towards users with an expected level of expertise that new to packaging users don’t have.
    combined with
  2. people are confused about what PyPA is and how packages end up in the PyPA organization and what it means for a package to be there. (speaking to @pradyunsg comment about a process for how packages end up in that org but not speaking for Pradyun of course :slight_smile: )

if that is what PyPA wants the guide to be, then that is ok. but there is a gap. In the end this expected level of expertise will confuse those who are new to packaging. Because right off the bat a question of which back end do i use is confusing because

“why are there different back ends?”, and “how do i decide which one” and “What is a back end vs a front end vs a envt management tool”.

SO perhaps the question we need to answer first is:

“WHO is this guide for?” and what do they already know?

3 Likes
12

Yes, precisely. I don’t think we should ever write any piece of documentation without answering this question. And most of the time, the answer should be in the first paragraph.

People get far less frustrated with documentation when they already know it’s not intended for them, even if they still read it and may still get value from it.

5 Likes
13

The PyPA is nothing more than a loose association of projects. Anyone who is a maintainer of a project that’s in the PyPA is a “PyPA member” but there’s no concept of “speaking on behalf of the PyPA”, we all just offer our own individual opinions.

Packages become part of the PyPA simply by asking - there’s no real criteria. The existing maintainers vote, but it’s mostly a formality, I don’t recall a project being refused. As to what it means for a project to be part of the PyPA, basically nothing. We have projects in the PyPA that compete with each other, so it’s not any form of seal of approval. Many of the major packaging projects (like pip, setuptools, Warehouse) are in the PyPA, but not all of them (conda isn’t, nor is Poetry, to give a couple of examples).

The problem is that people expect the PyPA to be more than this. And in some senses it is, because we have a percieved authority that means what “the PyPA says” matters to people. But there’s no “PyPA board” or anything that can make statements on behalf of the PyPA.

These strategy discussions can also give a misleading impression - they aren’t being managed by the PyPA (@smm is employed by the PSF, not the PyPA) and PyPA members are just like any other community member as far as these discussions are concerned. We (the community, not the PyPA) are trying to ensure they deliver something useful, but it’s not at all clear how the PyPA fits in - on workflow tools, for example, the discussion has covered many possibilities, including both PyPA and non-PyPA tools. If we agree on a strategy that is based on a non-PyPA tool, does that make the PyPA less relevant? I don’t know. No-one has said anything either way.

I’m not surprised people are confused, to be honest. I’m confused, and I’m a PyPA member :roll_eyes:

1 Like
14

I don’t think the target audience is going to think “how do I decide which one”. They are going to do is copy and paste the first thing they see and move on. I think I have yet to see someone say they got confused about selecting a backend, exactly because the audience doesn’t know enough to differentiate backends, and that’s fine. The point of the selector is to clearly visually indicate that this tutorial is generic and keep the “packaging.python.org” guide from forcing a single tool. Feel free to look at actual user feedback at GitHub - pypa/packaging-problems: An issue tracker for the problems in packaging - I’m a mod there, and I hear from the target audience all the time. The main issues we get are things like not being able to spell “hatchling” (hatchking?), and things like that. I haven’t seen one complaining because they found the selector and got confused by it. I didn’t see a sudden flurry of issues after this went live, just like I predicted - actually I think issues went down a little bit. I might be wrong about that, though.

I expect if a reader doesn’t know about multiple backends, they probably don’t even know the selector is there and something they can click on - it’s extremely unintrusive - they don’t even look like tabs, IMO, unless you are really used to them. Again, it’s only people who already know that questions like that even exist and are important that seem to then say that “other people” will be confused by it. I understand it is a deep and confusing topic, but I also know that new users don’t know that.

The original design was to allow the projects (like hatchling, flit, etc) to link to this page with their project pre-selected, see feat: allow tabs to be selected via URL by henryiii · Pull Request #30 · pradyunsg/sphinx-inline-tabs · GitHub - but that’s still sitting there, @pradyunsg :wink:). You can do that with scikit-hep.org, like: https://scikit-hep.org/developer/pep621?tabs=flit

The fact this tutorial works across multiple tools actually pushed setuptools to be smarter by default when using pyproject.toml! This is a big, real benefit that helps new users (even ones that don’t read this tutorial!) far more than hypothesizing about users discovering these tabs and be confused by them.

3 Likes
15

I don’t think there is anything wrong with that section being there. The problem is it needs a bit more information. Someone trying to follow this guide does at this point need to make a choice and no information is given to make that choice. Immediately when I first saw that section and the different options I thought:

  • There must be some reason for having these different alternatives like they have different capabilities or something.
  • I don’t know what those reasons are though.
  • Those reasons may or may not apply to my project but I don’t know how to evaluate that.
  • Possibly making the wrong choice here will mean that I have to change to another one later on which might be costly so it is important that I get this right now.
  • I have no idea how to get it right now though.

Does this mean that I now need to go and read the docs for each of those projects before I can continue with this guide?

This makes sense if you have the premise that the user reading this guide first new that they wanted to use scikit-build-core, read the docs for that, and then followed a link to the packaging guide. But how do I even know whether to use scikit-build-core?

Somewhere there should be something that actually compares these options and explains in simple terms how you might choose between them. I can rule out most options immediately just by knowing that I need to build extension modules and most of these don’t support that. I shouldn’t have to go read all of the docs for each packaging tool separately to find that information.

As far as I can tell all of poetry, pdm, hatch and flit ship with simple backends so that they can support the PEP 517 interface but there is no real reason to use any of those backends independently of the frontend and apart from pdm none of these frontends can be used with a different backend. The important information that I need to see when considering these is basically:

  • Apart from setuptools these backends only support pure Python projects.
  • The only reason to use any of these backends is if you are also using the corresponding frontend.
  • The different frontends just provide different ergonomics around development workflow and there is no strong reason why any pure-Python project would need to use any particular one of these compared to the others.
  • Having chosen one of these frontends you should really go and read their documentation rather than this guide.

I don’t know much about these different tools but I have used some. One that I haven’t used is flit but that looks like the simplest. If I was going to give one as an example for the beginner guide I would use flit because it is nice and simple and has good short docs. It shouldn’t be controversial to just say something like

We will use flit in this example because it is the simplest but there other options with more features and here is a link to a page that compares them.

2 Likes
16

^^^ yes this!! i think this might be where deciding who your (PyPA’s guide) target audience is - is profoundly important. maybe it’s not new users. I don’t think new / less technical users will be hanging around the github issue where you are talking about what is added to the guide. they might read the guide however! i want to share two images below from a question i asked online last week about packaging questions that summarize well some of the feedback we get. NOTE: our audience is science specific.

Screen Shot 2023-03-13 at 10.54.51 AM

Screen Shot 2023-03-13 at 10.54.42 AM
Screen Shot 2023-03-13 at 10.54.42 AM642×670 86.2 KB

17

FWIW we are creating that type of content @oscarbenjamin

and our working on this too:
(image below is a DRAFT still!)

python-package-tools-decision-tree (4)
python-package-tools-decision-tree (4)2200×1500 449 KB

@pradyunsg am i steering the goal of this thread to the left? i want to check in to make sure we are getting to where you’d like us to be here in this thread.

5 Likes
18
  • Apart from setuptools these backends only support pure Python projects.

Hatch has plugins that support compiled languages. I think PDM does or will. Other PEP 621 backends like scikit-build-core, meson-python, and Maturin certainly do.

  • The only reason to use any of these backends is if you are also using the corresponding frontend.

Sadly, everyone making a backend also made a frontend. I don’t like it, I wish Flit didn’t exist, and I wish Hatch/Hatching and PDM/pdm-backend didn’t share names as closely as they do. I probably haven’t helped matters by putting hatching under “hatch”.

Personally, I almost never use any of these frontends, and don’t recommend them unless you need something specific - PDM for lock files, or Hatch if you want to have a all-in-one that includes task running (replace tox/nox) but don’t need lock files. The backends are thankfully mostly completely unrelated to the frontends, though! You can use any backend with PDM. Either now or very soon, you can use any backend with Hatch. Flit is (now or soon) the odd one out here - you have to use flit-core if you want to use flit.

For backends, hatchling is the best choice - it’s simple, automatic, fast, uses .gitignore for file inclusion, and has great error messages. Flit-core is a no-go, the author strongly refuses to let it be the “suggested tool” or default, period. And it has issues with file inclusion if you have a non-simple package - it only uses git for files if you use the Flit frontend! Setuptools is bulky and can be difficult to configure, and some major packages limit setuptools to versions that doesn’t support PEP 621. pdm.backend is great, but not used as heavily as the others. It’s there so that the list isn’t just all PyPA projects (and it’s quite good).

I don’t think this is about frontends at all. We can (or do?) list/compare frontends elsewhere. Having a table (elsewhere!) and discussion comparing backends is fine, and could be linked here. But I think it shouldn’t interrupt the flow of the lesson too much. Most users should just move on and complete the lesson (and that’s what they do) without worrying about it, and then be able look into it more later. Most users will likely be perfectly happy with Hatchling, which is why it’s the default / first. It was quite intentional that this has very little discussion about differences between backends.

19

That is helpful background. However, at least for me, that is not at all what comes across in the actual docs. Specifically:

This tutorial walks you through how to package a simple Python project. It will show you how to add the necessary files and structure to create the package, how to build the package, and how to upload it to the Python Package Index (PyPI).

If the tutorial does not tell the user which backend to use, nor explain how to choose one, then it does not successfully walk the user through the necessary steps.

Open pyproject.toml and enter one of these [build-system] tables:

When I see “one of these”, I think that means the following table is exhaustive. It means these are the only options. That seems to be the opposite of what you intended, because you said you want it to show that the same structure can be used for various backends, even ones not listed here (or ones that don’t even exist yet). If the table is just a table of examples, then the word “example” (or some synonym like “sample”) needs to actually appear in the surrounding text.

Below the table is an explanation of what the requires and build-backend keys mean. This should come before the table (“add a section build-system containing two entries…”), making it more clear that the things in the table are just examples of this.

My experience is the exact opposite. Or rather, perhaps there are no complaints because people are so bewildered they immediately go look elsewhere. People who don’t even know what a backend is will be totally baffled by this section of the docs.

Someone writing a very basic package doesn’t need to be told how to use multiple backends. They only need to be told how to use one, namely whichever one makes it easiest to prepare a very basic package, and be told that others exist, and be linked to resources that explain how to choose one if and when their needs grow more complex.

Moreover, although I realize the ship has probably sailed on this, I don’t fully understand why it’s necessary for a basic package to encode a commitment to a particular backend as part of the metadata at all, when for a basic package probably any backend will do.

I confess that, beyond these issues, I still do not really see the purpose of telling people about the various backends. @lwasser’s flowchart is great and something like that should absolutely be included in the official docs somewhere. But what is the benefit of having to through that flowchart instead of just saying “Here is a tool that will do all of those things”? Looking at the flowchart, it seems like PDM works for almost all of the situations. Why not just say “use PDM unless you know you have a reason not to”? (Also, although the flowchart is great, it still doesn’t explain how to choose among the backend choices at the end of the flow.)

(Sorry for following up on the off-topic branch of this thread, but hopefully this can be split off into a separate thread that could focus on concrete improvements to the docs.)

1 Like
20

Agreed that these comments should be split to a separate thread. That hasn’t happened though and I do want to follow up on these points.

I also don’t really understand the point of talking about the backends here. Taking a step back the original purpose of allowing backends (as I understood it) was to allow people to make more complicated things like scikit-build-core, python-meson, pymsbuild etc without needing setuptools to be able to handle every case from the simplest to the most complex.

For the usecase described in the guide and the backends mentioned the differences between what these backends actually do are so small that there shouldn’t really be any need to distinguish between them in pyproject.toml. It basically comes down to:

  • Select some files to go into wheel/sdist.
  • Put them in zip files.

I realise in practice it is a bit more complicated than that but we really don’t need to have multiple tools for doing this and multiple (subtly different!) ways to configure it.

The guide says:

The files listed above will be included automatically in your source distribution. If you want to include additional files, see the documentation for your build backend.

  • Why is this statement so vague? The files listed above? What about other files not listed above?
  • Why should the guide bifurcate here on such a basic point as specifying which files should be distributed as part of the project?
  • Also why do all of these backends do this differently and require the information to be specified in different ways and in different sections of pyproject.toml?

It would be better to agree on a common section in pyproject.toml for pure Python projects that has sane defaults and standard syntax for include, exclude, etc with clearly defined semantics. Then the guide could explain how to fill out that section and just say that it doesn’t matter which of the mentioned backends is used for pure Python projects. As it is I can’t even understand the most basic point (specifying which files to distribute) without choosing a tool and then going and reading the docs for that tool instead of this guide.

The flowchart from @lwasser is good because it addresses what is actually the first thing that someone currently needs to do which is to choose a packaging tool. I note that it reframes the problem in terms of choosing a “tool” rather than a backend/frontend and the flowchart only distinguishes backends when talking about projects with extension modules. Again that suggests that we shouldn’t really need to talk about alternative backends in the way that the guide does unless we are talking about building something more complicated.

This draft doc page that @lwasser linked to is excellent. There is a lot more information on the linked page comparing these tools than just the flowchart.