# Python packaging documentation feedback and discussion **URL:** \ **Category:** Packaging\ **Created:** [March 9, 2023, 7:38pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833 "2023-03-09T19:38:46Z")\ **Posts on this page:** 20\ **Page:** 4
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [March 20, 2023, 3:39pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/61 "2023-03-20T15:39:27Z")
> [@BrenBarn](#): > > On the other hand, I also see reluctance to have [python.org](http://python.org) include documentation that is not about the tools that Python provides out of the box. This is the sentiment of “the documentation for tool X documents tool X, it’s not supposed to tell anyone how to decide whether to use that tool”. > > The problem is that this kind of “how do I decide” documentation is exactly what is missing for many users. People can search the internet for something like “how to install a Python library” and find various web pages mentioning pip, conda, poetry, and who knows what else. But it’s hard to find anything telling them how to winnow that down to get something that will work for their needs. Yeah, it’s this position that makes me so positive towards people doing their own opinionated distros of Python (including the tools, docs, and support that will be best for their users). I’ve challenged the position before, but most of the core CPython team _only want_ to do CPython. Even tools like the `py.exe` launcher are considered “outside” the core. And the primary deliverable is a bundle of source files - the pre-built distros are conveniences, not the product. It’s all designed to allow _someone else_ to put together a complete package for users to use. Realistically, Anaconda and ActiveState are the only ones who come close to achieving this vision, but the licenses allow anyone to enter into the space if they choose. So do bear in mind that, short a drastic change in the intentions of the CPython team (via the Steering Council, and probably requiring a governance change), none of this is going to happen at that level. And the people who would need to be convinced aren’t even reading these discussions. Efforts like PyOpenSci are far more in line with what everyone wants. If it went as far as having a pre-packaged distro with its own user docs, it would be totally fine to tell people to install that rather than the [python.org](http://python.org) packages or the Linux distro packages. The problem is still marketing, ultimately. But convincing the web site maintainers to link to your reliable, long-standing distro is far easier than convincing volunteer maintainers to switch to maintaining something that they don’t want to do. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [March 20, 2023, 3:57pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/62 "2023-03-20T15:57:00Z")
One further point I would add is that I think we should try to keep the scope carefully focused. It’s all too easy to take “packaging” as a blanket term to cover everything to do with developing with Python in all its forms. While there’s a clear need in the user community for clarity and guidance in many areas, I think we need to be careful not to assume the packaging documentation (and more specifically the PyPA) has to cover all of that. To make this more concrete, which of the following are “in scope” for the packaging documentation? - Specifying package metadata and build processes - Distribution of libraries for public use - Sharing libraries privately between individuals/teams - Creating standalone applications in Python - Sharing Python _scripts_ with their dependencies, between people with Python installed - Project structure - for libraries, applications, data science, etc (many different use cases here) - Tools and processes for testing - Tools and processes for documentation - Python environment management - Non-python environment management (shared libraries, etc) - Task runners - Workflow, whether automated or manual - Continuous integration and deployment - Deployment of the Python runtime itself All of these have at one stage or another been brought up in “packaging” discussions, but there’s clearly\[1\] no chance of covering all of that in any practical timescale. And there’s a lot of that which frankly isn’t the job of the _packaging_ community. So I think it’s important that we try to come to some consensus on what _is_ in scope, and then agree to restrict discussion to that scope. For what it’s worth, this question of scope is just as much of an issue for any opinionated distribution or documentation - and there’s no reason that every such distro/document should make the same choices (for example, conda includes environment management, but as far as I know ActiveState doesn’t). * * * 1. At least, it’s clear to me 😉 ---
**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\ **Post date:** [March 20, 2023, 4:34pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/63 "2023-03-20T16:34:10Z")
> [@pf\_moore](#): > > for example, conda includes environment management, but as far as I know ActiveState doesn’t I’m not _as_ familiar with it, but they had their [PyPM](https://code.activestate.com/help/faq/#pypm-index) which is being retired in favour of a higher level manager that is used for all their languages and includes the runtimes. The newer one at least seems to cover environment management. [Edit: sorry, forgot to link to [an interesting page for their new tool](https://docs.activestate.com/platform/state/create-project/)] ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [March 20, 2023, 5:08pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/64 "2023-03-20T17:08:16Z")
> [@steve.dower](#): > > The newer one at least seems to cover environment management. Fair enough. Nevertheless, it’s still not a given (IMO) that a distro/document _has_ to encompass environment management. ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [March 21, 2023, 11:02pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/65 "2023-03-21T23:02:09Z")
> [@pf\_moore](#): > > To make this more concrete, which of the following are “in scope” for the packaging documentation? Because no one asked for my opinion 😉 … - Specifying package metadata and build processes 👍 - Distribution of libraries for public use 👍 - Sharing libraries privately between individuals/teams 👍 - Creating standalone applications in Python - Sharing Python scripts with their dependencies, between people with Python installed 👍 - Project structure - for libraries, applications, data science, etc (many different use cases here) - Tools and processes for testing - Tools and processes for documentation - Python environment management 👍 - Non-python environment management (shared libraries, etc) - Task runners - Workflow, whether automated or manual - Continuous integration and deployment - Deployment of the Python runtime itself So for me it’s whatever has to do with packaging (which I know sounds obvious, but I view environments existing to manage packages, so falls under this purview). ---
**Author:** ![lwasser](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lwasser/32/10253_2.png) [@lwasser](https://discuss.python.org/u/lwasser)\ **Post date:** [March 23, 2023, 4:06pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/66 "2023-03-23T16:06:06Z")
> [@brettcannon](#): > > - Specifying package metadata and build processes 👍 > - Distribution of libraries for public use 👍 > - Sharing libraries privately between individuals/teams 👍 > - Sharing Python scripts with their dependencies, between people with Python installed 👍 > - Python environment management 👍 so based on this list of upvoted items… would it be valuable to do a bit of inventory of the existing pypa content in a high level outline form to help identify gaps or areas that need updates, etc prior to the packaging summit in april? and then creating a high level plan that is perhaps implemented by tackling one bullet item in the list above at a time? for example the first - package metadata and build process would be great to tackle. there are so many questions now about setup.py moving from it to pyproject.toml etc etc. ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [March 23, 2023, 4:19pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/67 "2023-03-23T16:19:27Z")
I think so, yes. I’d avoid environment management, personally, as I think that’s probably the _most_ controversial item (I can see why @brettcannon included it, but it very quickly leads into areas that he _didn’t_ select, such as task runners and workflow). > [@brettcannon](#): > > Because no one asked for my opinion 😉 … While I’m on the subject of (not) asking for your opinion 😉, do you think the scope of the PyPA should be wider than the scope of the packaging docs, or do you think the two should be the same? Personally, I think there’s enough confusion already, so we should keep them the same for clarity, if nothing else. I’m aware that puts workflow tools like hatch in a somewhat awkward position, as a result of being part of the PyPA while technically not being entirely in the PyPA’s scope. I think we can resolve that, though, if we need to. And regarding the question of environment management I hinted at above, my view is that the _machinery_ of virtual environments is the responsibility of the core devs (because it’s implemented in the core)\[1\], and the PyPA/packaging docs only cover tools for _managing_ environments. Does that match your view? * * * 1. And that includes the (implied) statement that virtual environments are the official means of creating isolated environments in Python. ---
**Author:** ![abravalheri](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/abravalheri/32/4990_2.png) [@abravalheri](https://discuss.python.org/u/abravalheri)\ **Post date:** [March 23, 2023, 5:43pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/68 "2023-03-23T17:43:29Z")
> [@pf\_moore](#): > > I think so, yes. I’d avoid environment management, personally, as I think that’s probably the _most_ controversial item I agree with this, environment management sounds out of scope. When I think about PyPA I think of “building and installing packages”. When I think about environment management, my mind goes immediately to tools like `tox` and `nox`. > [@pf\_moore](#): > > I’m aware that puts workflow tools like hatch in a somewhat awkward position, as a result of being part of the PyPA while technically not being entirely in the PyPA’s scope. I think we can resolve that, though, if we need to. I understand that we have tools like `hatch` nowadays in PyPA but I imagine that this is due to the fact that `hatch` is “tagging along” with `hatchling`. ---
**Author:** ![BrenBarn](https://avatars.discourse-cdn.com/v4/letter/b/74df32/32.png) [@BrenBarn](https://discuss.python.org/u/BrenBarn)\ **Post date:** [March 24, 2023, 4:40am UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/69 "2023-03-24T04:40:57Z")
> [@pf\_moore](#): > > One further point I would add is that I think we should try to keep the scope carefully focused. It’s all too easy to take “packaging” as a blanket term to cover everything to do with developing with Python in all its forms. While there’s a clear need in the user community for clarity and guidance in many areas, I think we need to be careful not to assume the packaging documentation (and more specifically the PyPA) has to cover all of that. > > To make this more concrete, which of the following are “in scope” for the packaging documentation? > > - Specifying package metadata and build processes > - Distribution of libraries for public use > - Sharing libraries privately between individuals/teams > - Creating standalone applications in Python > - Sharing Python _scripts_ with their dependencies, between people with Python installed > - Project structure - for libraries, applications, data science, etc (many different use cases here) > - Tools and processes for testing > - Tools and processes for documentation > - Python environment management > - Non-python environment management (shared libraries, etc) > - Task runners > - Workflow, whether automated or manual > - Continuous integration and deployment > - Deployment of the Python runtime itself As I mentioned before, I think that, by and large, this is the wrong way to slice it. It is better is to think about what people want to _do_ and then tell them everything they need to do to do that. Some of that may not even be packaging stuff; we still need to tell them how to do it. More and more I question what value PyPA qua PyPA is really adding to anyone’s experience. The topic is not packaging; it is the use of _Python_. Packaging tasks are an important part of using Python; those tasks should be comprehensively covered in the Python documentation. As a specific example, I think that environment management is absolutely essential and needs to be considered in tandem with packaging. Every package that is installed is installed into an environment; you can’t use packages effectively without at least having some awareness of where you’re installing them to… Similar to the discussion of “greenfield” exploration on another thread, I think that beginning by identifying some narrow scope risks a large waste of effort, like the joke about the drunk who dropped his keys in the alley but is looking for them near the lamppost because the light’s better there. The problem with the existing documentation is precisely its fragmentation and lack of comprehensive coverage of relevant user tasks. Creating additional documentation that doesn’t fix that problem will not just not help — it will actually make things worse, by adding yet another layer of partial coverage to the landscape. ---
**Author:** ![pitrou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pitrou/32/28_2.png) [@pitrou](https://discuss.python.org/u/pitrou)\ **Post date:** [March 24, 2023, 9:14am UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/70 "2023-03-24T09:14:29Z")
> [@lwasser](#): > > and our working on this too: > (image below is a DRAFT still!) If I look at this diagram, I conclude “PDM seems the most flexible, so let’s just use PDM everytime. Why should I bother with other less capable tools?”. In the end, the situation is befuddling: arguably there is choice because of differing needs, but nevertheless one tool is able to replace all others? (also, the backend/frontend distinction is frankly not very user-friendly) ---
**Author:** ![lwasser](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lwasser/32/10253_2.png) [@lwasser](https://discuss.python.org/u/lwasser)\ **Post date:** [March 27, 2023, 3:45pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/71 "2023-03-27T15:45:41Z")
i’ve reviewed all of the most common tools - the tools from the pypa survey were my basis for most common - we know that might be a bit skewed given the volume of conda users. I plan to publish some content on each as well just in blog posts so people can see. the BIGGEST difference in the front end tools PDM, HATCH, poetry is how easy the UI is to use (documentation quality and the UI itself) . Then each has a set of features that is slightly different. For instance hatch has nox like functionality & supports matrix environment builds for local testing. pdm doesn’t but pdm supports other back ends and conda and other environments out of the box without a plugin. from the science user perspective of all of the tools right now PDM is the only that supports using other build back ends. This is super valuable for anyone who has a more complex build because many of the core science packages are moving to meson-python. and PDM does work with meson-python. PDM and hatch both provide documented support for c/c++ extensions. but i believe the way to implement this may be a tad different (don’t take my word for that last statement however i haven’t tried it yet just reiterating verbal feedback i’ve gotten). Generally for most users creating pure python packages who aren’t too concerned about the exact structure of their SDist and wheels the backends for flit, pdm hatch and poetry are all fairly similar. the big difference seems hatch and pdm have documented support for c extensions. poetry doesn’t document such support. and all of them have slight differences in what is included in the sdist by default and how you specify what’s included. Finally flit has a lot LESS features compared to pdm and hatch it really just builds and publishes a package to pypi. this is why i do think it’s confusing for a user to see multiple back end choices on a page. if you are more advanced you really understand why you might pick one over the other. however for most i they won’t see a big difference using flit core, pdm-build poetry or hatchling UNLESS they actually unpack their distribution files and look at the contents carefully. our content is live now - there is a table here that broadly overviews features. i go into more detail for each tool below (note i did ask the tool maintainers review this content for accuracy and they did so with wonderful helpful feedback!). [Python Package Build Tools — pyOpenSci Python Packaging Guide](https://www.pyopensci.org/python-package-guide/package-structure-code/python-package-build-tools.html#package-tool-features-table) i hope this helps. i do think there are some great tool options out there. we just need more documentation that guides users regarding where to start if they don’t want to become a packaging expert. i think that is what is important. (just my two cents) ---
**Author:** ![lwasser](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lwasser/32/10253_2.png) [@lwasser](https://discuss.python.org/u/lwasser)\ **Post date:** [March 27, 2023, 3:46pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/72 "2023-03-27T15:46:33Z")
also please again note these are just my thoughts on the situation. i’m also learning a lot about this community as pyOpenSci evolves and our ONLY goal is to help here while supporting the scientific community! ---
**Author:** ![lwasser](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lwasser/32/10253_2.png) [@lwasser](https://discuss.python.org/u/lwasser)\ **Post date:** [March 27, 2023, 3:49pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/73 "2023-03-27T15:49:15Z")
@BrenBarn what do you think about considering a starting place with fewer topics given it would be really hard to start tackling everything at once. maybe an outline with more topics that can be covered as a plan? I only ask this because it took me … about a solid 3 months to get that packaging chapter sorted out and it’s just a high level overview. and feedback was still coming in after i “closed” the review. my worry is if we as a community tackle too much as once, nothing will get done. there are so many things to get right here. just a thought… ---
**Author:** ![sinoroc](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sinoroc/32/1607_2.png) [@sinoroc](https://discuss.python.org/u/sinoroc)\ **Post date:** [March 27, 2023, 4:39pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/74 "2023-03-27T16:39:03Z")
@lwasser Thanks for this effort. On a first skim, it looks very helpful. A first remark, before doing a deeper dive, I wonder if _flit_ needs to be considered at all. My understanding is that _flit_’s main purpose is to be used for _bootstrapping_, because it has zero dependencies. It has a very limited feature set, which as far as I understood is very unlikely to grow, and some hard to navigate corner cases (if I recall correctly, it behaves differently when used from its own CLI or as a pure build back-end). My feeling is that it would be a disservice to suggest it as a candidate for build back-end. Hopefully, others will correct me if my understanding of _flit_ is wrong. ---
**Author:** ![CAM-Gerlach](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cam-gerlach/32/3688_2.png) [@CAM-Gerlach](https://discuss.python.org/u/CAM-Gerlach)\ **Post date:** [March 27, 2023, 4:42pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/75 "2023-03-27T16:42:15Z")
@takluyver can explain it better, I’m sure, but as I understand it Flit’s primary designed use case is to make packaging as simple as possible for users with pure-Python packages who just want a tool that does the right thing with a minimum of options and complexity, particularly scientists and other non-full-time programmers. The design tradeoffs, such as a limited feature set, opinionated choices and not handling less-common corner cases, are all specifically oriented around that goal. ---
**Author:** ![sinoroc](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sinoroc/32/1607_2.png) [@sinoroc](https://discuss.python.org/u/sinoroc)\ **Post date:** [March 27, 2023, 5:01pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/76 "2023-03-27T17:01:05Z")
@CAM-Gerlach Thanks. I need to reconsider flit’s position in my mental model. And the guide even has a [“_Why you might not want to use Flit_” section](https://www.pyopensci.org/python-package-guide/package-structure-code/python-package-build-tools.html#why-you-might-not-want-to-use-flit), so it’s all good. ---
**Author:** ![abravalheri](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/abravalheri/32/4990_2.png) [@abravalheri](https://discuss.python.org/u/abravalheri)\ **Post date:** [March 27, 2023, 5:11pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/77 "2023-03-27T17:11:38Z")
Hi @lwasser, thank you very much for sharing this. Could you please correct the following sections: > - setuptools will build a project without a name or version if you are not using a **pyproject.toml** file to store metadata. > - Setuptools also will include all of the files in your package repository if you do not explicitly tell it to exclude files using a **MANIFEST.in** file Setuptools can derive name and version from multiple files that store metadata (`pyproject.toml`, `setup.cfg` and `setup.py`). The second bullet point is not precise. Setuptools has a set of files that will be included by default and that is not equal to all the files in the repository. We do recommend using `setuptools-scm` though, and the outcome of that should be more or less equivalent to what `hatch` and others do… ---
**Author:** ![lwasser](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/lwasser/32/10253_2.png) [@lwasser](https://discuss.python.org/u/lwasser)\ **Post date:** [March 27, 2023, 6:28pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/78 "2023-03-27T18:28:08Z")
@abravalheri happy to make changes! many thanks for noting this. Rather than diverting the convo here - would you be open to creating an issue in our repo here: [GitHub - pyOpenSci/python-package-guide: A guide to scientific python package recommendations curated by pyOpenSci](https://github.com/pyOpenSci/python-package-guide) i have a few questions for you about this and can show you what happened in a package i created as well 🙂 @sinoroc yup - i think a nice working case for flit also is people making small modules for personal use. it’s light weight and quick to get up and running. Although, i wish it might make a directory structure for someone if they wanted to start from scratch too (minor and maybe out of scope). here is a blog post that i intend to publish soon about Flit - [Learn About Using Flit for Python Packaging - HackMD](https://hackmd.io/GkZlPoQgSJSiO1CmLliuiw) just so you can see the interface. what would be really cool if is some of these tools could just built upon each other extending functionality for various groups of users rather than rebuiliing similar functionality that is slightly different (if that makes sense). i don’t have a good solution this is just an observation. ---
**Author:** ![sinoroc](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sinoroc/32/1607_2.png) [@sinoroc](https://discuss.python.org/u/sinoroc)\ **Post date:** [March 27, 2023, 6:35pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/79 "2023-03-27T18:35:19Z")
Maybe the feedback on pyOpenSci’s packaging guide should be split into its own thread? ---
**Author:** ![CAM-Gerlach](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cam-gerlach/32/3688_2.png) [@CAM-Gerlach](https://discuss.python.org/u/CAM-Gerlach)\ **Post date:** [March 27, 2023, 6:40pm UTC](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/80 "2023-03-27T18:40:49Z")
I can split it from [Python packaging documentation feedback and discussion - #71 by lwasser](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833/71) onward if people agree (there were some posts on that topic much earlier, but they were much more intermixed with other discussion). [Previous page](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833.md?page=3) [Next page](https://discuss.python.org/t/python-packaging-documentation-feedback-and-discussion/24833.md?page=5)