Python Packaging Strategy Discussion - Part 1

Packaging
136

Thanks @abravalheri for expressing that point of view. I have similar feelings around respecting maintainers time - both what they’ve done in the past, and what we may be asking of them in the future. And I think the standardization of metadata in pyproject.toml and of build interfaces (PEP 517 & co) is one of the success stories of Python packaging. No need to turn back on that one and aim for unification of build tools imho.

Overall I think we still are trying to figure out what is feasible and a good idea to unify, or not. The message from users is that the current state of things is still not great, and to please unify something - but it’s very much unclear what that something is.

@pradyunsg asked that question pretty explicitly. I’ll repeat it here, with my answers:

  • Unification of PyPI/conda models → NO
  • Unification of the consumer-facing tooling → NO
  • Unification of the publisher-facing tooling → NO
  • Unification of the workflow setups/tooling → YES
  • Unification/Consistency in the deployment processes → NO
  • Unification/Consistency in “Python” installation/management experience → NO
  • Unification of the interface of tools → YES (as much as possible)

It’d be great to see others’ answers to this.

Regarding some of the other topics in this thread, I think they come in because there’s a number of inter-related things here. Because if you say something should be unified, you should at least have some level of confidence that it’s a good idea to pursue that unification and that there are no hard blockers.

I wrote a blog post with a comprehensive possible future direction, however the content in there all follows from a few things: the what to unify Qs above, Steve’s question on system integrators, and the assumption that major breaking changes have to be avoided. I’d really like to get a better sense of whether others have a similar understanding at this very highest level.

2 Likes
137

OK, here’s my answers.

  • Unification of PyPI/conda models: NO.
  • Unification of the consumer-facing tooling → NO, with a caveat. I don’t think we should try to force maintainers to work on a single tool, but if competition between tools results in users choosing a clear winner, I think we should accept that.
  • Unification of the publisher-facing tooling → NO. I assume this means things like build backends.
  • Unification of the workflow setups/tooling → PARTIALLY. I very definitely don’t think that (like cargo) we should mandate that every time anyone uses Python, they should create a directory, containing a src subdirectory and a pyproject.toml. The workflow of writing a simple script (with dependencies) in a scratch directory full of “other stuff” is an entirely reasonable workflow that we should support. Having said that, I support unified workflows for the tasks of “write a Python package”, and “write a Python application” (althought I think the latter is something we’ve traditionally ignored in favour of “write a package with a script entry point”).
  • Unification/Consistency in the deployment processes → NO. Although I’m not 100% sure what this entails. It shouldn’t be user-facing, though, which is why I say “no”.
  • Unification/Consistency in “Python” installation/management experience → NO. Although I think we should accept that this is not under our control, and like it or not, the main Python website is where people go for advice on where to get Python from. So we should work with the guidance given there, not fight against it.
  • Unification of the interface of tools → YES (but see below).

I’m not sure I understand the difference between “consumer-facing tooling” and “workflow setups/tooling” though. For the purposes of the above, I’ve taken the former as meaning the actual software, and the latter as meaning the processes. So we can have hatch and PDM, but they should manage the same project layout, expect tests and documentation to be laid out in the same way, etc.

As regards “interface”, there are two aspects - low level details such as the names of options, configuration files, etc., and higher level concerns like what subcommands a tool supports. For example, I’d love to see a shared configuration mechanism, so that users can set their network details and preferred index once. And I’d like a common set of workflow commands (things like “run an interpreter in the project environment”, “run the tests”, “build the docs”, “build the project artifacts”, “publish the project”, …) But I don’t want this to be an excuse to argue endlessly over whether an option should be called --config or -C. And I definitely don’t want it to override questions of backward compatibility for individual tools (which should very much be the tool maintainer’s choice).

Regarding the other discussions in the thread, I support better integration with, and support for, other system distributors/integrators. But I strongly disagree with any suggestion that PyPI and wheels should no longer be considered the primary route by which (most) users get Python packages[1]. Having said that, I think such support needs to be a two-way street, and if “other system integrators” want to be supported, they need to engage with the community as a whole and get involved with this process - otherwise, we should accept that what support and integration we provide will, of necessity, be limited (e.g., I don’t think we should try to write a “how to use apt to install Python packages” page in the Python packaging documentation, but we could link to a Debian “how to use apt for Python users” page if they provided a suitable link that was written for the same audience that we are addressing).

I also don’t think it’s at all clear from what I’ve heard of the survey results, what the users are asking for in terms of the above questions. And I think that’s a far more important question than what we think[2].

  1. Maybe it’s a case of “worse is better”, but I strongly believe that without PyPI and wheels, Python would never have achieved the popularity it has. ↩︎

  2. Although if the users are asking for (for example) “cargo for Python”, then my response is “great, I hope someone writes it for them”. Just because the users want it, doesn’t mean that’s where I’ll personally devote my open source (volunteer) time. ↩︎

4 Likes
138

Language integration has been mentioned a couple times but without explanation of what that would be.
I don’t see what people mean. The language has an import system, with sys.path supporting multiple sources, and the site module handling site-packages and user-packages locations. Then separate installer tools can look for distributions and install them in the right locations. What else could the language do?

1 Like
139

I’m going to refrain from answering because every “no” makes my day job harder, but that doesn’t mean a “no” doesn’t make more sense for the community.

Same here since we are talking about programmers installing stuff to program. Is the difference, “I’m using Python code, but not writing any” (e.g. running some CLI app) versus “I’m writing Python code myself that requires installing something”?

And what does “workflow setups/tooling” mean? Would trying to standardize where environments are created, named, and stored make sense in this scenario (which I’ve been asking the community about over on Mastodon lately)? Is this standardizing on Nox and/or tox? Or is this more about src/ layout and how to specify your development dependencies separate from your installation dependencies?

2 Likes
140

I’m maintainer (and only user other than former colleagues inheriting my code) of snoap which admittedly is not a packaging tool but it does do packaging and deployment via Poetry and pip.

The main reason I devised snoap is to address this distinction in a way that was appropriate for my work environment at the time:

This thread has gone very deep into packaging with native and/or compiled dependencies. I think that’s totally valid because of you pull at any string in a Python environment, you’ll probably hit that stuff sooner or later. However, I’m not sure if the users who answered the survey were really thinking in those terms. I’d guess (and it is a guess, could be way out) that most respondents are not wrestling with those issues on a day to day basis. They are probably more bothered by nagging doubts like “if I use Hatch and my colleague uses Poetry, can we collaborate on a project?” or “I want to share this useful utility that I wrote in Python with my non-technical colleagues, but it has external depencies and I’m dreading having to talk them through how to create a virtual environment” or “is it best practice to set a maximum version of dependencies in pyproject.toml or not” or “I really feel like I should know what setup.py is as I see it all over the place but I’ve never needed one”

This isn’t to say low-level packaging is not the root of some of these issues, but it would be useful to have a clearer breakdown of exactly what user experience is driving the desire for unification, what those users mean by unification, and how they think it would solve their problems.

8 Likes
141

（Maintainer of PDM here)
I’ve actually been following this discussion for a long time, but I’ve been hesitant to respond from what position and how to organize my language.

An interesting fact is, among the tools mentioned in the discussion, PDM and Poetry are the only ones that are not under the umbrella of PyPA, whose maintainers don’t show up here, and they are also the tools that offer the most functionality, which should be a good start of a unified package manager. Admittedly, Poetry’s package metadata format does not adhere to PEP 621(they will, hopefully), but PDM might be among the first few, if not one, that supports PEP 621, shortly after it is accepted. Although PDM began its life as the only package manager to support PEP 582, it is a bit frustrating that people continue to resist adopting it because of the incomplete draft PEP, even though PDM has switched to venv by default after 2.0. In my opinion, PyPA seems to favor single-purpose tools that do one thing well, rather than all-in-one tools. The most adopted package managers(or build backends) inside PyPA are flit and hatch.

So hereby let me promote PDM a little: it provides all the mentioned features here and has its CLI directly inspired by npm. I am also rewriting the build backend to provide a similar extensible interface to hatchling. However, PDM itself doesn’t adhere to any specific build backend and users can choose whatever they like. The next step is to add workspace support similar to how Cargo works.

14 Likes
142

I got the impression one missing part is this:

143

This prompted me to think. I don’t want to focus on the Hatch vs PDM vs Poetry[1] debate here, but let’s suppose for a moment that someone waves a magic wand and we get consensus on one tool. What would we then actually do to make that consensus effective?

I don’t actually have a good answer to this. But it feels like the user community expects the PyPA to have some sort of influence (or authority) over this, and our biggest issue is basically that we don’t…

  1. vs any other tool with a similar scope that I’ve missed… ↩︎

2 Likes
144

Well, the sky’s the limit really, but off the top of my head:

  • enforcing metadata consistency / accuracy
  • using that metadata to do useful things in the python ecosystem
    • for example: regularly run CPython main against the ecosystem (compare Rust’s crater runs, or scala’s community build), identify regressions early, make them release blockers
  • solving the “I have to run arbitrary untrusted code” to install a package problem
  • preventing very confusing splits like venv vs. virtualenv
  • remove major installation footguns with the same urgency as other big UX problems (like pip not having a resolver for years and/or how easy it is to mess up your base python install)
  • use “happy path” defaults that solve the 90% case, leave expert mode to explicit opt-in
  • bring PyPA and SC much closer together
  • etc. etc. etc.

In the absence of all that, you get small groups of volunteers trying their hardest for their use cases / user base / niche, but anyone who’s exposed to several use cases / user bases / niches will run into the chasms between the various insular tools.

Some interoperability bridges have been built over time, but frankly, I don’t find it fair to put the responsibility for “Step 0” of installing/using/distributing something written in Python on a free-floating set of volunteers and hope they coalesce on a common strategy.

Yes, all of this is hard (especially since Python inherits a large part of the “lack of language integration” problem from C/C++ for code it wraps), and it’s not as sexy as a shiny new library or performance improvements, and everyone’s busy, and almost everyone’s working on this in their free time, etc. I’m aware there’s no panacea, but fixing these things should start with a commitment / plan / direction on the language level, because that’s the only place where common goals can really be set.

PS. For a particularly egregious illustration of the combined effects of bad metadata, arbitrary code execution & lack of tooling consistency (i.e. compiler flags), see this blog post.

PPS. Just saw @pf_moore’s comment

Addressing this is exactly what I meant by the above.

2 Likes
145

I should qualify my answers in the light of this. As an end user, my answer would be “YES” to everything (with the usual proviso that I’d be unhappy if I didn’t personally like the chosen unification :wink:)

My answers were very much from the perspective of a packaging specialist knowing the trade-offs. And maybe that’s actually the wrong way of looking at this[1]? I feel as though we’re resisting what is in fact a very clear message from the users, and the reason is that we simply don’t have a good anwer to @smm’s follow up question “how do we go about doing it”, so we’re chipping at the edges of what we feel we can manage - which from an end user perspective is disappointingly little.

  1. Actually there’s no question - it is the wrong way. ↩︎

4 Likes
146

Methodology aside, that is an interesting blogpost, thanks! I don’t agree with many details (including some rather opinionated conclusions), but I think over all it’d be fair to say that it makes the same point as the survey respondents (“way too many tools!” / “unify or bust!”), quite forcefully.

It also discusses this very thread, plus there’s pretty active discussions about the article on Hacker News and Reddit, but – perhaps unsurprisingly – the tone there is less polite than on DPO, so maybe don’t click if this thread is already exhausting to you.

Still, if there’s something to be gleaned from the usual internet chaos, it provides a window into just how unhappy people are with python packaging. Which is not helpful for most people in this thread who poured their free time into making things better, but my take-away from reading those comments is that it underscores the point that average users[1] overwhelmingly prefer homogeneity (even if enforced) compared to the the tooling diversity / freedom / innovation that has been the MO so far.

Here’s one of the more cogent examples:

  1. as opposed to a more maintainer-heavy audience here ↩︎

6 Likes
147

I notice that that table lists “packaging C extensions” as being the missing (or partially missing) functionality. I’m not entirely sure what that means but no tool can “unify” the others if it does not support a major packaging use case because there will always be the need for some separate tool that does handle that case. Perhaps that could look more like pdm or poetry (or hatch) plus some backend extension or something though so at least a single tool could be used for all the other tasks.

There should be some consideration though of what people who are working with C extensions are supposed to do. It’s not just a case of maintainers packaging things for PyPI/conda but also the bigger group of people who would install from source.

148

I’ve read this whole thread, and it seems no conclusion/action item was generated in 2 weeks/146 replies. I think this is indicative of the problem. And there are other previous mega-threads.

I think Python is the only major language where environment activation is widespread (while not truly obligatory). Maybe the discussion should start from the basics and establish a minimal set of hard-requirements for the “common solution asked by the users”. One big question is should activation be used or not? A yes/no should be derived, maybe with SC involvement. Otherwise the discussion goes in circles.

2 Likes
149

Can you describe what you mean by “environment activation?” Also, to
be clear, if you’re talking about sourcing “activate” from a venv,
you’re aware that’s optional right? (Or at least it is on the
platforms I use, being Linux distributions and Unix derivatives.)
You can just call entrypoints directly inside a venv without any
prior “activation” step, it’s my primary way of running things with
Python. I honestly can’t remember the last time I sourced an
activate stub from any of the many venvs on my systems.

150

Personally I would be happy to just be told what the new tool is and have clear documentation for how to use it. If PyPA said “we now recommend tool X” and there was clear documentation for using it and migrating to it then I would be happy to go along with that. I think that’s where most users are at.

The discussion here shows that it’s unlikely that a tool could cover all use cases but I expect that any of hatch, pdm or poetry[1] would be sufficient for the bulk of what most “users” would want to do. If these can replace a whole bunch of different things like pip, setuptools, venv, etc with a single tool and a single frontend then that would be a significant improvement.

Where this potentially goes wrong for me though is if I start trying to use the tool and it turns out that there are significant problems with it and it actually does not do the things that I need and the big one here is building native code extensions. This would not be a complete showstopper for me though if:

  1. The tool does do some of what I need i.e. it can at least be a useful part of my workflow in replacement for some combination of tools that I currently use.
  2. There are ways to use whatever else I need to use (e.g. to build C extensions) in combination with the tool so I can at least use it right now even if it’s still a bit awkward for some things.
  3. There are plans to eventually have better support for important use cases like building C extensions even if that part is actually not completely unified for now.

  1. I haven’t used any of these and they all sound similar to me. ↩︎

9 Likes
151

If your entrypoint needs another entrypoint or needs any other modified environment variables, then you must activate the environment. This is more common than you may think. One use case is calling the command line cythonize from a entrypoint script. You can work around it by very carefully modify env when calling subprocess.Popen to control which cythonize is being used, but more likely you will call some other cythonize than the one inside your venv.

1 Like
152

I’m not sure what you’re referring to but… to the best of my knowledge, this isn’t a CPython concern. We do need to enforce elsewhere in the toolchain.

Outside of the scale issue with trying to run/test CPython against the ecosystem[1], this is work that redistributors like Red Hat do already, as well as maintainers for various projects (eg: Cython) – and issues are treated as release blockers as and when appropriate. I don’t see how adding more work on the volunteers who maintain CPython is going to meaningfully change things.

This isn’t a CPython concern, it’s an installer one: https://github.com/pypa/pip/issues/9140

This goes to, somewhat, the core of what we’re supposed to be discussing here: unifying tools/terminology etc.

This would be nice – if CPython decides to go down the route of trying to have a single “official” cross-platform installation + invocation experience, that’ll make some of our problems less bad… but it won’t fix many of the issues we see.

That’s unrelated to CPython and, honestly, I’d prefer to have this be the case and consistently document this.

Would be nice, but also, not going to go into this to avoid derailing the discussion. x-ref Adopting the concept of "Teams" (from PEP 8015)

On CPython + PyPA relationship, I noted down my understanding over the weekend. I suggest that this conversation (about CPython / PyPA relationship) get split into its own discussion.

  1. And the various challenges like not having a consistent test runner experience. ↩︎

2 Likes
153

(NixOS developer, maintain some Python packages in nixpkgs and has knowledge of the issues we encounter classically with the Python packaging ecosystem wrt to native dependencies. I don’t represent the NixOS project or community.)

I read the whole thread and found the remarks of @steve.dower very interesting. Firstly, I have to say we know that Nix ecosystem is probably a small subset of Python users and probably the one for which Python packaging works more or less the best in our closed garden. We have native cross-compilation support to a certain extent, and I can even trivially run stuff on RISC-V without thinking too much about what needs to be done.

The concept of a key infrastructure and letting distributions & system integrators provide packages for users is compelling for us, as this is 99 % what you need to do on NixOS. (of course, we have escape hatches, but this is a very much desired state in my perspective).

Currently, we have around ~5K7 Python packages (not all of them are working) and we do not support all versions. It’s already a lot of efforts to manage this set of packages, notably because of those native dependencies and various intricacies caused by pleasant features which does not work that well in our model (setuptools-scm for example.)

Sometimes, we have posts such as https ://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741 which shows the difficulty to do this at scale on particularly challenging pieces of software, e.g. TensorFlow.

We have many attempts to bottle Python packages automatically into our Nix ecosystem, I won’t do a list of them, one of them which works really well is poetry2nix (which has support for multiple build backends in fact: https ://github.com/nix-community/poetry2nix/blob/master/overrides/build-systems.json).

Unfortunately, this labor of love (IMHO) requires careful overrides to represent many of the lacking information in the packaging metadata: poetry2nix/default.nix at master · nix-community/poetry2nix · GitHub ; cryptography being one of the most difficult case : https ://github.com/nix-community/poetry2nix/blob/master/overrides/default.nix#L373-L421 where we need to manually pin the vendor hashes of the Cargo dependencies for each release.

So, even if, pip, let’s say would use our Nix expressions to download our binaries or kick in some “from source” compilation using the right environment, it would still force some level of efforts on the maintainer side and may not provide what a user expects, i.e. all versions of a PyPI package.

Given the current discussion, I am not certain what will be done to address such things and whether current policies will impose everyone to find solutions downstream.
More specifically, multiple people talked about more standardization, for example, I wonder if Python considered integrating CUDF formats into his core and if there are reasons to not adopt this.
Building on SBOM standardization process, if there was a way to specify native dependencies using a more or less unique identifiers, I think, it would help drastically the ecosystem.

Also, I have to add that Nix community is probably very interested into all of this and is interested into helping how they can and collaborate on this, we probably don’t have anything for Windows (except through WSL2 I suppose or if someone finishes a Nix for Windows at some point) though.

BTW, I see that other language ecosystem were mentioned, I have to say that I’m not exactly sure that Node.js or Java is somewhat better at all, as distribution maintainers, we sometimes have worse problem than Python ecosystem which provided stuff like site and all to enable our use cases.

Thank you for your time and the interesting exchanges here.

[Had to split the links because of Discourse hostility to new users…]

2 Likes
154

How would the PyPA say that? That’s a serious question - I genuinely don’t know how the average user distinguishes between a formal recommendation from the PyPA and a bunch of random documentation they found on the internet. Is packaging.python.org genuinely that influential?

I say that very much aware that if you take packaging.python.org as definitive, the PyPA recommendation is already to use hatch. It’s a pretty mild recommendation, because there was a lot of agonising when we did the rewrite about not supporting people who chose other tools, or repeating the fiasco of pipenv, but it’s definitely there. And yet the survey results suggest that the vast majority of users aren’t following that recommendation.

As regards “clear documentation”, we can write documents on packaging.python.org, but ultimately a lot of it is going to be down to the tool maintainer(s), not the PyPA.

Yeah. So setuptools, then. People are working on alternatives (although they are mostly working on the hard problems in that area, not so much on simpler “I wrote an accelerator in C” use cases) but right now, native code means setuptools, and unpleasant problems like fighting with gcc vs MSVC on Windows.

We simply don’t have a better answer for native code extensions yet. So does that mean we offer no answer to the user complaints about complexity, or do we offer an answer now, with the qualification that it doesn’t cover native code very well, so for that you need to fall back on other, well-supported and stable but not “unified” solutions?

Again, that’s a genuine question that I don’t know the answer to. The survey didn’t (as far as I know) break respondents down by whether they care about native code, so we have no information there. Nor did it distinguish between people who are happy to just consume prebuilt binaries, and people who want (need) to build from source.

155

I suspect for most users, “preinstalled with the python.org installer and on PATH by default” is what it would take. (And yes, I know that’s not a PyPA thing. But users don’t, and that’s the point.)

8 Likes