Python Packaging Strategy Discussion - Part 2

Packaging
42

In my view, this (especially the part in parentheses is critical). For many apps and libraries (in some sense for all of them), the Python version is part of the dependency spec. I do not see how a pip-like solution — that is, one that lives inside a particular Python install rather vice versa — can ever adequately handle this fact.

Maybe these threads should really be called “Packing and environment strategy” :wink: . I don’t think we can think of packaging separately from environment management. And I also don’t think any tool(s) can provide a full-featured enviropackaging solution without being manager-first. The manager needs to be the top-level tool and should manage Python, rather than the other way around.

1 Like
43

You’ve described conda :slight_smile:

44

Well, yes. :slight_smile: I mean, conda is a bundle of different things, and that’s one feature. But I think in terms of concrete desiderata for improving Python packaging it’s a big one to consider.

45

I recently stumbled over packaging issues again since I wanted to migrate my command line applications from setup.py to the new setup.cfg.

I had a small discussion with @sinoroc because setup.cfg does not allow an external file as a dependency for install_requires. While this makes sense for a library which is used by other applications it doesn’t make sense for an complete application to ship without pinned dependencies.
So imho there are two fundamental different use cases:

  • distributing a library
  • distributing an application

and both have contradictory requirements.
For an application I go through great lengths (CI, local tests, etc.) to ensure that everything works as expected and as an application user I expect that the application installs with the correct and tested dependencies. Of course as a user I should be able to update the dependencies manually but the first installation should always be a working (and tested) one.

Packaging an application

As an application developer I’m interested in defining my used dependencies only once and currently I use a requirements_setup.txt. The file content is is used for CI, local testing and of course currently in the setup.py in install_requires.
I have an additional file requirements.txt for people who want to contribute which adds some testing dependencies and references -r requirements.setup.txt.
That way I have a single source of truth for the both the dependencies needed for a successful installation and for developing and running the tests.

Since I can’t use the the file my current setup won’t work with the setup.cfg.
Not pinning the dependency versions is also not an option because countless times dependency changes have broken the application.
So with the current state of tooling the only option would be to manually copy the dependencies back and forth between the requirements_setup.txt and setup.cfg which is impractical and error prone.

Installation of the app

Since my target is small computers or linux machines like raspberry pis it’s okay to assume that a more or less recent version of python is already installed.
Still I have to document the installation:

  • create folder for application
  • create venv in that folder (in the subfolder venv)
  • activate the venv
  • pip install my_application
  • run it with my_application (via entrypoint)

I’ve discovered in this thread pipx which seams nice but then the installation flow would be

  • create folder for application
  • pip install pipx (because it’s nowhere available)
  • pip install my_application
  • run my_application

Which is not much better.
Also for me it’s unclear how I can install the same application twice in different folders (e.g. a dev and stable version) Also currently with pip I can provide users with git links for tests/hotfixes and they can just install it in another folder and test which is very nice and convenient.

Improvement

I’d really love to have some standard tooling for installing/publishing libraries and installing/publishing applications.
There are wonderful tools already available but still everything is hard because

  • I don’t know all the good tools
  • The tools still are missing on the users computer and have to be installed beforehand

Python claims to have “Batteries included” so why not include an easy and convenient way to install applications. Including simething like pipx that adds a way to install an application locally in a folder (with a venv) would be a good start and already simplify things.

For developers differentiate between libraries and applications and have appropriate tooling.

Also patronizing developers by not allowing application owners to specify a file as dependency will just lead to frustration. People will hate the manual effort and do it anyway.

1 Like
46

This is possible in setup.cfg as well as in pyproject.toml since setuptools v62.6.0 released 2022-06-19. And, for context, the discussion is here.

47

This is very much the intention. Most of the existing tooling that comes under the umprella of “packaging” is focused on library development. We are aware of the gap, but unfortunately no-one seems motivated to do much about it (the existing solutions, like entry point wrappers and shipping applications as pip-installable projects that include executables, are “good enough” for the majority of developers, it seems).

Personally, I would love to see a dedicated set of documentation, workflows and tooling for the “deliver an application” case. Maybe there could be a lot of overlap, maybe it would be very distinct - I honestly don’t know at this point. But it won’t happen unless volunteers step forward with an interest in making it happen, and the time to do so.

Sorry the situation isn’t more helpful for you…

48

Hatchling supports that with a plugin GitHub - repo-helper/hatch-requirements-txt: Hatchling plugin to read project dependencies from requirements.txt

49

Tbh. the entry wrapper is (from a developer perspective) actually good enough.

I think from a user perspective it would already be helpful to include a small thin wrapper around venv & pip to install an application into a folder. If this could also add the executables for the entrypoints into the folder root it would already be a tremendous improvement because it would make a 6 step installation a one-liner.

How about merging it with pip so it’s still clear the application comes from pypi:
pip install_app my_application_name

No worries - everybody is contributing in their spare time and you’re not personally responsible for this situation :wink:

1 Like
50

I guess that is pipx (which you already mentioned), isn’t it?

1 Like
51

As far as I understand pipx installs to the user directory and adds the entry point to the shell.
This would not work if I want to install the application two times in different folders (e.g. a dev and a prod version).
I mainly create applications for home automation (e.g. for reading smart meters or smart home rule engine) which normally run as a systemd service under a different user. So I am not sure how that will work when the application is installed to another user folder.

Also since it’s not distributed with the standard installation and thus (more or less) useless.
If I have to write a guide on how to install pipx I can also write a guide on how to create and set up the venv.
The user already has to do multiple steps and I have to document multiple steps then I might as well do it “properly”. :man_shrugging:

2 Likes
52

I do not use pipx a lot myself, but if I understood correctly a common pattern is to use the --suffix to install multiple versions of same application:

pipx install httpie
pipx install --suffix @dev 'httpie @ git+https://github.com/httpie/httpie.git'

then I can use httpie and httpie@dev. Maybe that helps for your use cases.

Indeed, it would be a better UX to have a pipx-like tool out of the box (bonus points if this tool can install the Python interpreters).

1 Like
53

That would definitely be sufficient. Thank you for your input!
If you know also know how to install as a user who doesn’t have a home directory then this might actually be the tool we’re looking for.

Yes - that would be nice but I’d take any improvement over the current status quo.
Most distros switched to python3.10 anyway so there’s propably little to gain.

54

The environment variable PIPX_HOME lets you change where pipx creates its environments. And PIPX_BIN_DIR for the executables. The command pipx --help documents this.

55

If the tool is going to install the Python interpreters, then it comes in a different box. The suggestion will have to go to your operating system vendor if you want it in that box.

The only “out of the box” experience we control here is when you’ve already chosen a version of Python and installed that.

(Now, maybe we should exert more control over other “boxes”? Or create a new one that we can control? That’s certainly a potential strategy, but it’s unlikely we’d get something to be as widely available as Python currently is, given it’s so normal these days to install something. I’m throwing out scattered thoughts here, rather than any kind of plan or proposal.)

1 Like
56

Same. That is why the part of my message you quoted is poorly formulated, my bad. Anyway, the focus of this thread is on “better support”, so I will keep my further thoughts on this “magical tool” for another thread :smiley:

1 Like
57

This is pretty much the same use case I’ve faced before and ended up writing installer scripts which end up doing something very similar to pipx. In my case the additional step of installing pipx wasn’t a total showstopper but I really needed to keep the app contained in one directory (particularly on Windows) and I also needed to install user-editable config files alongside the executable (an entry point, like pipx). With the benefit of hindsight I could maybe make this work using pipx now, but at the time it just didn’t seem to be quite enough and there wasn’t much help online for this scenario.

Of course, any approach using virtual envs has one potential issue which is that it relies on the continued existence of the original python used to install it. On Linux this generally isn’t a big deal as you get a system python and it’s unlikely to go anywhere. However on Windows someone might decide to update from 3.11 to 3.12 and delete the old version. Suddenly all venvs are broken in a non obvious way. Admittedly I’m not sure this is within the scope of a packaging tool to solve (in fact I’m pretty sure it isn’t). It’s more an argument for pyinstaller, but that brings plenty of issues of its own.

58

So more on the topic of “support”…

Somewhere, some time ago when the “Packaging Python projects” page of the Python packaging guide was created (or updated), the topic came up that what was missing was a way to compare and choose a build back-end. Much later, actually right after Chris Warrick’s article which contains some comparison tables, I gave a try at writing comparison tables of my own for the Python packaging tools. The pull request is here and the preview is here, I consider it as just a starting point (there is a lot missing, most notably conda) and I am waiting for feedback (even if it is to say that it is not helpful) or someone to take over.

I also recall some discussion about adopting the Diátaxis writing framework for the main Python documentation. Is it something that is being considered for the packaging guide?

I think a packaging newsletter as suggested the other day would be helpful as a support tool. Just to say things like “the [project] spec is ready and already implemented in [these packaging tools: […]; and also by Dependabot on GitHub” so that people like me can relay the info on StackOverflow or wherever.

1 Like
59

Indeed… the choice of “throw it away and make a new one, don’t try to
upgrade it” is fine for your own development (even though once a year I
still stab myself on something by not remembering to first make a freeze
as the spring Fedora rolls out a new version) but makes it IMO
suboptimal for application delivery.

60

FWIW, while there’s certainly a lot of areas the PyPUG needs improving in, and it would certainly be helpful to revise and update the individual pages to better serve their indicated reader needs using the Diataxis guidelines for each content type/reader need as a guide, it already does follow the basic Diataxis layout quite closely—aside from a handful of top-level single pages, the content is organized into the four Diataxis categories of Tutorials, Guides, Reference (Specifications) and Explanations (Discussions).

1 Like
61

Thank you to all who have participated. It is very impressive the amount of thought and work that has gone in to improving the experience for the different users of Python. My background: a long-time Python extension module author and creator. SciPy and NumPy are what I’m most known for. Lately I spend more time in business conversations than technical conversations, but my love is still technology. I have learned a lot over the years by experiencing and seeing many, many people struggle with getting Python installed and managed. I’m not always able to articulate my views, and I’m always trying to learn from the experience of others.

I was involved in promoting and building conda at a time when there was nothing else that would solve the problem of installing scikit-learn, for example on Windows, Mac-OS, and Linux. The solution we came up with was very-much a “user-facing” solution. It had a little-bit to offer to the developer (in the form of build tools) which conda-forge has really done a great job of improving.

In my view the most important things to keep in mind are:

  • There are different users of packaging and most users of Python should really be using a distribution that is suitable to their “consumer-driven” need
  • So many Python “users” are not hard-core developers – this is critical. These “occasional developers” are the ones most hurt by over-rotation on tool promotion by the PyPA because they get confused as to where they should be looking for help.
  • In a related fashion – the PyPA should be careful about over-promoting a particular distribution of Python. In particular, the “emergent” distribution of python.org binary download plus binaries on “PYPI” is not appropriate for all users. It works for many to get started, but it won’t and can’t serve all needs.
  • The PyPA would be wise to focus on standards and interoperability and developer or “integrator” work-flows. It is important to recognize that it will be difficult to help all people do all things (and the more “end-user” the person is the harder it is to serve them. It can be a source of education, summarization, and standardization.

I know others have expressed these ideas in different ways, but this is my $0.02. I’m excited by all the incredible growth of Python users and all the incredible tools that are being built. I would just love to see that the knowledge gained by past investment into tools is fully understood (which is why I am grateful for the knowledge shared by so many of you).

9 Likes