Best way to install bundle of applications from multiple packages

TL;DR Is there a nice way to install a bundle of python applications spread across multiple packages into one venv, and create a bin dir containing only executables from those projects?

At work, I’m in charge of making a number of meta-packages available to users. That is to say, I want to make a bunch of apps available in one place. The packages containing these apps are designed to work together. It is therefore handy to have the installed in a single virtual environment, for those who want to use them as a library. However, I would like a clean bin directory containing only these apps (and none from their dependencies).

In case that isn’t clear, say I have pkg1 which provides apps foo and bar and pkg2 which provides baz, I want to end up with a single dir bin containing foo, bar, and baz

I know that if I wanted to do this for a single package,pipx would definitely work. It’s not clear to me though whether the inject functionality will do what I want for a bundle of packages.

My current workaround for one of these projects relies on the fact that all the apps in the packages have the same prefix (to make tab completion easy and avoid name collisions). So I create a virtual env (.venv), install all the packages into it and then create a bin dir, find all binaries with that start with the prefix in .venv/bin and create symlinks to them in bin. Then bin can safely be added to the users path (we use environment modules to do this). I also created a script that you can execute to activate the venv for easy interactive use. But it doesn’t completely solve the library use case. (The ideal here would be something that could create new venv using this venv as a base, this probably exists, but I can’t find it.)

On thing to keep in mind is these packages don’t exist on an internal PyPI anywhere (and it’s non-trivial to make that happen because of access control issues).

In any case, I’d be happy to here any suggestions on improving this workflow. Since my current thought is I’ll need to write custom code wrapping pipx to do the required finagling when apps don’t share a nice prefix.

It sounds a bit like you’re building a Python distribution, rather than an environment?

I’m not aware of any existing tools that will do this kind of thing for you automatically, but you should be able to set it up manually. There are no real tricks in Python packages - if you copy the site-packages files onto any machine, you can put PYTHONPATH=<that dir> python3 -m <the command> into a shell script wherever you like and it’ll work.

I’d suggest approaching this as creating your own copy-and-extract installer for those packages, rather than trying to share a venv. You’ll likely find it easier to just do things directly.

1 Like

I think my problem may be that I want both a distribution (is that would you’d call conda?) and something that is a bundle of apps, some of which happen to be Python apps. I guess I’m trying to re-package a bunch of stuff into a single meta-package.

I guess another thing I didn’t make clear is that some of these packages depend on each other, and so (since they don’t reside on a PyPI) I can’t see a way to build them other than to consecutively pip install them into some environment. The group of packages is also (explicitly) designed to work together.

I guess this is a perfectly valid use of setting PYTHONPATH, I’m mostly trying to avoid this for users who only want to use this as a library. I think I’m seeing what your saying about copying the dir though, I could pretty easily provide a script that sets up a venv and then copies/symlinks to everything in the site-packages of the env I create.

I may also be overly conservative in avoiding copies of data, but it seems to be a bit of a silly consequence of the venv for every app situation that you end up with 50 copies of setuptools or whatever.

In any case, I’m happy I’m not missing something obvious. Also thanks for the suggestions, I think I was definitely over-engineering things. At the very least, I can do something simpler than what I was planning by just using importlib.metadata

conda is a tool that helps assemble distributions, but a distribution is just a single installable thing[1] that includes everything the user needs. You can put Python itself in here,[2] along with the files of the packages you want to be there. Then copy that entire thing to the machine where you want it and it’ll (usually) work.

pip and venv etc. are convenience tools, but are not essential. Once pip has installed files, those files can usually just be copied to another machine using whatever mechanism you like (except for the generated script launchers, which you’d likely want to manage manually anyway if the “public” part of your bundle is going on PATH). venv cannot be copied, so it is best avoided entirely for this case.

This part complicates things, though if some users are intending to write Python scripts that import modules, they can modify sys.path as part of their Python script if setting PYTHONPATH in a launcher script is inconvenient.

Yes. Again, if you control everything that is going into your bundle/distribution, then you don’t need a venv for each. You may need to do some work to align the dependencies between each tool that you include, but it’s worthwhile to do that. It’ll make things much easier to maintain and much easier on your users.

I know a lot of what I suggest sounds like more work for you, and it is. But to be honest, and as someone who does this work too, if it’s not worth that much effort to get right then you may as well put in no effort and tell your users “good luck”.[3]

Start by pip install the things you need into a single directory (using --target) and then create the shell scripts to launch PYTHONPATH=<target dir> python3 -m <command>. Put those shell scripts on PATH and you have what you are trying to build - no venv or importlib needed.[4] Then ZIP/TAR/etc. the whole thing up, send it to your users, and tell them to extract it and put the directory of scripts on their PATH. Instant distribution.

  1. I’d usually say “package” but I don’t want to imply a PyPI package or a Conda package or a Python package. I literally just mean a bundle of files with an install script. ↩︎

  2. Trivially on Windows, fairly easily on macOS, easy enough on Linux if you know the exact OS it’ll be installed on. ↩︎

  3. This trade-off changes when users start demanding arbitrary libraries. Then it’s okay to say “good luck”, because that becomes a major scale issue. But as long as you are in control of the list of libraries, just be in control of them. ↩︎

  4. On Windows and I believe macOS, you can put Python in that directory as well. Linux you’ll need to know which version will be installed in your OS, or find a portable build from somewhere. ↩︎