Help packaging optional application features, using extras?

[LewisGaul]

I appreciate the responses, it’s all pretty much as I expected but there’s some useful points that have been made.

My current thoughts can be summarised in response to following:

@CAM-Gerlach

That’s right; extras are really just (more or less) groups of optional dependencies.

@CAM-Gerlach

@LewisGaul

I’m left feeling this use-case isn’t really properly supported in python packaging and I might need to roll my own solution.

As I understand your needs, it is more or less supported, it just requires some restructuring of your project, as many others do.

I understand your perspective, but I think what I was really hoping for and referring to when I said “not properly supported” is having finer-grained control over what gets packaged/installed from within a single project.

This was actually the reason I originally set things up as namespace packages - my intention was for all code to be installed under the ‘minegauler’ namespace, but for the packages within that namespace to have clear separation such that they could be installed separately (with ‘app’ being the core/default, and other subpackages being ‘extras’).

My hope was that I could control subsets of the code, data and entrypoints that would be installed. I originally came up with two possible approaches: using extras, or using separate package names. Unfortunately it seems this is not supported by extras (which only control which dependencies are installed), and to have separate package names I’d need multiple pyproject.toml or setup.py files (either in some kind of non-standard monorepo setup or across separate repos).

Therefore I still feel the level of control I’m looking for isn’t properly supported within a single project. Does that sound fair?

---

Point-replies below…

@sinoroc

If a user installs minegauler without the bot extra, and then calls minegauler-bot , I would output:

Please install 'minegauler[bot]' to use this command.

That’s a reasonable suggestion, although the error message would only make sense in a pip environment (i.e. not in a PyInstaller package), but that’s probably not a concern in practice. I guess what I was really hoping for was a clean way to exclude all of the code, data and the entrypoint related to the bot.

I did have another idea: create a skeleton python project named ‘minegauler-bot’ that has a hard dependency on the main ‘minegauler’ project, but the entrypoint is defined in the ‘minegauler-bot’ package. Then minegauler[bot] would simply depend on minegauler-bot as an extras_require.

@CAM-Gerlach

I’m not entirely clear if you mean "moving away from having a setup.py " in terms of making your existing config declarative, or switching to a build backend other than Setuptools.

I mean moving to pyproject.toml instead of setup.py, although I’m expecting to continue using setuptools unless there’s something that seems a better fit.

I appreciate the input on my current packaging setup, I actually did some testing of the new setuptools support for PEP 621 recently (see Help testing experimental features in setuptools and my branch).

@CAM-Gerlach

It looks like you’re using PEP 420 namespace packages? They aren’t that commonly used these days, can make things much more complex, and it isn’t clear to me why they’re needed within the same project.

I was briefly, but added an __init__.py in when I started hitting problems… What you see in my setup.py is actually an effort to only package some of the subpackages, but maybe I should just package it all and support the other subpackages via other ‘dev extras’. This comes back to my desire for extras to also control the project code that’s packaged as well as managing dependencies…

@CAM-Gerlach

Right now, extras doesn’t currently offer the ability to remove deps from the default, as it gets rather complex if you specify multiple extras, some of which add deps and some of which remove them, but there have been plenty of proposals and discussion on potentially adding a “default” extra that would be a minimal set of deps for the package, upon which the other extras could build: Adding a default extra_require environment

Yeah, I had read this discussion and realised it would serve as a possible solution to what I was describing, sorry I probably should have dug it out and linked it.

@CAM-Gerlach

Consider the PyPA-recommended src layout, which combined with the previous should help avoid some of the current complexities in your find_packages() config, as well as other issues.

Yeah, I have read that and decided against the src layout - personally I prefer to avoid using editable installs, which I believe become somewhat required (at least unless all imports are relative)?.

@CAM-Gerlach

@LewisGaul

What’s the expected case where this is useful to the user performing a ‘pip install’?

Often you’ll see test, lint and other development-related dependencies (basically the contents of your requirements-dev.txt file) consolidated into a dev extra (which is easier now that at least with current pip, extras can depend on other extras, reducing duplication). The intention is to allow developers/contributiors, and potentially repackagers, who are e.g. installing from a GitHub clone, Git tarball, sdist, etc. to more easily install and run the tests without having to manually install (and maintain a duplicate list of) dependencies in requirements-dev.txt , which isn’t always shipped or easily available as a file.

This resonates with me, since this project does attempt to maintain a requirements-dev.txt and have it be in sync with requirements.txt. I might consider switching to using extras for specifying these deps, although my current dev workflow doesn’t involve pip installing my own project (I’m assuming people who use a ‘test’ or ‘dev’ extras in development would use editable installs?).

@CAM-Gerlach

@LewisGaul

packages and package_data include everything needed for all extras - I haven’t found a way to conditionally include based on requested extras

Yes; if you have hefty package data that you want to separate out along with them, you’ll want to move that to a separate package.

I guess this is the main kind of thing I was hoping there was a better answer to

@CAM-Gerlach

@LewisGaul

The problem with this is that in my case it’s my implementation of the ‘extra’ that depends on the main application code, not the other way around…

I’m a little confused by this, sorry. In general, your implementation of the extra is going to depend on the main package [snip]. So I’m not sure how your case differs.

My case is the standard case, apologies for the confusing wording My point was that it wouldn’t be possible to simply move out the code for minegauler[bot] into a new minegauler-bot package while keeping the minegauler[bot] extra, because the two projects would then depend on each other.

@CAM-Gerlach

@LewisGaul

In theory I could refactor out into package dependencies such as minegauler -> minegauler-core and minegauler[bot] -> minegauler-bot -> minegauler-core

Yeah, this would be a good way to do it, as I understand. Keeping your package more modular is generally recommended, as it decreases coupling, aids reusability, replaceability and pluggability, and allows downstream users to be more specific about their own dependencies, but it does increase overhead to some degree.

Sure, I’m totally on board with modularity. However, using separate projects feels like huge overkill in this case - I’m able to maintain pretty good discipline with modularity within the project, and don’t want to have to worry about interdependencies and versioning between what are really just internal parts of the project!

@CAM-Gerlach

@LewisGaul

I’m left feeling this use-case isn’t really properly supported in python packaging and I might need to roll my own solution. Does anyone have any better suggestions?

I’d generally advise trying to advise against rolling your own tooling unless you’ve exhausted other options (which there appear to be a several), the benefits are clear (in this case, they don’t appear all that decisive), and you’re willing to accept the long-term maintenance risks and costs of doing so.

I certainly don’t plan to go overboard with any kind of roll-my-own solution - I was thinking more along the lines of having multiple pyproject.toml files (perhaps generated using a script) to allow me to push different parts of the codebase as different packages from within what’s otherwise a normal single-project layout. I’d definitely prefer to avoid any kind of hacky solution like this though, hence asking for input!