Multiple packages from same src using pyproject.toml

Packaging
1

I have projects using namespace packages that need to be distributed as separate packages but live in the same source tree. I currently use setup.py files to do this and put each in their own directory, e.g.

src/
   acme/    
       core/    
          ...
       client/
          ...
       server/
         ...
   acme-core/
        setup.py
   acme-client/
        setup.py
   acme-server/
        setup.py

Each setup.py simply cds to its parent directory and uses requirements
and package selectors specific to that package. This is a little kludgey but
allows me to work with just one source tree in my IDE while still being able
to generate separate installable packages.

The question is, how do I accomplish the same thing using pyproject.toml files?

As far as I can tell, there is no way for pyproject.toml to specify the root directory.
It is simply assumed to be the same directory as the file or perhaps in a src/ subdir.

Obviously, I could split up my project into multiple source trees, but I would rather not do that if I can avoid it. I would like to maintain the original project structure but just change the project build configuration.

I could resort to a hack where a package specific pyproject.toml file gets moved into place temporarily during builds, but that seems pretty horrible. A better option would probably be to customize a build tool like hatchling to provide a way to alter the root.

Really I wish I could just write something like this in a pyproject.toml file:

[project]
name = acme.core
root = ..
...

and someday have tools recognize that.

Any other ideas?

Please just assume that I have a good reason to want to keep my packages in one source tree.

2

I don’t think the pyproject.toml source format really supports this sort of layout. The section in PEP 517 describing what a “source tree” is pretty much assumes a directory structure rooted in the directory containing pyproject.toml.

You may be able to get something to work, but I think it’s likely to be a bit of a hack however you do it.

Supporting multiple source trees under a single project root sounds like something that could do with being considered as an independent use case, so that the existing standards can then be updated to cleanly take it into account. But obviously, that’s a longer-term approach and doesn’t really help you much right now.

Sorry there’s not a better answer that I can give.

1 Like
3

I think you might be able to do something like this right now with hatch: Build - Forced Inclusion or maybe Build - Rewriting paths.

1 Like
4

Yeah, I think it is pretty obvious that this use case was not considered at all. I get the impression that there is very little use of namespace packages in the open source domain, so this issue probably has not come up that much for many users.

I guess one question is whether the ability specify a relative root using a project.root entry in pyproject.toml is worth proposing as a PEP. Short of that, I suppose I could propose it as a feature for specific backend build tools.

5

Thanks. I will definitely give that a try.

6

I wouldn’t be too quick to jump to a solution here. We should probably take a better look at whether there are other examples of where people view a “project” as containing multiple “source trees” (the monorepo pattern seems like it might be important here as well, for example) and make sure that we support those as well.

Personally, I don’t have any real experience with these sorts of project structure, so there’s not much more I can add beyond this level of “how to frame a discussion on putting a proposal together” advice.

7

I have very similar needs, with a monorepo where I want to apply global settings [1] across multiple namespace packages under the same git repository, but I want to release them as independent distribution packages. monas looks interesting, especially because it comes from @frostming who also wrote PDM, which we use, but I haven’t dug into it much yet.

  1. think linting rules, GitHub actions, unified tested, etc. ↩︎

2 Likes
8

You might also try out https://www.pantsbuild.org/ as well, for Python monorepo support.

9

This use case seems like a great example of the broader context behind Projects that aren't meant to generate a wheel and `pyproject.toml` . Granted, in this case we are talking about building multiple wheels, rather than none. But the idea is the same: challenging the utility of the standard project structure, and distinguishing build information from project configuration.

10

I suppose one possible hacky way around this is to use a src layout and symlink the src dir into alternate pyproject.tml directories.

1 Like
11

This could be a use case for dynamic metadata plugins.

12

Unless there’s a PEP to fix this[1], this can’t be done according to PEP 621. The name field is not allowed under any circumstances to be dynamic. This means using a project field locks to you one pyproject.toml per package. This comes up over and over again for larger packages that are split into smaller ones; ITK is hitting this wanting to switch to pyproject.toml. pybind11 ships two packages from one setup.py. Etc.

Currently the only true way around it is to specify the metadata entirely in a custom format (not [project]), which releases you from the one package per pyproject.toml requirement. But for some backends (like scikit-build-core), we really don’t want to have to have two different ways to enter the same information, so I’ve been holding off on supporting that.

The workaround is to have two pyproject.toml’s in separate directories. You can tell pip to target a subdirectory even when installing from git. Awkward does this with awkward-cpp, hatch/hatchling (and most frontend/backend combos) do this, etc.

  1. The above linked discussion would result in at least one PEP that would allow partially specified dynamic metadata, but wouldn’t, as it stands now, release the name requirement to allow it to be listed as dynamic. This used to have issues with being unable to access things outside the pyproject.toml subdirectory, but builds are no longer copied to new directories, so I think that’s not an issue anymore. ↩︎

1 Like
13

I went ahead and made an example repo using hatch. Each of the subprojects can be built independently.

Here’s the acme-core/pyproject.toml:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "acme.core"
version = "0.0.1"
description = ""

[tool.hatch.build.targets.sdist.force-include]
"../src/acme/core" = "src/acme/core"

[tool.hatch.build.targets.wheel]
packages = ["src/acme"]
1 Like
14

Cool! That might solve my immediate problem and is way nicer than the symlink hack.

15

Monas may still be too immature to rely on (not sure), but I think the structure it advocates is the most standard-compliant way to go. Instead of spreading your packages in the project root, each sub-project is nested inside, each having their own project structure with a pyproject.toml and Python packages to manage. The Hatch demo above doesn’t contain much details, but it seems to also goes a similar direction. As long as this general layout is followed, you should be able to hack together some tooling around populating all sub-projects into one single virtual environment and work on them as a whole.

One thing I’m not sure about though is how editable implementations work with multiple namespace packages. This could vary between build backends, I’m not entirely sure.

16

I think you missed editing the acme-server/pyproject.toml :slight_smile:

I think it’s entirely reasonable to have separate TOML files per project, where each project can build at most one distributable wheel/sdist. The thing that strikes me as awkward about the above workaround with Hatch (I assume other tools can do something similar) is the need to create folders to store separate pyproject.toml files with the same name, and then add directives to ensure the sdist contains the right one.

What I’d like for build tools to be able to do instead, is take a config filename from the command line (defaulting to pyproject.toml) and work with whichever config is specified, potentially building the corresponding project; then, any sdist built would include the named file (and not any other config files), but renamed as pyproject.toml. That way, whoever installs the package gets a setup that can work with the defaults.

3 Likes
17

I had basically the same thoughts last night as @effigies and @kknechtel. One thing I’d prefer over the hatch approach is that there is a top-level pyproject.toml that contains all the global settings, e.g. for linting/testing tools, etc. It would even be nice if you could “stack” sub-distro-package pyproject.toml files on top of the global one, and then only specify the additional settings you need for individual packages, but that’s probably asking too much.

3 Likes
18

At $work, I played around with a very simple idea for building multiple different distribution packages from a single monorepo source. While I can’t share the code, I can explain it pretty easily. It’s just a PoC, and it’s really simple, but maybe it will be a workable solution while we figure out a better way. I built this on tox and pdm.

  • I created a subdirectory called scripts/ [1] and inside there I put multiple pyproject-{envname}.toml files. An example might be scripts/pyproject-build-foo.toml.
  • I added a top-level toxfile.py file and a simple tox plugin that implements a tox_on_install() hook [2].
  • Inside this hook, I dig out the environment name from toxenv.name. I look for a scripts/pyproject-{envname}.toml file that matches.
  • If found, I symlink the top-level pyproject.toml file to this environment-specific file [3].
  • I create a [testenv:build-foo] section that just called pdm build.

Now, all I have to do is run tox -e build-foo and it symlinks in the proper pyproject.toml file and builds the package quite nicely.

One downside is that you have to repeat a bunch of metadata in each pyproject-{envname}.toml file, but oh well. The nice thing is that I can say build a namespace distribution package that contains no code but only dependencies that all get installed when I install the namespace package. And I can build whatever collection of distribution packages from a single repo by just fiddling with pdm.build.excludes and pdm.build.includes and such.

  1. the name is unimportant ↩︎

  2. I originally tried tox_before_run_commands() hook but that does not get executed if you run tox with the -n flag ↩︎

  3. with some caveats, like top-level pyproject.toml must not exist or it must already be a symlink into scripts/, and if that does not exist, it falls back to symlinking to pyproject-default.toml ↩︎

1 Like