Example of monorepo with pypi publishing

Packaging
1

I am wondering if anyone can post a link to a sample python monorepo that is publishing to pypa multiple packages, preferably PEP-621 compliant.

I tried to find an example for very long time and I failed to find one, so I suppose that nobody did it so far.

PS. Please don’t ask why or other “you are holding it wrong” answers.

1 Like
2

I don’t have any public repos I can share, but I just have one directory per package. Each directory has a pyproject.toml. There’s some shared tools in another directory. I don’t use labels to track individual package version (it’s not a requirement for what I’m doing).

Does that answer your question, or are you looking for something else?

1 Like
3

Thanks for the quick reply, I guess that feedback should be fine w/o example source. It might prove useful to mention challenges you faced and how you addressed them, like CI/CD testing, partial publishing, changelog management.

I was expecting the need for an extra deep layout in order to make it possible to have a pyproject.toml file for each library (subproject). That means some repetition across repo but I think that with the current dynamic field options, version could be retried from a single place (assuming all repos wout share the same version, as tagging would be a nightmare otherwise).

I want to squeeze ~20 very small libraries which are basically plugins in a single repo but I am not sure yet how to deal with release pipeline. Should I just release all packages at once, even if in fact we updated only one of them? Is there a way to skip publishing a package if its content was not changed since last release? – i was considering looking at the archive but if I use dynamic version, I would at least have one version file that is generated by setuptools-scm, so the archive would change only due to this.

4

Actually I keep version numbers in Makefiles (one in each directory), and from there I generate pyproject.toml, from a template file in the same directory. I do this because I need access to the version number from non-python code. Basically others steps in the Makefile need the version number, too. Each library has its own version.

I don’t have any CI/CD hooked up in the repo. The Makefile builds the wheel file(s), copies them elsewhere, and that’s where testing happens. Kicking off make is a manual step.

Other than the occasional “make all” at the top level, I just build the wheels manually as files are edited. I don’t try to skip builds if nothing changes. I’ve got about 25 libraries in this repo.

As I said, I don’t do any git tagging. If need be, I can track down versions based on Makefile commits. But in 20 years I’ve never needed to.

5

I am wondering if anyone can post a link to a sample python monorepo
that is publishing to pypa multiple packages, preferably PEP-621
compliant.

I publish from a monorepo: GitHub - cameron-simpson/css: my personal kit, containing thousands of scripts and hundreds of Python modules and assorted other code

I tried to find an example for very long time and I failed to find one,
so I suppose that nobody did it so far.

Oh, I’m sure quite a few people do.

I generate the pyproject.toml file and the README.md etc using an
elaborate script. The package metadata live in the DISTINFO top level
name in the package/module, eg:

I keep the release versions in this file:

but anything convenient will do.

Cheers,
Cameron Simpson cs@cskk.id.au