I work in a research setup and we faced a similar problem.
I ended up just building my own lightweight package manager (called oaipkg), because editable installs are great and can be instant. It’s scaled not terribly, even as my company 20x-ed in headcount, 70x-ed in number of projects, and requirements have changed.
The rough description is:
- Every project is defined by a pyproject.toml (inside a directory of a matching name)
- A pyproject.toml can define a
tool.oaipkgsection, which has amonorepo-dependencieskey - oaipkg discovers projects by directory walking (with some optimisations to make this fast)
- To install a target, oaipkg walks the first party graph to collect first and third party dependencies. It shells out to
uvorpipfor third party dependencies and it implements a fast editable install itself with no build backend in sight:- Make an
.egg-info,.egg-link, write toeasy-install.pth, install entrypoints - You could also make a
.dist-infoand separate.pthfile, this is more modern, but interpreter startup is slower
- Make an
There are some twists for extension modules (we build wheels and key them by a tree hash), bootstrap, first party projects in other repos, locking, testing, various interesting integrations. But if you’re predominantly pure Python, I think it’s a good setup. Installs are really fast (e.g. if you don’t need to install a wheel, it’s sub-second).