# Dependency management with many private / local packages

**URL:** <https://discuss.python.org/t/dependency-management-with-many-private-local-packages/66213>\
**Category:** Packaging\
**Created:** [September 30, 2024, 3:20pm UTC](https://discuss.python.org/t/dependency-management-with-many-private-local-packages/66213 "2024-09-30T15:20:06Z")\
**Posts on this page:** 1\
**Showing post:** 7

<div class="post-metadata">

**Author:** ![hauntsaninja](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hauntsaninja/32/12451_2.png) [@hauntsaninja](https://discuss.python.org/u/hauntsaninja)\
**Post date:** [October 1, 2024, 12:02am UTC](https://discuss.python.org/t/dependency-management-with-many-private-local-packages/66213/7 "2024-10-01T00:02:17Z")

</div>

I work in a research setup and we faced a similar problem.

I ended up just building my own lightweight package manager (called `oaipkg`), because editable installs are great and can be instant. It’s scaled not terribly, even as my company 20x-ed in headcount, 70x-ed in number of projects, and requirements have changed.

The rough description is:

- Every project is defined by a pyproject.toml (inside a directory of a matching name)
- A pyproject.toml can define a `tool.oaipkg` section, which has a `monorepo-dependencies` key
- oaipkg discovers projects by directory walking (with some optimisations to make this fast)
- To install a target, oaipkg walks the first party graph to collect first and third party dependencies. It shells out to `uv` or `pip` for third party dependencies and it implements a fast editable install itself with no build backend in sight:
  - Make an `.egg-info`, `.egg-link`, write to `easy-install.pth`, install entrypoints
  - You could also make a `.dist-info` and separate `.pth` file, this is more modern, but interpreter startup is slower

There are some twists for extension modules (we build wheels and key them by a tree hash), bootstrap, first party projects in other repos, locking, testing, various interesting integrations. But if you’re predominantly pure Python, I think it’s a good setup. Installs are really fast (e.g. if you don’t need to install a wheel, it’s sub-second).

---

_[View the full topic](https://discuss.python.org/t/dependency-management-with-many-private-local-packages/66213)._