Dear Python community, I am very excited to share this tool I have been building with you!
UniDep streamlines Python project dependency management by unifying Conda and Pip packages in a single system.
Handling dependencies in Python projects can be challenging, especially when juggling Python and non-Python packages. This often leads to confusion and inefficiency, as developers juggle between multiple dependency files.
Unified Dependency File: Use either requirements.yamlorpyproject.tomlto manage both Conda and Pip dependencies in one place.
Build System Integration: Integrates with Setuptools and Hatchling for automatic dependency handling during pip install ./your-package.
One-Command Installation: unidep installhandles Conda, Pip, and local dependencies effortlessly.
Monorepo-Friendly: Render (multiple) requirements.yamlorpyproject.tomlfiles into one Condaenvironment.yamlfile and maintain fully consistent global and per sub packageconda-lockfiles.
Platform-Specific Support: Specify dependencies for different operating systems or architectures.
pip-compileIntegration: Generate fully pinnedrequirements.txtfiles fromrequirements.yamlorpyproject.tomlfiles usingpip-compile.
Integration with conda-lock: Generate fully pinnedconda-lock.ymlfiles from (multiple)requirements.yamlorpyproject.tomlfile(s), leveragingconda-lock.
Example
Example requirements.yaml
Example of a requirements.yaml file:
name: example_environment
channels:
- conda-forge
dependencies:
- numpy # same name on conda and pip
- conda: python-graphviz # When names differ between Conda and Pip
pip: graphviz
- pip: slurm-usage >=1.1.0,<2 # pip-only
- conda: mumps # conda-only
# Use platform selectors
- conda: cuda-toolkit =11.8 # [linux64]
local_dependencies:
- ../other-project-using-unidep # include other projects that use unidep
- ../common-requirements.yaml # include other requirements.yaml files
- ../project-not-managed-by-unidep # 🚨 Skips its dependencies!
platforms: # (Optional) specify platforms that are supported (used in conda-lock)
- linux-64
- osx-arm64
unidepcan process this duringpip installand create a Conda installableenvironment.yamlorconda-lock.ymlfile, and more!
For a more in-depth example containing multiple installable projects, see the
exampledirectory in the repository on GitHub.
Example pyproject.toml
Alternatively, one can fully configure the dependencies in the pyproject.toml file in the [tool.unidep] section:
[tool.unidep]
channels = ["conda-forge"]
dependencies = [
"numpy", # same name on conda and pip
{ conda = "python-graphviz", pip = "graphviz" }, # When names differ between Conda and Pip
{ pip = "slurm-usage >=1.1.0,<2" }, # pip-only
{ conda = "mumps" }, # conda-only
{ conda = "cuda-toolkit =11.8:linux64" } # Use platform selectors by appending `:linux64`
]
local_dependencies = [
"../other-project-using-unidep", # include other projects that use unidep
"../common-requirements.yaml" # include other requirements.yaml files
"../project-not-managed-by-unidep" # 🚨 Skips its dependencies!
]
platforms = [ # (Optional) specify platforms that are supported (used in conda-lock)
"linux-64",
"osx-arm64"
]
This data structure is identical to the requirements.yaml format, with the exception of the name field and the platform selectors.
In the requirements.yaml file, one can use e.g., # [linux64], which in the pyproject.toml file is :linux64 at the end of the package name.
FAQ
Here is a list of questions we have either been asked by users or potential pitfalls we hope to help users avoid:
Q: When to use UniDep?
A: UniDep is particularly useful for setting up full development environments that require both Python and non-Python dependencies (e.g., CUDA, compilers, etc.) with a single command.
In fields like research, data science, robotics, AI, and ML projects, it is common to work from a locally cloned Git repository.
Setting up a full development environment can be a pain, especially if you need to install non Python dependencies like compilers, low-level numerical libraries, or CUDA (luckily Conda has all of them).
Typically, instructions are different for each OS and their corresponding package managers (apt, brew, yum, winget, etc.).
With UniDep, you can specify all your Pip and Conda dependencies in a single file.
To get set up on a new machine, you just need to install Conda (we recommend micromamba) and run pip install unidep; unidep install-all -e in your project directory, to install all dependencies and local packages in editable mode in the current Conda environment.
For fully reproducible environments, you can run unidep conda-lock to generate a conda-lock.yml file.
Then, run conda env create -f conda-lock.yml -n myenv to create a new Conda environment with all the third-party dependencies.
Finally, run unidep install-all -e --no-dependencies to install all your local packages in editable mode.
For those who prefer not to use Conda, you can simply run pip install -e . on a project using UniDep.
You’ll need to install the non-Python dependencies yourself, but you’ll have a list of them in the requirements.yaml file.
In summary, use UniDep if you:
-
Prefer installing packages with conda but still want your package to be pip installable.
-
Are tired of synchronizing your Pip requirements (
requirements.txt) and Conda requirements (environment.yaml). -
Want a low-effort, comprehensive development environment setup.
Q: How is this different from conda/mamba/pip?
A: UniDep uses pip and conda under the hood to install dependencies, but it is not a replacement for them. UniDep will print the commands it runs, so you can see exactly what it is doing.
Q: I found a project using unidep, now what?
A: You can install it like any other Python package using pip install.
However, to take full advantage of UniDep’s functionality, clone the repository and run unidep install-all -e in the project directory.
This installs all dependencies in editable mode in the current Conda environment.
Q: How to handle local dependencies that do not use UniDep?
A: You can use the local_dependencies field in the requirements.yaml or pyproject.toml file to specify local dependencies.
However, if a local dependency is not managed by UniDep, it will skip installing its dependencies!
To include all its dependencies, either convert the package to use UniDep (
), or maintain a separate requirements.yaml file, e.g., for a package called foo create, foo-requirements.yaml:
dependencies:
# List the dependencies of foo here
- numpy
- scipy
- matplotlib
- bar
local_dependencies:
- ./path/to/foo # This is the path to the package
Then, in the requirements.yaml or pyproject.toml file of the package that uses foo, list foo-requirements.yaml as a local dependency:
local_dependencies:
- ./path/to/foo-requirements.yaml
Q: Can’t Conda already do this?
A: Not quite. Conda can indeed install both Conda and Pip dependencies via an environment.yaml file, however, it does not work the other way around.
Pip cannot install the pip dependencies from an environment.yaml file.
This means, that if you want your package to be installable with pip install -e . and support Conda, you need to maintain two separate files: environment.yaml and requirements.txt (or specify these dependencies in pyproject.toml or setup.py).
Q: What is the difference between conda-lock and unidep conda-lock?
A: conda-lock is a standalone tool that creates a conda-lock.yml file from a environment.yaml file.
On the other hand, unidep conda-lock is a command within the UniDep tool that also generates a conda-lock.yml file (leveraging conda-lock), but it does so from one or more requirements.yaml or pyproject.toml files.
When managing multiple dependent projects (e.g., in a monorepo), a unique feature of unidep conda-lock is its ability to create consistent individual conda-lock.yml files for each requirements.yaml or pyproject.toml file, ensuring consistency with a global conda-lock.yml file.
This feature is not available in the standalone conda-lock tool.
Check out GitHub - basnijholt/unidep: Single source of truth with requirements for pip and conda and please ask me to clarify anything when its unclear!
