Enabling editable installs for local subpackages

Python Help
,
1

Hi All,

I am attempting to create setup.py and pyproject.toml scripts for a package I am working on. I have included a toy example for reference here:

package layout:

.
├── pyproject.toml
├── setup.py
├── subpkg
│   ├── setup.py
│   └── subpkg
│       ├── __init__.py
│       └── subpkg_mod.py
└── toypkg
    ├── __init__.py
    └── toypkg_mod.py

setup.py of toypkg:

from pathlib import Path
from typing import List

from setuptools import find_namespace_packages, setup


def local_pkg(name: str, relative_path: str) -> str:
    """Returns an absolute path to a local package."""
    return f"{name} @ file://{Path(__file__).parent / relative_path} "

requirements: List[str] = [
    local_pkg("subpkg", "subpkg"),
    "numpy",
]

setup(
    install_requires=requirements,
    name="toypkg",
    packages=find_namespace_packages(include=["toypkg", "toypkg.*"]),
    include_package_data=True,
    version="0.2.0",
    zip_safe=False,
)

pyproject.toml of toypkg:

[build-system]
requires = ["setuptools >= 64"]
build-backend = "setuptools.build_meta"

[tool.black]
line-length = 88
target_version = ['py38']

Running pip install . works as expected. I would like to enable the ability to have an editable install of the package as well as the subpackage. If I run pip install -e ., I am able to get an editable install of toypkg but not of subpkg. I have also tried using a requirements_dev.txt file, with a command pip install -r requirements_dev.txt

requirments_dev.txt:

numpy
-e subpkg
-e .

But I am met with an error message stating:

ERROR: Cannot install -r requirements_dev.txt (line 21) and toypkg 1.0.0 (from /home/user/toypkg/subpkg) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested subpkg 1.0.0 (from /home/user/toypkg/subpkg)
    toypkg 0.2.0 depends on subpkg 1.0.0 (from /home/user/toypkg/subpkg)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Which has me really perplexed given that the version numbers match.

I assume I am going about this all very wrong. If anyone knows the proper way to achieve what I am trying to do, I will be in your debt. Any and all help is greatly appreciated.

2

Maybe conflicting deps was not the best error message and it just couldn’t find it? Where in subpkg’s code/config is it declared to be v1.0.0?

3

Hi @JamesParrott the version number for subpkg is defined in its setup.py:

from pathlib import Path
from typing import List

from setuptools import find_namespace_packages, setup

requirements: List[str] = [
    "pytest",
]

setup(
    install_requires=requirements,
    name="subpkg",
    packages=find_namespace_packages(include=["subpkg", "subpkg.*"]),
    include_package_data=True,
    version="1.0.0",
    zip_safe=False,
)

I can provide additional context if needed.

4

Thanks. I’ve not tried using -e before in a requirements.txt file. I would manually install anything that needed -e in a separate prior step.

It’s possible to have two packages within the same Python project (sharing a pyproject.toml and specifier if on PyPi). If it will never be used outside of toypkg I’d do that. Otherwise I’d refactor subpkg into its own separate project directory, outside of toypkg’s.

It’s possible to put more info into the dependency specifier to, e.g. "project-name" @ uri to make sure the project name isn’t being confused with the directory name.