Pyproject using external sources for packages

Python Help
1

Dear all,

I would like to know whether is possible to make a pyproject.toml point to an external source of libraries. Something like:

[tool.setuptools]
packages = [ 
    git+https://github.com/username/repo.git@branch_or_tag#egg=package_name
]

I. e. I would like to know whether is possible to make pyproject.toml behave like meta.yaml in conda, regarding the source section, as follows:

source:
  git_url: https://github.com/ilanschnell/bsdiff4.git
  git_rev: 1.1.4 # (Defaults to "HEAD")
  git_depth: 1 # (Defaults to -1/not shallow)

I would like to pack the source code from another project, eventually write wrappers and interfaces but not modify the original one. Alternative ideas are welcome!

1 Like
2

The former is possible with a package_name @ url dependency specifier. I’ve also seen many metadata fields used in pyproject.toml. Project urls and version specifiers are so common they’re usually provided in default templates. But unfortunately I’ve not seen a field for git_depth before (what tool to be used with pip needs this?).

This worked just now for me, allowing me to install a dep from a private repo’s github page (with a read permissions fine grained PAT set up):

[project]
name = "test-proj"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
authors = [
    { name = "Muggins", email = "muggins@example.com"}
]
requires-python = ">=3.12"
dependencies = ["private-dep-name @ https://github_pat_1.................w@github.com/your_org_or_user_name/private-dep-name.git"]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

Using hatchling instead of setup tools, I also had to set:

[tool.hatch.metadata]
allow-direct-references=true
3

Note that, as far as I know, distribution packages with URLs in their dependencies will be rejected by PyPI.

4

I would not recommend adding an access token to pyproject.toml. You never want to commit secrets to your repo! Even in a private repo it’s a liability.

A local installer will be able to pull the dependency from a private repository as long as you’re properly authenticated to github (or wherever you’re getting it from)

2 Likes
5

@JamesParrott thanks for the reply.

I am sorry, I just copied the example they have on the conda recipe page, I have not really used git_depth so far and I do not need it as far as I know to prepare conda packages.

I mitght be interpreting your suggestion wrongly but I think that dependencies refers to other packages that my package depends on, right?

What I would like to do is to have a git repository, with the following contents:

dummysrc/
└── cli
    └── foo.py

and another git repository that contains the pyproject.toml, like:

dummypkg/
└── pyproject.toml

on pyproject.toml I somehow specify that it needs to point to dummypkg.

I ran the test just to rule out any possibility with the following pyproject.py:

[project]
name = "dummypkg"
version = "0.0.5"
dependencies = [
    "dummysrc @ git+ssh://git@gitlab.xxxxxxxxxxxxxxxxxxx/dummysrc.git@main"
]

[project.scripts]
foo = "cli.foo:main"

[tool.hatch.metadata]
allow-direct-references=true

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

and the result was the expected:

ERROR: dummysrc@ git+ssh://git@gitlab.xxxxxxxxxxx/dummysrc.git@main from git+ssh://****@gitlabxxxxxxxxxxxxxx/dummysrc.git@main (from dummypkg==0.0.5) does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

I just found hacky ways of doing that so far — like defining custom methods inside the setup.py file, that might not be that hacky but I have a feeling that it is — what I would like to avoid. I might opt to use conda instead if I can not find such functionality.

6

It’s definitely sub-optimal!

I’ve taken great care to use fine grained permissions to scope the token as “read this repo only”. And it only ever gets checked in to the private repo it refers to, and another private repo. So if someone can read the token, worst case scenario, they already had read access to the repo it granted read access to. And there’re no hugely valuable secrets in either, just a bunch of Python code.

There’re definitely better ways for sure, but this slight hack might be useful to someone one day. , and the point of it was just to show that package_name @ url can be used perfectly well in pyproject.toml’s project.dependency array

7

@jamestwebber thanks for your inputs regarding security. Yes, I am aware of secrets in the repository. I do not think I will need authentication for this situation since our projects are all public and open.

8

@sinoroc I did not know that URL in dependencies might be rejected by PyPI. That is something to consider on the choice for the final solution.

9

@georgekontogiorgos Have you considered something like git submodules? To me it does not seem like you need any Python packaging tricks. Seems like you simply need to build the project source directory tree first (with git submodules for example), and then do a relatively straightforward Python packaging.

10

@sinoroc thank you for the suggestion, it was indeed what I was looking for. Unfortunately the repository I want to point to have much more misalignments with standards of pyproject.toml and pip packages in general and thus I gave up on the idea, instead I will contact the maintainers of the repo to try to refactor it to be compatible with the standards.