How specify a lib dependency

Python Help
,
21

To an extent this is because the pyproject.toml is aimed at
distribution; how to put local-to-you things in such a thing?

Probably the best way is to make an environment. Typically that is a
venv, eg:

 python -m venv your-venv-directory-here

and to install into that - now you have an area where the concept of
“this package is/is-not present” makes sense.

For my own dev tree that’s usually a subdirectory venv with needed
third party packages installed. When I run the code in dev mode I use a
little shell script which:

  • puts my dev dir’s bin subdir (with the local scripts in it) at the
    front of $PATH followed by the venv/bin directory to get the lcal
    venv python executable, which is wired to find the packaged installed
    in the venv
  • puts the local dev python code at the start of $PYTHONPATH so that
    the dev python code is found and used
  • runs whatever command I intended to run

So the other day I was doing a lot of:

 dev python3 bin/hashindex ls

or:

 dev hashindex ls

to run my new hashindex script which relied on the unreleased code in
my local dev tree. Indeed, the script itself was also undergoing change.

Now, when things are local you can do an “editable install” of a
package, which is particularly handy for dev work.

For your personal dev tree you’re ok I’d imagine? Just tweak
$PYTHONPATH?

I’m assuming specifying one of your own packages as a dependency is for
use by others, or in another (sub)project needing the package. That’s
the time for an editable install, maybe. See:
https://pip.pypa.io/en/stable/topics/local-project-installs/

If you’re using pip’s requirements file (just a file with required
packages and versions) you can put a -e line in there. For example,
I’ve got these lines:

 -e git+https://github.com/kkroening/ffmpeg-python.git@master#egg=ffmpeg-python
 -e git+https://github.com/ytdl-org/youtube-dl.git@26035bde4#egg=youtube-dl

to obtain a specific unreleased version of a package as ffmpeg-python
and youtube-dl respectively. But you can put a local filesystem path
in there as well, outlined at the URL above.

Then:

 ./venv/bin/python -m pip install -r requirements.txt

can apply to your environment.

This doesn’t go in the pyproject.toml AFAIU, that is for distribution
and I think assumes use by a build/install tool which access to a
distribution source like PyPI.

2 Likes
22

That is helpful. The way you have articulated the motivation got me to a better understanding. It highlights a gap. The packages will be used without publishing. So, the only way to include the local dependency is by updating my venv, or my PYTHONPATH as you have outlined.

Perhaps useful to include a [dependencies.local] or the like in the pyproject.toml to enable a “local/private distribution”? (a rhetorical question that I will track when reading the docs and proposals over time).

23

That is helpful. The way you have articulated the motivation got me to a better understanding. It highlights a gap. The packages will be used without publishing. So, the only way to include the local dependency is by updating my venv, or my PYTHONPATH as you have outlined.

Well, you don’t have to publish to PyPI. It is possible to run local or
institutional repos. (I have never done this.)

Perhaps useful to include a [dependencies.local] or the like in the
pyproject.toml to enable a “local/private distribution”? (a rhetorical
question that I will track when reading the docs and proposals over
time).

If you’re sharing the code locally, maybe have a shared venv? Then you
could put the packages in it for others to use.

If it’s pure Python you don’t even need that - just have a public
“released” (for whatever criteria you want that to mean) tree of the
code, and get other users to put that in their $PYTHONPATH when they’re
using the code.

Or even a shared requirements.txt file - you can do noeditable local
installs too, just include the path to the required modules (if I’m
reading the earlier URL correctly) - your users could then:

 python3 -m pip install -r /the/shared/reqs.txt

to pull your packages into their local python environments.

It’s not clear to me what the target use case is for your need to define
“local” requirements. Can you elaborate a bit on that?

24

I’ll just add that all of this is a heck of a lot easier if you’re using GitHub or a similar service that will serve up your code as a VCS. I use private dependencies all the time, but I do it via git repo, and pip is just fine with that. pip install git+ssh://git@github.com/[orgname]/[repo].git works seamlessly as long as you’re authorized to pull from the repo.

The only potential hiccup is if you’re using pip within another subprocess[1]–it doesn’t allow an interactive prompt in this situation, so if you need a password for your ssh key it will fail to install. The solution there is to use ssh-agent.

If the goal is “share private code within an org” then this is a good solution (and VCS is a Good Idea™ anyway). If you really need to share code locally then maybe it’s not the right way.

  1. for instance, as part of setting up a conda env ↩︎

25

This is part of a larger discussion that has proven very complex and difficult so far.

1 Like