I’m posting here following a conversation on pip issue #609, where Chris Jerdonek suggested I bring the topic to this forum for a broader discussion.
My use case goes as follow:
- it’s about applications (not libraries)
- I add top level dependencies in setup.py install_requires
- when I need to use some unreleased version of a dependency (top level or not), I add a VCS reference in requirements.in (eg, -e git+https://github.com/someorg/somelib@somebranch#egg=somelib).
- pip install -r requirements.in -e .
- pip freeze > requirements.txt, which pins exact commits of any vcs url I’ve put in requirements.in
I assume this workflow is fairly common?
Notice the editable (-e) option to install vcs dependencies, which is necessary so pip freeze works correctly, preserving the exact vcs url (including commit sha for git) that was installed. pip freeze works in presence of editable installs because it knows about egg-links to find the source and look if it’s a vcs checkout. Given the modern state of things, relying on egg-link and egg-info is not really future proof I suppose.
My first question would be: is there an variant to the above scenario that would work today when vcs dependencies are using another pep517 backend than setuptools (eg flit)? I tried pip-tools which also relies on editable installs for vcs dependencies. A quick look at poetry hinted at a similar behavior.
Working without -e (ie implementing pip issue #609) is not trivial because it requires a new metadata to store the VCS origin and defining it’s semantics might not be easy.
So I tend to think implementing editable installs for pep517 would be a more rewarding approach (given the other use cases it enables).
From my initial research, I must say I find the approach that flit proposes for editable installs is quite elegant and simple (ie generate dist-info + symlink or pth extension). Could it be amenable to standardization with a reasonable effort? I humbly sketched a possible approach for a pep517 extension in https://github.com/pypa/pip/issues/609#issuecomment-478333485. I copy it here for completeness:
- backend provides prepare_metadata_for_install_editable() and install_editable()
- backend install_editable() has the choice to install with the .egg-link method (would be for setuptools only), or .dist-info + symlink, or .dist-info + any other mechanism it wants (.pth, etc)
- when doing an editable install, frontends add a SRCLOCATION metadata to record the source path of the install (ie the local directory from where the install was requested)
- pip uses the new SRCLOCATION metadata (and fallbacks to egg-link) to detect editable installs so pip freeze can continue working normally
- when editable installs create a .dist-info, pip uninstall would work normally (no need to special case, since the RECORD would simply contain the symlinks or pth files to be removed)
I’m willing to put some time and energy into this topic (eg elaborating the proposal, doing some implementation work in pip), but since this problem has been considered hard so far I guess there are probably many hidden complexities I’ve not identified.
Looking forward to your thoughts.