Draft PEP: Recording the origin of distributions installed from direct URL references

Structurally, I think the key thing remaining to do is to nominate a long-lived URL under “packaging.python.org/specifications” where this spec will live, and change the references to “amend this PEP” to instead say “write a PEP to amend this specification”.

At a design level, you identified something I had missed with editable installs, which is that they may be specified in more detail at installation time than just “install from this existing local path”.

Relatedly, for local directories that happen to be VCS directories, it may be useful to take a “snapshot” of the VCS state at the time of installation.

However, I think there’s enough additional complexity in trying to track that as part of the “dir_info” struct that it makes sense to defer tackling that to a possible future iteration (e.g. for editable installs, snapshots of discovered VCS info would likely be better created by “pip freeze” itself, whereas for regular installs, the VCS or archive info from the time of installation would be more relevant. For local directories that happen to be VCS directories, it would likely make sense to extract relevant remote repo information).

For now, I think we can safely duck all those questions by saying “The initial iteration of the PEP does not attempt to make environments that include editable installs or installs from local directories reproducible, but it does attempt to make them readily identifiable”.

A quick reality check here - with that qualification, does the PEP (still) address the requirement that originally started this whole discussion (back in this topic)? It’s not obvious to me that it does (because reproducibility seems to be relevant to that use case).

There’s been a lot of water under the bridge since that original post, and it’s worth checking that we’re still solving the problem we started with.

It does. There were two possible approaches to to achieve the original goal (namely make pip freeze work in presence of VCS requirements): 1/ make editable work with pep 517 or 2/ not using editable for that purpose and writing a new spec. Along the way came a better understanding that using editable for the purpose of pip freeze to work correctly with VCS requirements is a distortion of what editable installs are meant for (namely installing a local directory in development mode). So what we have here is a much cleaner and robust approach that does not require editables at all for that purpose.

The latest version of PEP 610 also achieves another, independent goal, which is making the project location of “modern” [1] editable installs identifiable.

[1] By “modern” I mean editable installs that also provide .dist-info metadata, which is the only assumption we are making about them so far, irrespective of the mechanism used to install them.

1 Like

I can add that to the spec. Although, as you noted, discovering the location of the local project directory is sufficient for known use cases of editable installs.

1 Like

I added the last suggestions from @ncoghlan at https://github.com/python/peps/pull/1282 and https://github.com/pypa/packaging.python.org/pull/690.

For people interested in testing this, a proposed implementation in pip is available at https://github.com/pypa/pip/pull/7612.

1 Like

While testing the implementation I found one minor issue with the stripping of user:password information from URLs. In the case of a requirement such as packaging @ git+ssh://git@github.com/pypa/packaging stripping the git@ part does not improve security and makes pip freeze produce a non-working requirement, providing a sub-optimal UX.

So I propose to add the following language in the PEP:

Additionally, the user:password section of the URL MAY be composed of well-known, non security sensitive strings, such as git in the case of URLs such as ssh://git@gitlab.com.

I made a last PR including my previous remark and a small correction.

To the best of my knowledge all review comments have been handled.

@ncoghlan, all, do you think it is ready to move forward?

Aye, I think so, which means the next step would be to ask @pf_moore as the default BDFL-Delegate for packaging PEPs if he’d like to handle this PEP himself, or if he’d prefer to appoint another volunteer to handle it.

Does anyone want to volunteer for this? Otherwise, I’m fine with making the decision.

@pf_moore I’m happy to volunteer to handle the BDFL-Delegate part of this PEP, in case you don’t have the time to do so. :slight_smile:


As far as I can tell, this PEP has “settled down” and is nearly ready to be decided upon.

  • has no outstanding review comments.
  • had no major comments from non-authors in over a month.
  • has a working implementation
    • tested by users via an opt-in (pip install [the PR ref]), with positive feedback
  • has a PR for documenting this in packaging.python.org w/ minor comments
1 Like

@pradyunsg Agreed. The PEP looks pretty good to me, and I was mostly just waiting for the formal request for pronouncement. I normally like to give the PEP a final review myself before pronouncing, but as I’m not particularly familiar with the subject matter, I’d be perfectly happy with you being BDFL-delegate on this one.

@sbidoul once you’re ready, just confirm here that the PEP is good to be submitted for approval, and @pradyunsg can take it from there. And congratulations on a well-run PEP discussion - I know it took a while, but thanks for sticking with it.

2 Likes

The implementation in pip did raise only one minor issue for which the PEP text has been updated. It is also complete, except for two optional features which can be implemented in follow-ups: 1/ generating the hash for URLs pointing to archives 2/ resolved_revision_type/resolved_revision.

So yes, @pf_moore @pradyunsg, I do think it is ready to be submitted for approval.

1 Like

@pradyunsg soft ping.

1 Like

@pradyunsg If you’re happy to formally pronounce acceptance for this, then the next steps would be:

  • post the acceptance here
  • submit a PR to the PEPs repo that changes the PEP status, links to that post in the Resolution field, and updates the BDFL-Delegate field (if Paul didn’t already update that)

Whoops, I didn’t. Thanks for the reminder - @pradyunsg go ahead and include that change in the acceptance PR, as @ncoghlan suggests.

Thanks for the nudge @ncoghlan @pfmoore!


I’m happy to provisionally accept this PEP! We’ll mark it as “Final” once the rollout is done, to allow for changes (if any are needed) after the initial release of the implementation.

Congratulations @sbidoul @cjerdonek!

2 Likes

PEPs repo PR: https://github.com/python/peps/pull/1330

A big thank you to all involved!