I don’t believe that’s necessarily true for packaging interoperability PEPs. I’d assume it’s not an issue for now, and go ahead. (I’m not even quite sure what the role of a “sponsor” is - if it’s just guiding someone through the process, I’d be OK with being the sponsor, but as overall BDFL-delegate I’d rather not take on a role that presumed I supported the proposal, as that’s a conflict of interests).
@ncoghlan - as previous packaging BDFL-delegate, and a member of the steering council, what’s your view here?
Yes, and it’s been very valuable work (please don’t be offended by my “personal” comment above on the complexity of the VCS stuff!)
That’s sort of what I was implying - we need to start off by understanding clearly what the current state of affairs is. Then we can look at standardising - I got the feeling that @sbidoul felt that keeping the behaviour as a pip implementation detail was OK for now, whereas I think it’s the perfect opportunity to get it standardised. I would be surprised if “just do what pip does” became the standard - but I get the feeling that the issue here is fundamental enough that doing it as a pip-specific change just to avoid having that debate is a bad choice.
I get that all the time - probably because I talk too much But I do think Discourse gets a bit too judgmental at times
True, which is one of the reasons why I dislike VCS installs. I think I need to provide some context to my previous comments, however.
“Install from VCS” and “editable install” are two orthogontal ideas. A VCS can be installed either editable or not, and an editable install can either come from VCS or not. Editable installs, in general, is a good thing to have (albeit impossible to be theoratically correct), and I believe is essential.
Install from VCS, however, is another story. Every non-editable VCS package source is only a web/CI service away from a regular package, and for editable installs, you can simply clone/checkout the code yourself to a local path, and install from that instead (and I believe it is basically what pip does). There really is not a theoratical need for pip (or any build frontend/backend) to know anything about VCS, and that is why I think it is not worthwhile to standardise, at least at the current moment.
So if efforts are going this direction, I’d suggest to start with tackling only editable or VCS support. Either drafting out a strategy on local editable install (likely mainly consists of a replacement to symlink, and how metadata should be collected for uninstall, or extend PEP 517 to read/store metadata from VCS first, to correctly support non-editable VCS install. Editable VCS install would come more naturally after those.
From my point of view (as someone who argued for requiring sponsors before it became the rule, and have chatted with the SC about it since then), it’s primarily about avoiding overloading the council with unreviewed PEPs. It just ensures that at least one core developer has reviewed it and thinks it’s a good idea.
So it’s totally fine for you to consider a PEP without a sponsor, just as you also have permission to reject a PEP without spending any time on it on the basis that nobody you know is in support of it. Sponsorship is just the (positive) formalisation of that rejection reason, and it’s designed to reduce and distribute the workload.
@uranusjr can you elaborate this, I don’t understand?
What I have in mind is (simply) to ask pip install <vcs url> to record a source_url metadata, from which pip freeze can generate a requirement that will later produce the exact same vcs download.
I don’t really know what people using VCS install generally try to achieve, and cannot offer much concrete recommendations. What I was saying is that it’d be better to focus to get it right and useful.
Using the source_url metadata as an example, it is very debatable what the “exact same VCS download” means. Pipenv, for example, locks the URL to an exact commit, since that guarentees the same code is checked out (as possible). That would make the process much more involved than simply recording the URL passed in. Moreover, that’s not what some users sometimes want.
As others already noted, there are a lot of hairy details. I’d also guess nobody(?) is exactly sure what everyone else is trying to do, so you’ll need to somehow try to find some use cases to observe, and work hard to get it right.
Correct. A sponsor is required to get it checked into the PEP repo and get a PEP number. And that requirement is to just make sure that at least one core dev views the idea as good and can help the person through the PEP process (you can think of it as a PEP mentor). Everything else up to getting a PEP committed can be done however the PEP author wants without a sponsor.
a proof of concept implementation of pip#609: https://github.com/pypa/pip/compare/master...sbidoul:pip609-sbi (there is still quite a bit of work to do of course - tests, doc, polishing -, but it’s largely functional already and it fits quite nicely in the current pip codebase)
Shall I create the two pull requests to continue the discussion (I’m not quite sure sending a PR to python/peps is the right process)?
I don’t know the preferred location to discuss PEP’s, but I would suggest holding off on proposing any PR’s to pip implementing the PEP or parts of it until it’s clearer what the final PEP language will be.
(Edit: I’m going to hold off on making any comments on the contents itself until it’s clear where that discussion will be happening.)
I’d say that the draft PEP needs at least a round of discussion prior to being submitted for a PEP number. It’s not obvious where that discussion should happen, but I’m strongly of the opinion that distutils-sig needs to be consulted as well as Discourse - a proposal that’s exclusively discussed on Discourse is likely to not be noticed by at least some of the community. Posting a draft there and noting that follow-up discussions are preferred on Discourse is fine, but you still need to monitor distutils-sig for comments.
Some practical notes on the draft:
“They are also capable of downloading and installing source code from arbitrary URLs and Version Control Systems (VCS).” This should be supported by reference to the relevant standards clarifying what arbitrary download capabilities are standardised, as opposed to pip-specific.
“Freezing an environment” - this is solely based on the pip-specific freeze functionality (with a brief mention of Pipfile.lock). As such, it’s a relatively weak justification in itself - you’re either proposing functionality in support of a pip-specific feature, or you should be considering whether the PEP is actually proposing a standard for storing metadata to allow reproduction of a precise environment from installed data only (essentially standardising the ability to write an external “freeze” tool).
“When the package was installed from an arbitrary URL, the (name, version) tuple is obviously not sufficient to reinstall the same distribution.” This is a very fundamental point, and it’s buried in the document. While we do not yet have any standards around tools like resolvers or package finders, explicitly documenting in a standard that (name, version) is not intended to be a unique identifying characteristic of a distribution is a major change - unless you’re extremely careful how you word things, you risk making pip’s current behaviour (of not re-downloading packages where the name and version match) non-compliant.
I’ve not fully read the proposed PEP, just skimmed the first sections. I’ll defer a more detailed review for when it gets published for review. But the above 3 items immediately struck me as needing work before the proposal is ready to be submitted as a PEP.
I believe the relevant standard is PEP440 direct references. Reference added.
To clarify, I replaced some occurrences of pip freeze with “freeze” so it’s more explicit that I’m referring to a generic operation.
So in short this PEP is indeed to solve a pip issue, using source_url, to be useful to others with similar needs.
In a way, it’s also “just” reviving PEP426 source_url, that was considered useful for a PEP at the time.
If the motivation is too weak for a PEP, then I feel stuck, because dist-info is the only place I see to store the information needed to enhance pip freeze, and one thing I got out of this thread so far is that pip should not define it’s own metadata format.
I’m just saying (name, version) is not a unique identifying characteristic when the requirement was a arbitrary URL (as opposed to a regular non-url requirement which gets downloaded from the index or via --find-links). Is that not correct? I’d be happy to improve the wording.
To clarify, I expanded the Examples section to show which pip commands generate a source_url metadata, and which do not.
I’ll create a PR to python/peps, it’s what PEP 1 recommends, and it’s the most comfortable to discuss details IMHO. And I’ll post to distutils-sig to draw a wider audience.
FYI I just closed the PR against the peps repo as this PEP needs to be fully discussed at least once and have a sponsor before we will consider accepting it to the index. And we explicitly discourage people using the peps repo for work-in-progress PEPs to keep traffic down on that repo.
@brettcannon Can @sbidoul choose where he’d like the discussion to take place primarily (e.g. Discourse or distutils-sig)? If done here, it should be a new topic I think and dedicated for that purpose (and noticed on distutils-sig).
Agreed, that sounds about right to me (although I’d be inclined to include the full text of the proposal in the announcement on distutils-sig, to give people who haven’t already signed up to Discourse some context).
The Discussions-To field exists specifically to help people direct the discussion to the appropriate place. It sounds like everyone likes a new topic here with an announcement email on distutils-sig so that’s also what I would recommend to @sbidoul.
People can read the raw reST without much issue. You can also point people to your own fork of the peps repo with your proposed PEP and then GitHub will render the PEP (mostly) appropriately.
Another trick that’s sometimes useful: if you have some formatted html open in a web browser, and you copy/paste from the web browser into the discourse comment form, then there’s some magic where it will automatically try to convert the formatting into discourse markdown. It does a pretty good job, too.
But I do think Discourse gets a bit too judgmental at times 