An update on pip and dependency resolution

Packaging
#1

I just posted an update on my blog, about the work being done toward pip’s dependency resolver (https://github.com/pypa/pip/issues/988) and what the plans are on that front.

https://pradyunsg.me/blog/2019/06/23/pip-update/

Thanks @sumanah for suggesting that I do this! =)

4 Likes
#2

I’m going to use this thread to share some in-progress notes on the resolver work.

As of right now, people who install pip from GitHub master will have the ability to run pip install --unstable-feature=resolver and test the new resolver code. And less than half of the test suite fails! Expect errors and missing features, but it’s there!

A proper alpha or beta release will probably happen in May, and we’ll be sharing a “what to test” guide and asking for a lot of testing at that time. I’m working on a pre-announcement now that will go to the PSF blog and that we’ll be publicizing from there, and of course linking to from our wiki page.

Thank you to Mozilla (through its Mozilla Open Source Support Awards) and to the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation, for funding enabling this work! And thanks to @ei8fdb and Georgia Bullen from Simply Secure, and to @pradyunsg @pf_moore @uranusjr and Ilan Schnell for your work on the project.

2 Likes
#3

The new PSF blog post is up. It discusses what is going to change, when, and how you can help (including some low-effort things you can do right now).

1 Like
#4

Current work and near-term plans:

  • Our user experience researchers @ei8fdb and Georgia Bullen have been helping us think through specific decisions about the rollout. And they continue to interview you (please sign up if you haven’t already!) to learn more about people’s mental models, use cases, workflows, etc. They aim to put together a preliminary report this month to give pip’s developers some more systematic info, and I hope this will also be of use to some other packaging tools developers.
  • Ilan Schnell is working with us to improve our existing tests/testing and to add more tests to validate the new resolver.
  • We’ll be releasing pip 20.1b1 early this week, and it’ll include an alpha version of the new dependency resolver that people can test if they so feel inclined using --unstable-feature=resolver. Here’s context and some testing instructions. I’ll be publicizing that somewhat. This is thanks to the work of @uranusjr @pradyunsg & @pf_moore.
  • Then, in May, we’ll be releasing a pip version that will include a beta version of the new resolver, and I will publicize that (along with a more in-depth what-to-test guide) even more.

Thanks again to Mozilla & CZI for funding enabling this work!

4 Likes
#5

I’ve posted our mid-year report at the PSF blog. One key thing to know: later this month we’ll be releasing pip 20.2 which will have a beta version of the new dependency resolver (pip 20.1 had an alpha version) available via an optional flag “--use-feature=2020-resolver”. We’ll be publicizing pip 20.2 a lot and asking a lot of users to put the new resolver through its paces.

#6

We’ve released pip 20.2 and are discussing that at Announcement: pip 20.2 release! .

#7

We made a small bugfix release, 20.2.1.

Also, members of the pip team gave interviews about the dependency resolver changes in two podcasts recently: in late May on Podcast.__init__ and more recently on Test & Code.

1 Like
#8

Since pip 20.3 will come out next month and the new resolver’s behavior will be on by default, the pip team made a 2-minute video to explain what’s up:

And https://twitter.com/ThePSF/status/1311038036013199363 is a good tweet to retweet if you want to help us get the word out.

#9

The links from the video are:

#10

Oh boy. Lemme take a look at this.

#11

@brettcannon thanks for the bug reports.

I can’t reproduce your problem getting to https://bit.ly/pip-ux-studies (which should direct to http://www.ei8fdb.org/thoughts/2020/03/pip-ux-study-recruitment/ ). What browser and so on are you using?

The YouTube metadata and the Twitter post include a more direct link to the migration guide. https://pip.pypa.io does not currently link to a migration guide on the front page – it will once we make another point release and what’s at https://pip.pypa.io/en/latest/ right now shows up at https://pip.pypa.io/en/stable/ . As a kind of workaround, since the “learn more” link points to an explanatory blog post, I’ve added a link to the migration guide at the top of that post. :\

#12

Firefox 81.0 (64-bit) on Windows 10.

#13

You probably have an extension or a Firefox setting requiring HTTPS, but the page is only served in HTTP. Firefox is raising an error (specifically PR_END_OF_FILE_ERROR code) instead of falling back to HTTP.

To test, try heading to: https://www.ei8fdb.org/thoughts/2020/03/pip-ux-study-recruitment/

#14

I have HTTPS Everywhere installed, although typically it asks if I want to make an exception if there is no HTTPS. But I guess because the HTTPS handshake is failing it is just flat-out blocking me.

1 Like
#15

pip 20.2.4 is out right now and has a bunch of performance improvements compared to previous 20.2.x releases, so it’s worth trying out.

We intend on releasing pip 20.3, with the new dependency resolver as default, next week, maybe Wednesday or Thursday the 28th or 29th of October.

#16

pip 20.3b1 is now out. We decided to release a beta this week and then release 20.3 in mid-November partly to avoid disruption that might affect people dealing with and countering US election-related unrest. Here’s the changelog.

1 Like
#17

Announcement: pip 20.3 release! pip 20.3 is out.