Pypi.org recently changed?

Good afternoon —

Suddenly a week or two ago all of our dependency installs started going bonkers with this type of message:

Processing dependencies for aws-cfn-bootstrap==1.4
Searching for pystache>=0.4.0
Reading https://pypi.python.org/simple/pystache/
Couldn't find index page for 'pystache' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
No local packages or download links found for pystache>=0.4.0
error: Could not find suitable distribution for Requirement.parse('pystache>=0.4.0')

Yes, we’re still using easy_install for a few things, but I digress.

It seems that pypi.python.org has been replaced with pypi.org — which is great! Except it’s broken a bunch of things!

Is this change documented somewhere, is there a recommended solution other than, “Use pip!”, and is there a place I can watch or subscribe to for future changes of this nature?

Thank you!

1 Like

This a transition doc to help you is at Migrating to PyPI.org - Python Packaging User Guide.

But do note that use of pypi.python.org has not been available for uploading new packages for over 3 years, so this transition has been planned for quite some time (hence why you have not heard anything as I suspect a lot of people like me forgot that pypi.python.org still existed in any form).

https://status.python.org/ has a “Subscribe to Updates” button you can use to get updates pushed to you.

1 Like

Much appreciated!

2 Likes

Do you know if something changed with the old URL as of 4/14/2021 or thereabouts? It seems like something may have, because lots of things started to fail around then.

I’m going to venture a guess that since the URL was changed on 4/13/2018 that there is an SSL cert that expired on 4/13/2021.

Unfortunately no. If it’s really important you can reach out to the infrastructure staff for PyPI, but honestly I would just update to use pypi.org at this point.

Thanks - do you know where I can find their contact info?

Hi all,

We’ve been experiencing the exact same issues with the AWS CloudFormation bootstrap scripts. The weird thing is that it’s completely intermittent, and only happens when pip is run inside that script. When running pip install pystache manually it can resolve and install the dependencies.

Maybe it is the redirect from pypi.python.org, however wouldn’t we experience the same issue when running pip manually?

Because of the intermittent issues running the bootstrap, I thought it may be related to the non-SNI client deprecation rollout explained here: https://github.com/pypa/pypi-support/issues/978

And lo and behold, if I get the bootstrap script running outside the “brownout” window (Currently :20 - :30 of every hour) the dependencies are resolved absolutely fine. However, this non-SNI client deprecation apparently will only affect Python 2.6, however our instance uses Python 2.7 and Pip 8.1.1 (we are unable to upgrade pip at this time). We also did a test run of this using Python 3 and the 2.0.6 version of the AWS CF Bootstrap script, and the same intermittent issue occurs, so it’s not isolated to older python versions it seems :thinking:

Jimmy, i’d be interested to hear if your cloudformation script works between 20 past and half past each hour, if so it would seem we have discovered some potential fallout from this deprecation rollout.

1 Like

Yes, you are exactly right. This brownout thing is/was a terrible idea because when people are trying to troubleshoot the issue and it suddenly is working again it can certainly make you feel crazy. Compound that fact with very little information online about this and it brings us here.

In addition to the AWS EC2 CloudFormation bootstrapping issue, we have also seen issues installing dependencies on EMR clusters. The issues there were even worse because we were seeing 0 exit codes from these types of failures, meaning the servers spun up just fine (seemingly) but when the scripts went to run they would complain about missing dependencies. Digging into the bootstrapping logs revealed the issue.

The infra team’s choice to do this brownout style has made it exceptionally difficult to diagnose for ops folks.

It’s interesting you say it impacts higher versions. We upgraded to 2.7.9 and thought we resolved it, but looking back I see we didn’t test between 20-30 past. I will do more testing today.

If it’s helpful, this is what our ops did to fix this on the AMI we use:

apt-get update
apt-get -y upgrade
apt-get -y install build-essential
apt-get -y install libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
wget https://www.python.org/ftp/python/2.7.9/Python-2.7.9.tgz
tar -xvf Python-2.7.9.tgz
cd Python-2.7.9
./configure
make
make install
cd ..
wget https://bootstrap.pypa.io/pip/2.7/get-pip.py
python get-pip.py
/usr/local/bin/pip install aws-cfn-bootstrap

Dependency list came from here: Installing Python 2.7.9 on Ubuntu 14.04 LTS | Ralphlepore.net

He said he recompiled it and then downloaded the correct pip that supported SNI, which resolved the issue.

Good luck!

Well, it did make you take notice that something is coming. :wink: It’s really hard to get people to notice this sort of thing, even when there’s stuff announced in various places (e.g. Twitter, etc.).

Well sure, but it really makes it difficult to diagnose and fix. Just turn it off.

@brettcannon I quite like this as there are opportunities to get things working for 10 minutes every hour, and the rest of the time can be spent testing a long term fix. That said, it took me ages to actually find this issue page, but that may be down to my debugging practices.

@jimmy0x52 Thanks for the fix suggestion from your ops team. I will try it out today.

Just wanted to note something for anyone else coming here, it seems that I skipped over this comment in the above PyPi issue:

However, it turns out that the setuptools package itself is the important one here. Setuptools added SNI support with version 36.8.0 in pull request #1190

The CF bootstrap script uses setup-tools it seems, and my Ubuntu instance has a really old version of this, so I am going to try using a newer version of setuptools explicitly and hope it fixes it.

Hi all,

Just an update from me around this.

For anyone experiencing issues here, I ensured that before the bootstrap is run that I:

  1. Download and use a newer version of Pip (in my case for using Python 2.7, we use Pip with a specification of any version <21.0)
  2. Use pip to explicitly install setuptools with version >=36.8.0.