For some additional context, this is not just affecting PyPI but many other pieces of FOSS infrastructure as well: https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/
10 Likes
Thanks for sharing. In the simple-repository-browser project I also added a rudimentary search. It handles queries for description, and specific search based on summary text (e.g. name:xarray, summary:xarray for finding info specifically in the name or summary), but I didn’t yet get the time to implement classifier search and full-text description search. I am also super keen to add search for dependants, e.g. depends-on:tensorflow.
There is a lot of space for improving my implementation, but I wonder if there is a way that we can pool our knowledge to produce a powerful alternative search which we can embed into our respective applications (please reach out if this is of interest!).
1 Like
Evidently packages doing 700 releases in one day is now also a thing. 
I’m amazed that PyPI still works at all.
7 Likes
we do what we can 
9 Likes
I implemented Fastly challenge solver in my pip_search fork.
It’s only intended for low-volume manual searches, and I also added USE_POLICY.md in my hope it reconciles potential conflict of interests @kpfleming
I also want to mention that IMO it would be relatively cheap to design and implement new PyPI search API which would be less prone to issues which caused the former one to go.
The one approach is JWTs so that requests with invalid or expired ones could be dropped at the edge before they would reach the service.
Level up would be e.g. Cloudflare API Shield and actively blacklisting valid JWTs which are part of requests with high bot-score.
In any way, this is engineering problem which has a clear solution, and in case of Cloudflare, free tier is not None in terms of value you get (disclaimer - I’m not affiliated with Cloudflare in any way).
With the caveat I have limited availability, I would be happy to contribute to such an API work, just in case there is potential interest.
FWIW I also had a go at a pip search replacement a while ago. This one behaves a bit like a typical Linux distribution package manager in that it builds one database which the client then downloads and searches locally. It’s less information but you can hammer as many queries as you like without affecting PyPI since it’s only your own flops your spending.
2 Likes
Open data is substantially strained by automated tooling. E.g. Openstreetmap is also seeing massive traffic increase from scraping. #opendata #osm #openstreetmap #ai #bots #abuse | OpenStreetMap | 287 comments
It’s good if the current measures still hold, but it’s possible/ likely that they will be worked around in the future.
I notice that fastly has now added a capcha as well. That sucks, as it means that my pypi xxx search shortcut in my browser (which I only ever used to go direct to module xxx) is no longer really a shortcut. So now I have to try to remember where I go in Firefox to fix my search to go direct to the project page.
I totally support efforts to stop automated scrapers from overloading PyPI and other open source projects, but this definitely feels like one of those “this is why we can’t have nice things” situations 
2 Likes
The CAPTCHA is being generated by Fastly, but the decision to enable it was made by the PyPI infrastructure team 
2 Likes
I assumed so, and as I say, I support their efforts to keep the service running. I’m annoyed at the people running the bots that abuse the service, but not at the PyPI team.
9 Likes
Given solving captchas in an automated manner is trivial in the world of website scraping, I assume that the inevitable next steps are users are only allowed access to search if they are authenticated, and then only allowed access to search if they have 2FA enabled.
It’s a shame that the bot creators can’t easily be identified, or contacted, and be transitioned into paying customers for their usage.
4 Likes
I’m assuming a lot of it is AI companies
2 Likes