# Fallback links on PyPI for the same file **URL:** \ **Category:** Packaging\ **Created:** [March 29, 2022, 5:17am UTC](https://discuss.python.org/t/fallback-links-on-pypi-for-the-same-file/14678 "2022-03-29T05:17:16Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![EpicWink](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/epicwink/32/17968_2.png) [@EpicWink](https://discuss.python.org/u/EpicWink)\ **Post date:** [March 29, 2022, 5:17am UTC](https://discuss.python.org/t/fallback-links-on-pypi-for-the-same-file/14678/1 "2022-03-29T05:17:16Z")
I’ve had a thought for allowing external hosting of files in PyPI while addressing previous concerns: link to the externally-hosted file in the package’s (simple-index) page on PyPI, but also link to the exact same file hosted by PyPI after it. The dependency installer (eg Pip) would first attempt to install the first link\[1\] on the HTML page provided by PyPI (which would be the external link), then fall-back to later links for the same file. It seems Pip’s implementation will only attempt the first result currently, so I would assume there has to be buy-in from all the current dependency-installers to supports this. Correct me if I’m wrong, but it seems like Pip associates one link per candidate (ie file). This would reduce PyPI’s file-storage bandwidth cost, and storage cost (see below). If there’s interest in this, I could cook up a PEP (or similar). To address the prior concerns: - If the connection fails or the download time (normalised by content-length - must be present) exceeds a threshold, fall-back to the PyPI option (optimisation at implementer’s discretion). This addresses reliability, speed and firewalls - Externally-hosted links must come with a check-sum (computed by PyPI on upload). Addresses security - The implementation should be wholly transparent to the end-user, so no UI/UX changes at all (perhaps a flag to disable external hosting) - Perhaps “uploading” external file links should not be included in Twine’s features to increase the barrier to entry, not sure what this addresses - PyPI could store the fall-back files on an intelligently-tiered file-store (I’m used to [S3 terminology](https://aws.amazon.com/s3/storage-classes/)), where files would be moved to cheaper-to-store (but expensive-to-access) tier when not accessed for a while (frequent access would move it to a high-availability tier). Addresses cost - PyPI could randomly check external files for existence to see whether to (temporarily?) remove the external links - Because the file is still being uploaded to PyPI, virus checks (does PyPI do this?) and Terms of Service agreements still happen Further concerns: - Privacy: external hosts can log requests with IP/agent details. Perhaps PyPI (or its CDN) can route the requests to the external providers through itself (note: not redirect) - I haven’t thought of a good way to report to external hosts why the dependency installer aborted a file-download. A CLI warning doesn’t seems like a good idea - Backwards compatibility: old versions of installers would start downloading the externally hosted files. Perhaps the PyPI-hosted files should be listed first? Previous discussion on external hosting: - [External hosting linked to via PyPI - Packaging - Discussions on Python.org](https://discuss.python.org/t/external-hosting-linked-to-via-pypi/8917) - [What to do about GPUs? (and the built distributions that support them) - Packaging - Discussions on Python.org](https://discuss.python.org/t/what-to-do-about-gpus-and-the-built-distributions-that-support-them/7125) - [PEP 470](https://peps.python.org/pep-0470/): - [Mailman 3 PEP 438, pip and --allow-external (was: “pip: cdecimal an externally hosted file and may be unreliable” from python-dev) - Distutils-SIG - python.org](https://mail.python.org/archives/list/distutils-sig@python.org/thread/2BLAJACYCV4A5CIUD2D667W7UEGRERBW/#2BLAJACYCV4A5CIUD2D667W7UEGRERBW) - [Mailman 3 PEP 470 Round 2 - Using Multi Index Support for External to PyPI Package File Hosting - Distutils-SIG - python.org](https://mail.python.org/archives/list/distutils-sig@python.org/thread/IVTLGEZLYT3YELE74SED3PWRBDBRASWZ/#IVTLGEZLYT3YELE74SED3PWRBDBRASWZ) - [Mailman 3 new PEP submission: transitioning to hosting release files on PYPI - Python-ideas - python.org](https://mail.python.org/archives/list/python-ideas@python.org/thread/M6B6H6OVQ72ITD6TLFGD42RPC77MIWRX/#M6B6H6OVQ72ITD6TLFGD42RPC77MIWRX) - [Mailman 3 PEP 470 - Once More, with Feeling - Distutils-SIG - python.org](https://mail.python.org/archives/list/distutils-sig@python.org/thread/UE3C7LNPCPIH7TA3KVOG4L6FTO5DTM5D/#UE3C7LNPCPIH7TA3KVOG4L6FTO5DTM5D) - [Mailman 3 PEP470, backward compat is a … - Distutils-SIG - python.org](https://mail.python.org/archives/list/distutils-sig@python.org/thread/X762YYUOM6VCDROYUEJJR3LIT5MT5SMX/#X762YYUOM6VCDROYUEJJR3LIT5MT5SMX) * * * 1. link ~ URL ~ internet reference ~ anchor HTML element ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [April 5, 2022, 9:18am UTC](https://discuss.python.org/t/fallback-links-on-pypi-for-the-same-file/14678/2 "2022-04-05T09:18:18Z")
I think privacy would be an important concern, the eternal link will need to be proxied. Come to think of it, if the link is always proxied, maybe we don’t even need the API and client side code to change at all, but simply point them to the proxied link, and do the fall back there. ---
**Author:** ![EpicWink](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/epicwink/32/17968_2.png) [@EpicWink](https://discuss.python.org/u/EpicWink)\ **Post date:** [April 5, 2022, 9:49pm UTC](https://discuss.python.org/t/fallback-links-on-pypi-for-the-same-file/14678/3 "2022-04-05T21:49:45Z")
Downloading files through the index server would be a drastic increase in the index server’s utilisation (compute and network bandwidth). Is there some solution using the CDN? ---
**Author:** ![ppwwyyxx](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ppwwyyxx/32/4231_2.png) [@ppwwyyxx](https://discuss.python.org/u/ppwwyyxx)\ **Post date:** [January 1, 2023, 9:27am UTC](https://discuss.python.org/t/fallback-links-on-pypi-for-the-same-file/14678/4 "2023-01-01T09:27:04Z")
Thanks for bringing this up! I also looked at the topic earlier at ([External hosting linked to via PyPI](https://discuss.python.org/t/external-hosting-linked-to-via-pypi/8917)). Is there any follow up work on this? If anyone wants to draft a PEP I’m happy to help out.