Handling modules on PyPI that are now in the standard library?

Core Development
2

Dustin’s great summary already covers most of the relevant details, thanks.

For reference, here’s a few relevant examples of backports that were brought up:

Name Links Stdlib Backport Maintained Wheel Uppercap Downloads/mo.
argparse pypistats pepy 2.7/3.2 2.3/3.1 2015 Yes No 6.7 M
asyncio pypistats pepy 3.4 3.3 2015 Yes No 2.1 M
typing pypistats pepy 3.5 2.7/3.4 2019/2021 Yes Latest 7.9 M

Observations:

  • The latest version of typing is uppercapped (unlike the other two), but while this is likely to change in the near future, pip just backsolves to older ones. You can see the effect of this in the pepy stats—the previous non-uppercapped version shows a proportion of downloads (≈38%, vs. 52% for the latest, and 10% for older versions) nearly exactly consistent with the total percentage of downloads by Python versions would be affected by the upper cap (42%) minus the percentage of those latter downloads (10%, or 4 percentage points) separately pinned or constrained to other older versions.
  • Something else interesting is going one with typing: the proportions of users on older Python versions, particularly 2.7, is far higher for typing than for the others. Given the version usage patterns, it seems probably it is some specific set of CI services or similar.
  • By contrast, an very large majority of asyncio downloads (≈>95%) and overwhelming majority of argparse downloads (>98%) were for versions in which the stdlib version is present, indicating they are almost certainly by mistake. The distribution of Python versions and OSes, however, was generally consistent with most installations being driven by deep-in-the-stack dependencies and automated processes rather than users manually installing such by mistake.

In any case In addition to those above, there’s two other options to handle this:

  • Release a new, sdist-only version lower-capped to only Python versions on which the module is in the stdlib, and that just raises an informative error when installing
  • Do the above, except making it a warning rather than an error and still installing as normal
    • Or, perhaps better, it could just install an empty import package

My personal conclusions:

  • Writing, discussing, deciding and implementing a PEP:
    • Seems like a large amount of work for relatively little practical gain
    • Would likely take a substantial amount of time to see any of that benefit
  • Yanking seems strictly superior to deleting, as it:
    • Behaves similarly on newer pip versions (which people will progressively upgrade to)
    • Allows raising an informative, customizable warning on older pip versions Python versions where it actually might be desired) without erroring
    • Even older pip versions (that don’t support yanking) are unaffected, which are least important here
    • Preserves history
    • Clearly marks the versions as yanked on PyPI
    • Breaks valid (old Python) unpinned use cases but allows unbreaking them
  • Releasing a new stdlib-version-only sdist that errors on install seems in turn mostly superior to yanking, as relative to that, it:
    • Shows a more helpful warning on recent pip versions
    • Has consistent behavior on older pip versions (and other installers)
    • Only affects Python versions that don’t have the stdlib module
    • Avoids backsolving for earlier versions that are upper capped
    • Downside: Older pip versions will also error rather than warn
  • The status quo seems generally preferable to releasing a stdlib-only sdist that errors, as it:
    • Will immediately break users of libraries or applications that inadvertently depend on these packages
    • Many of those users won’t be able to do much about it except flood the package developers with issues
    • It might never get fixed, as many such packages may be unmaintained, and thus permanently broken on all Python versions
    • The practical benefits/status quo harms seems relatively small, at least compared to the large breakage
  • Releasing a new stdlib-version-only sdist that warns but doesn’t error seems generally superior to the status quo, as it has all the advantages of erroring, and in addition:
    • Doesn’t actually break anything, avoiding the problem above
    • Still provides an informative warning of the problem likely to reach developers
    • Minor downside relative to status quo: a relatively small-ish increase in install time for users on newer Python versions
  • Best of all, IMO, is releasing a sdist that warns and installs an empty package, as it is strictly superior to the previous given it has all the same advantages but also:
    • Minimizes the bandwidth and space costs of the status quo
    • Only breaks in highly unlikely pathlogical cases (that are probably broken, perhaps more subtly, anyway)
    • Reduces (though not eliminates) the one downside of the previous (modestly increased install time)

So, TL;DR, I suggest for these packages we release a new sdist-only version lower-capped to the Python versions that contain the stdlib package that raises a warning on install and only installs an empty package, because it raises a helpful warning on relevant versions, minimizes most of the costs of the status quo (network bandwidth, local disk space), and avoids upper-cap backsolving while not breaking any working use cases or affecting non-stdlib Python versions at all, at only a small cost of install time.

1 Like