@dstufft I think you will find that zstd’s memory usage is the smaller of the window or the size of the (decompressed) data.
Replace everything but the .dist-info directory with a .data.zip with the same name as the existing .data directory plus the .zip extension. The inner .zip file includes members at the root of the wheel.
(With improved compression it would be equally valid to put every file in the .data/purelib and .data/platlib folders and not use the root, but that would not be the case when converting existing wheels).
Only .zip or .zip.zst not .bz2 etc.
If it is a .zip, STORE the files in the inner .zip and compress them with standard zipfile compression in the outer zip. Otherwise STORE the files in the inner zip, compress the inner zip with zstd, and STORE in the outer zip.
To preserve the hash of MANIFEST when converting back from an extra-compressed wheel to a regular wheel:
When converting from a regular wheel to an extra-compressed wheel, follow the file order in MANIFEST so that the same order is expressed in the inner zip’s index. Regenerate MANIFEST with the inner zip and .dist-info entries in that order.
When converting from an extra-compressed wheel to a regular wheel, copy the inner zip’s index into MANIFEST in order, again leaving .dist-info last.
To resolve ambiguity:
Allow empty directories as empty files with names ending in / per zip standard, plus an optional mention in MANIFEST with the hash of an empty file.
Allow zip standard symlinks.
Install files based on whether they start with a prefix package-1.0.data/{purelib,platlib,scripts,…}/ or none of the above in which case they are deemed to be in the root; so allow .data/unkownlib/ to be installed as a directory under PURELIB or PLATLIB, depending on where the root of the wheel is installed, with a warning. To allow additional install paths to be added in the future.
There would be a reference converter. Before installers have the time to write optimized logic for this update, they trap extra-compressed wheels, run through the converter, and install with their existing wheel code. Alternatively an iterator producing (‘path inside wheel’, filelike) transparently extracting any inner zip.