Speculative: Wheel 2.0 and migration strategies

[dholth (Daniel Holth)]

A wheel 2.0 would be smaller and faster. In a previous thread (no longer hosted of course),

Making the wheel format more flexible (for better compression/speed)

I’ve put up an index of 925 popular wheels in the prototype “nested .data.zip.zst” format. It is at https://w2test.website-us-east-1.linodeobjects.com/ unless I run out of transfer. It is suitable for use as pip install --extra-index-url=https://w2test.website-us-east-1.linodeobjects.com/simple/ [package] but you will need a patched pip from https://github.com/dholth/pip/tree/zstd-wheel Went from 2.6G originals to 1.5G using zstd -19 -T2.

Since this idea represents exactly the same information (down to the individual ZipInfo objects in the nested zip, compared to even using a nested tar), this is like an encoding change - you could losslessly translate back to the old format and use the old tools and vice versa.

I don’t know how many people have trouble using the current version of wheel for size and speed reasons. But pypi’s bandwidth would get a large percentage of the benefit if the only wheel 2.0 package was tensorflow.

The less developed ideas would have something to do with representing new features, for example a more flexible installation scheme (fix the data-files feature) or maybe something to do with linking to system packages.