On the topic of data layout in zip archives: while it’s convenient for streaming purposes that the zip format has its index at the end (you can create zip files in parallel recursively this way), I think @zwol noted that that’s an example of optimizing for ease of writing (which happens just once), rather than reading (more than once, usually).
I believe this complaint can also be levied against tar archives, which don’t even support seeking to an entry without being augmented with an index like estargz provides (stargz-snapshotter/docs/estargz.md at main · containerd/stargz-snapshotter · GitHub).
In fact, even the recursive parallel merge approach for creating zip files just throws away the index entries for every intermediate zip except the final one. Which would seem to imply that the problem isn’t which end you crack your egg on, but that we have to crack the egg in the first place!
Which is to say that encoding the entry index inline with the entries themselves seems like an assumption worth revisiting. In discussions around 2020 or so with Jon Johnson I recall it was exciting to realize --fast-deps and e?stargz had independently discovered the same principle, but at the time I was convinced it was an advantage over tar archives that zip files already have that index available. Now, I see that estargz can claim:
This extension is a backward-compatible so the eStargz-formatted image can be pushed to the registry and can run even on eStargz-agnostic runtimes.
and even just the ability to specify the TOC format like this as a generic name-value mapping is a huge advantage stargz-snapshotter/docs/estargz.md at main · containerd/stargz-snapshotter · GitHub
APPNOTE.TXT is reasonably parseable, but the zip index is really inflexible because it’s also part of the binary file format, which sets restrictions on it for versioning and just technical parsing reasons. Understanding it requires understanding both the structure of the index as well as the encoding of the index into a serializable form.
estargz supports identifying the TOC separately from the tar itself: stargz-snapshotter/docs/estargz.md at main · containerd/stargz-snapshotter · GitHub
{
"version": 1,
"entries": [
{
"name": "bin/",
"type": "dir",
"modtime": "2019-08-20T10:30:43Z",
"mode": 16877,
"NumLink": 0
},
{
"name": "bin/busybox",
"type": "reg",
"size": 833104,
"modtime": "2019-06-12T17:52:45Z",
"mode": 33261,
"offset": 126,
"NumLink": 0,
"digest": "sha256:8b7c559b8cccca0d30d01bc4b5dc944766208a53d18a03aa8afe97252207521f",
"chunkDigest": "sha256:8b7c559b8cccca0d30d01bc4b5dc944766208a53d18a03aa8afe97252207521f"
},
{
"name": "lib/",
"type": "dir",
"modtime": "2019-08-20T10:30:43Z",
"mode": 16877,
"NumLink": 0
},
{
"name": "lib/ld-musl-x86_64.so.1",
"type": "reg",
"size": 580144,
"modtime": "2019-08-07T07:15:30Z",
"mode": 33261,
"offset": 512427,
"NumLink": 0,
"digest": "sha256:45c6ee3bd1862697eab8058ec0e462f5a760927331c709d7d233da8ffee40e9e",
"chunkDigest": "sha256:45c6ee3bd1862697eab8058ec0e462f5a760927331c709d7d233da8ffee40e9e"
},
{
"name": ".prefetch.landmark",
"type": "reg",
"size": 1,
"offset": 886633,
"NumLink": 0,
"digest": "sha256:dc0e9c3658a1a3ed1ec94274d8b19925c93e1abb7ddba294923ad9bde30f8cb8",
"chunkDigest": "sha256:dc0e9c3658a1a3ed1ec94274d8b19925c93e1abb7ddba294923ad9bde30f8cb8"
},
}
Honestly, my first thought about the TOC json is that it looks quite a bit like the json format for a simple repository API page… Simple repository API - Python Packaging User Guide
{
"meta": {
"api-version": "1.4",
"project-status": "active",
"project-status-reason": "this project is not yet haunted"
},
"name": "holygrail",
"files": [
{
"filename": "holygrail-1.0.tar.gz",
"url": "https://example.com/files/holygrail-1.0.tar.gz",
"hashes": {"sha256": "...", "blake2b": "..."},
"requires-python": ">=3.7",
"yanked": "Had a vulnerability",
"size": 123456
},
{
"filename": "holygrail-1.0-py3-none-any.whl",
"url": "https://example.com/files/holygrail-1.0-py3-none-any.whl",
"hashes": {"sha256": "...", "blake2b": "..."},
"requires-python": ">=3.7",
"dist-info-metadata": true,
"provenance": "https://example.com/files/holygrail-1.0-py3-none-any.whl.provenance",
"size": 1337
}
],
"versions": ["1.0"]
}
(And I realize PEP 658 works great because it too takes this approach to adding metadata—putting it adjacent and not inline!)