# Require packing wheels without files in the root directory
**URL:** https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599
**Category:** Packaging
**Created:** [December 2, 2022, 12:42pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599 "2022-12-02T12:42:35Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@pradyunsg](https://discuss.python.org/u/pradyunsg)
#### Post date: [December 2, 2022, 12:42pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/1 "2022-12-02T12:42:35Z")
Currently, as the wheel format is designed, files are permitted to exist in the root of the wheel. The `Root-Is-Purelib` key within the WHEEL file also indicates whether those files should end up in `purelib` or `platlib`.
Would it make sense to have a wheel 1.1 format, that enforces that contents within wheels be packed such that there’s no files in the root, and they _need_ to use the `.data/{scheme}/` directories instead?
The primary consqeuence of this would be that wheels would no longer be zipimportable – something that is considered an undesirable design consequence in the original format:
From [PEP 427 – The Wheel Binary Package Format 1.0 | peps.python.org](https://peps.python.org/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file:)
> Technically, due to the combination of supporting installation via simple extraction and using an archive format that is compatible with `zipimport`, a subset of wheel files _do_ support being placed directly on `sys.path`. However, while this behaviour is a natural consequence of the format design, actually relying on it is generally discouraged.
---
### Author:  [@pradyunsg](https://discuss.python.org/u/pradyunsg)
#### Post date: [December 2, 2022, 12:43pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/2 "2022-12-02T12:43:42Z")
For context, as it stands, the wheel format allows for wheels that do something like:
```auto
❯ unzip -l ../fancy-1.0.0-py3-none-any.whl
Archive: ../fancy-1.0.0-py3-none-any.whl
Length Date Time Name
--------- ---------- ----- ----
0 12-02-2022 12:32 root.py
0 12-02-2022 12:30 fancy-1.0.0.data/
0 12-02-2022 12:29 fancy-1.0.0.data/data/
0 12-02-2022 12:29 fancy-1.0.0.data/data/data.py
0 12-02-2022 12:29 fancy-1.0.0.data/headers/
0 12-02-2022 12:29 fancy-1.0.0.data/headers/headers.py
0 12-02-2022 12:29 fancy-1.0.0.data/platlib/
0 12-02-2022 12:29 fancy-1.0.0.data/platlib/platlib.py
0 12-02-2022 12:29 fancy-1.0.0.data/purelib/
0 12-02-2022 12:29 fancy-1.0.0.data/purelib/purelib.py
0 12-02-2022 12:30 fancy-1.0.0.data/scripts/
0 12-02-2022 12:30 fancy-1.0.0.data/scripts/scripts.py
0 12-02-2022 12:34 fancy-1.0.0.dist-info/
236 12-02-2022 12:30 fancy-1.0.0.dist-info/METADATA
457 12-02-2022 12:34 fancy-1.0.0.dist-info/RECORD
84 12-02-2022 12:30 fancy-1.0.0.dist-info/WHEEL
--------- -------
777 16 files
```
With the files being installed as the following in a MacOS venv (behaviour is same on Linux, and Windows only differs in the `Scripts` vs `bin`):
```diff
diff --git a/../out.txt b/../out2.txt
index 004644c..d7e24a1 100644
--- a/../out.txt
+++ b/../out2.txt
@@ -1,6 +1,10 @@
/tmp/venv
+/tmp/venv/ __pycache__
+/tmp/venv/ __pycache__ /data.cpython-311.pyc
/tmp/venv/bin
/tmp/venv/bin/Activate.ps1
+/tmp/venv/bin/ __pycache__
+/tmp/venv/bin/ __pycache__ /scripts.cpython-311.pyc
/tmp/venv/bin/activate
/tmp/venv/bin/activate.csh
/tmp/venv/bin/activate.fish
@@ -11,11 +15,23 @@
/tmp/venv/bin/python -> /Users/pradyunsg/.asdf/installs/python/3.11.0/bin/python
/tmp/venv/bin/python3 -> python
/tmp/venv/bin/python3.11 -> python
+/tmp/venv/bin/scripts.py
+/tmp/venv/data.py
/tmp/venv/include
/tmp/venv/include/python3.11
+/tmp/venv/include/site
+/tmp/venv/include/site/python3.11
+/tmp/venv/include/site/python3.11/fancy
+/tmp/venv/include/site/python3.11/fancy/ __pycache__
+/tmp/venv/include/site/python3.11/fancy/ __pycache__ /headers.cpython-311.pyc
+/tmp/venv/include/site/python3.11/fancy/headers.py
/tmp/venv/lib
/tmp/venv/lib/python3.11
/tmp/venv/lib/python3.11/site-packages
+/tmp/venv/lib/python3.11/site-packages/ __pycache__
+/tmp/venv/lib/python3.11/site-packages/ __pycache__ /platlib.cpython-311.pyc
+/tmp/venv/lib/python3.11/site-packages/ __pycache__ /purelib.cpython-311.pyc
+/tmp/venv/lib/python3.11/site-packages/ __pycache__ /root.cpython-311.pyc
/tmp/venv/lib/python3.11/site-packages/_distutils_hack
/tmp/venv/lib/python3.11/site-packages/_distutils_hack/ __init__.py
/tmp/venv/lib/python3.11/site-packages/_distutils_hack/ __pycache__
@@ -23,6 +39,13 @@
/tmp/venv/lib/python3.11/site-packages/_distutils_hack/ __pycache__ /override.cpython-311.pyc
/tmp/venv/lib/python3.11/site-packages/_distutils_hack/override.py
/tmp/venv/lib/python3.11/site-packages/distutils-precedence.pth
+/tmp/venv/lib/python3.11/site-packages/fancy-1.0.0.dist-info
+/tmp/venv/lib/python3.11/site-packages/fancy-1.0.0.dist-info/INSTALLER
+/tmp/venv/lib/python3.11/site-packages/fancy-1.0.0.dist-info/METADATA
+/tmp/venv/lib/python3.11/site-packages/fancy-1.0.0.dist-info/RECORD
+/tmp/venv/lib/python3.11/site-packages/fancy-1.0.0.dist-info/REQUESTED
+/tmp/venv/lib/python3.11/site-packages/fancy-1.0.0.dist-info/WHEEL
+/tmp/venv/lib/python3.11/site-packages/fancy-1.0.0.dist-info/direct_url.json
/tmp/venv/lib/python3.11/site-packages/pip
/tmp/venv/lib/python3.11/site-packages/pip/ __init__.py
/tmp/venv/lib/python3.11/site-packages/pip/ __main__.py
@@ -1233,6 +1256,9 @@
/tmp/venv/lib/python3.11/site-packages/pkg_resources/extern/ __init__.py
/tmp/venv/lib/python3.11/site-packages/pkg_resources/extern/ __pycache__
/tmp/venv/lib/python3.11/site-packages/pkg_resources/extern/ __pycache__ / __init__.cpython-311.pyc
+/tmp/venv/lib/python3.11/site-packages/platlib.py
+/tmp/venv/lib/python3.11/site-packages/purelib.py
+/tmp/venv/lib/python3.11/site-packages/root.py
/tmp/venv/lib/python3.11/site-packages/setuptools
/tmp/venv/lib/python3.11/site-packages/setuptools/ __init__.py
/tmp/venv/lib/python3.11/site-packages/setuptools/ __pycache__
@@ -1645,4 +1671,4 @@
/tmp/venv/lib/python3.11/site-packages/setuptools-65.5.0.dist-info/top_level.txt
/tmp/venv/pyvenv.cfg
-178 directories, 1467 files
+186 directories, 1485 files
```
In case you’re curious, I generated the above by:
- `tree /tmp/venv/ -f -i --sort name > out.txt`
- `pip install ../fancy-1.0.0-py3-none-any.whl`
- `tree /tmp/venv/ -f -i --sort name > out2.txt`
- `git diff --no-index -- out.txt out2.txt`
---
### Author:  [@steve.dower](https://discuss.python.org/u/steve.dower)
#### Post date: [December 2, 2022, 1:04pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/3 "2022-12-02T13:04:47Z")
Counter-argument: [bottle · PyPI](https://pypi.org/project/bottle/)
This is probably better handled by tools that red-flag packages that install names other than what their own name is. There are plenty of good reasons to do this, and I don’t see why we should disallow it, but there are also good reasons why installers should _know_ that wheels are doing this.
---
### Author:  [@pradyunsg](https://discuss.python.org/u/pradyunsg)
#### Post date: [December 2, 2022, 1:12pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/4 "2022-12-02T13:12:39Z")
I don’t understand the counter argument. Could you elaborate on that?
```auto
❯ unzip -l bottle-0.12.23-py3-none-any.whl
Archive: bottle-0.12.23-py3-none-any.whl
Length Date Time Name
--------- ---------- ----- ----
151823 08-03-2022 13:39 bottle.py
151810 08-03-2022 13:42 bottle-0.12.23.data/scripts/bottle.py
1308 08-03-2022 13:42 bottle-0.12.23.dist-info/AUTHORS
1061 08-03-2022 13:42 bottle-0.12.23.dist-info/LICENSE
1836 08-03-2022 13:42 bottle-0.12.23.dist-info/METADATA
92 08-03-2022 13:42 bottle-0.12.23.dist-info/WHEEL
7 08-03-2022 13:42 bottle-0.12.23.dist-info/top_level.txt
643 08-03-2022 13:42 bottle-0.12.23.dist-info/RECORD
--------- -------
308580 8 files
```
The proposal is to have:
```auto
bottle-0.12.23.data/purelib/bottle.py
bottle-0.12.23.data/scripts/bottle.py
```
instead of the current:
```auto
bottle.py
bottle-0.12.23.data/scripts/bottle.py
```
---
### Author:  [@steve.dower](https://discuss.python.org/u/steve.dower)
#### Post date: [December 2, 2022, 1:24pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/5 "2022-12-02T13:24:16Z")
And the question then is, why? The only consequence, as you point out, is to break a discouraged scenario without any apparent benefit. It also complicates “normal/cheap” installation, which currently lets you extract a set of wheels and reference the directory from `sys.path`.
Feels like churn for build backends, installers, and users, for no real reason.
---
### Author:  [@sinoroc](https://discuss.python.org/u/sinoroc)
#### Post date: [December 2, 2022, 1:55pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/6 "2022-12-02T13:55:05Z")
Maybe helpful: [https://inspector.pypi.io/project/bottle/0.12.23/packages/63/96/2713bb6e0ca10ee2b7be568d6e6d112cf0a1b604e14879fe744f48ebbed6/bottle-0.12.23-py3-none-any.whl/](https://inspector.pypi.io/project/bottle/0.12.23/packages/63/96/2713bb6e0ca10ee2b7be568d6e6d112cf0a1b604e14879fe744f48ebbed6/bottle-0.12.23-py3-none-any.whl/)
---
### Author:  [@njs](https://discuss.python.org/u/njs)
#### Post date: [December 2, 2022, 3:00pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/7 "2022-12-02T15:00:59Z")
Before we get into counter-arguments, can we hear the argument? 🙂 the initial post doesn’t seem to mention any benefits from the change…
---
### Author:  [@dholth](https://discuss.python.org/u/dholth)
#### Post date: [December 2, 2022, 3:08pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/8 "2022-12-02T15:08:30Z")
I don’t consider zip import-ability to be undesirable; just not guaranteed to work. Why not get extra features for free.
The magic of the root directory is that most wheels don’t need a .data/ folder at all.
This is also good for file size since the zip filenames are not compressed.
If we did go to nested archives, those filenames get compressed by the nesting.
---
### Author:  [@fungi](https://discuss.python.org/u/fungi)
#### Post date: [December 2, 2022, 3:39pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/9 "2022-12-02T15:39:38Z")
Setting aside the exact container formats, this is conceptually
reminding me a lot of Debian’s deb packages. For those unfamiliar
with the design, it’s a GNU ar format archive which contains
multiple compressed tarballs of separate file trees (each of which
can use different compression algorithms too):
```auto
$ ar t util-linux_2.29-1_amd64.deb
debian-binary
control.tar.gz
data.tar.xz
```
That model dates back to 1995, prior to which Debian just
concatenated a pair of tarballs into a file with no real outer
container. Of course, there is probably much older prior art for
similar designs, just pointing out that the idea of nested archives
isn’t remotely novel or untested for package management.
> **[deb (file format)](https://en.wikipedia.org/wiki/Deb_%28file_format%29)**
>
> deb is the format, as well as extension of the software package format for the Debian Linux distribution and its derivatives.
> Debian packages are standard Unix ar archives that include two tar archives. One archive holds the control information and another contains the installable data.
> dpkg provides the basic functionality for installing and manipulating Debian packages. Generally end users don't manage packages directly with dpkg but instead use the APT package management software or other AP...
---
### Author:  [@pf\_moore](https://discuss.python.org/u/pf_moore)
#### Post date: [December 2, 2022, 8:38pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/10 "2022-12-02T20:38:24Z")
I agree. I don’t see enough benefits to justify the disruption. And having the “main” files in the distribution at the root of the zipfile seems natural to me.
Definitely not worth spending any of our limited “churn budget” on, IMO.
---
### Author:  [@FFY00](https://discuss.python.org/u/FFY00)
#### Post date: [December 6, 2022, 10:37am UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/11 "2022-12-06T10:37:43Z")
One argument in favor is that the current format works on limited use-cases, and it’s probably a bad UX for users to find this when trying to use it based on very reasonable expectations. I think removing this confusing and unintended feature would provide a better UX in isolation, but having things that currently work and users might be relying on suddenly stop working will probably have much more of a negative effect in the UX than what we would gain from this change.
This is definitely something we should keep in mind when working on a new major version of the spec though.
---
### Author:  [@dholth](https://discuss.python.org/u/dholth)
#### Post date: [December 6, 2022, 2:59pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/12 "2022-12-06T14:59:01Z")
I think it is generally important to build an open system that winds up being more powerful than expected, instead of trying to limit people from using that system in unintended ways. If we were revising the language of the specification we might say something like “use this feature if you control the contents of the wheel”…
---
### Author:  [@pradyunsg](https://discuss.python.org/u/pradyunsg)
#### Post date: [December 6, 2022, 6:32pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/13 "2022-12-06T18:32:53Z")
I guess I should’ve added “why do this” in the OP.
tl;dr: The technical changes are “cheap” and it’d be a good thing to see how a new WHEEL version rolls out. 🙂
* * *
The basic _technical_ motivation here is simplifying the wheel unpacking logic and closing the “oh, you can do this thing that works but don’t because it’s fragile” loophole in the format (the loophole of `python foo-vvv-py3-none-any.whl/foo` running `foo. __main__ ` from the wheel’s zip). With this change, you can tell _exactly_ where each file in a wheel will end up since it makes it so that the format doesn’t treat any scheme “specially” – everything ends up a `{name}-{version}.data/scheme/{path}`, which makes introspecting the file a bit easier as well. As mentioned, it closes
In all honesty, I don’t think these are strong reasons on their own. Installers will already read the WHEEL file unconditionally, so bumping the WHEEL version for this doesn’t really add much value. 🙂
The more compelling reasons for me is that (1) it gives us an opportunity to see how build-backends/packagers respond to a new WHEEL version (albeit, a minor one) which can help inform how to go about rolling out a major version bump of WHEEL in the future and (2) it’s easier to block WHEEL 1.0 uploads in the future on PyPI; if we choose to do so at some point.
* * *
I’ll note a couple of opinionsTM of mine:
- This, on its own, is a backwards-compatible change for installers (hence 1.1). Every wheel 1.0 installer will correctly install a wheel 1.1 wheel (albeit, with a warning).
- The argument that “oh, it’ll break someone who relies on doing something we explicitly discourage and we shouldn’t do that” isn’t a good one IMO.
---
### Author:  [@layday](https://discuss.python.org/u/layday)
#### Post date: [December 6, 2022, 7:05pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/14 "2022-12-06T19:05:24Z")
Well, who’d we be simplifying the wheel unpacking logic for? Surely not pip or installer – a corporate wheel unpacker if one such exists? Otherwise, what’s the time frame we are looking at here, optimistically a decade before you are able to stop parsing 1.0 wheels, assuming that all build backends follow suit? And that would involve sunsetting the 1.0 spec, emitting warnings and all the rest. I think whatever simplification’s to be gained from removing this (very minor) quirk long-term will be negated by everything else that’s required to make it happen.
As for build backends, I think they are bound to respond very differently to a backward-compatible minor version bump than they will to an all-new wheel spec. I imagine that should there be a v2 wheel, build backends will opt to generate both v1 and v2 wheels during the transitory period, for example.
---
### Author:  [@pradyunsg](https://discuss.python.org/u/pradyunsg)
#### Post date: [December 6, 2022, 7:11pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/15 "2022-12-06T19:11:51Z")
> [@layday](#):
>
> Well, who’d we be simplifying the wheel unpacking logic for?
I do literally say:
> Installers will already read the WHEEL file unconditionally, so bumping the WHEEL version for this doesn’t really add much value. 🙂
> [@layday](#):
>
> As for build backends, I think they are bound to respond very differently to a backward-compatible minor version bump than they will to an all-new wheel spec. I imagine that should there be a v2 wheel, build backends will opt to generate both v1 and v2 wheels during the transitory period, for example.
/me nods.
Rather than what each of us thinks, I do think it’s a stronger statement if we actually see this happen though. 🙂
---
### Author:  [@pf\_moore](https://discuss.python.org/u/pf_moore)
#### Post date: [December 6, 2022, 7:36pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/16 "2022-12-06T19:36:45Z")
Thanks for clarifying. With that background, the argument is a little more persuasive.
I still think this is too small of a change to be worthwhile. It seems extremely likely to me that build backends simply won’t bother implementing the new version, as there’s no practical benefit for them. And frontends will simply change the “valid version” check to say that 1.1 is supported. Sure, frontends could check for files in the root and raise an error if they are present, but again, why bother? “Be lenient in what you accept” argues against doing it, and honestly I don’t think it’s the installer’s place to police the validity of wheels beyond what’s necessary to be sure they can be installed correctly.
> [@pradyunsg](#):
>
> (1) it gives us an opportunity to see how build-backends/packagers respond to a new WHEEL version (albeit, a minor one) which can help inform how to go about rolling out a major version bump of WHEEL in the future
I understand the value of getting a feel for how a new wheel version would go, but it’s the case of a _backward incompatible_ wheel that’s the concern, and I don’t think we gain enough by trying a practice run with a compatible change.
> [@pradyunsg](#):
>
> (2) it’s easier to block WHEEL 1.0 uploads in the future on PyPI; if we choose to do so at some point.
Why? I don’t see how having a 1.1 version makes it any easier to block 1.0. Or indeed why blocking 1.0 and _not_ blocking 1.1 would be of any value anyway.
> [@pradyunsg](#):
>
> I’d really like us to better understand how a new version of wheel would roll out, and not try an incompatible version bump and _hope_ that we get the rollout right on the first attempt based on educated guesses.
I absolutely agree that this is a worthwhile goal. I just don’t see the proposed change giving us useful information on that front.
Surely there are other formats that have gone through backward incompatible changes (maybe outside the packaging ecosystem, or even in non-Python areas). Would we be better served looking for advice from the people involved in those transitions?
---
### Author:  [@dholth](https://discuss.python.org/u/dholth)
#### Post date: [December 6, 2022, 7:37pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/17 "2022-12-06T19:37:21Z")
In some wheel unpacking code I wrote, the installer maps category name e.g. ‘purelib’, ‘platlib’ → list of prefixes inside the zip file → {category: [list of files]} → installed locations in other words the added complexity of “empty prefix maps to purelib depending on WHEEL” is about two lines of code, and does not spread to any other part of the installer.
---
### Author:  [@dholth](https://discuss.python.org/u/dholth)
#### Post date: [December 6, 2022, 7:46pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/18 "2022-12-06T19:46:42Z")
In conda, we have .tar.bz2 and our own .conda format which is two nested .tar.zst’s (one for metadata). We continue to provide both, in a year or so we will be able to stop providing the .tar.bz2’s as the support runs out on the last version of the conda installer that didn’t support .conda packages.
If we decided to go the other direction the only thing we need to add to wheel to make a warranted executable feature would be to provide a way for the ` __main__.py` to be omitted from installations - python somefile.whl would run the ` __main__.py` which could figure out anything related to sys.path etc, while installs would not put a ` __main__.py` in the site-packages/ root.
---
### Author:  [@pf\_moore](https://discuss.python.org/u/pf_moore)
#### Post date: [December 8, 2022, 12:36pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/19 "2022-12-08T12:36:27Z")
> [@pf\_moore](#):
>
> I still think this is too small of a change to be worthwhile.
If you want to propose a “Wheel 1.1” spec, that serves both as a useful improvement, _and_ as a vehicle to get experience for upgrading the wheel spec, how about someone puts together a proposal for supporting symlinks in wheels? That would be a useful addition in itself (there’s a request for it [here](https://github.com/pypa/wheel/issues/453) and it would be another approach that editable wheels could use), while still giving all the benefits you’re after. Also, as 1.1 wheels would have a capability that 1.0 wheels don’t support, we’d get some experience with the transition questions you’re concerned about.
---
### Author:  [@pradyunsg](https://discuss.python.org/u/pradyunsg)
#### Post date: [December 8, 2022, 6:59pm UTC](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599/20 "2022-12-08T18:59:12Z")
> [@pf\_moore](#):
>
> how about someone puts together a proposal for supporting symlinks in wheels?
That won’t be backwards compatible tho.
Wheel 1.0 installers won’t be able to install a wheel containing symlinks correctly.
[Next page](https://discuss.python.org/t/require-packing-wheels-without-files-in-the-root-directory/21599.md?page=2)