# Moving all stdlib packages into wheels **URL:** \ **Category:** Ideas\ **Created:** [July 19, 2019, 11:13pm UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056 "2019-07-19T23:13:22Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [July 19, 2019, 11:13pm UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/1 "2019-07-19T23:13:22Z")
I wrote a little thing that builds wheels for each top level name or directory in cpython’s Lib/ . It is simple but could be augmented with dependency declarations and more sophisticated splitting. If developed further a special installer might build your custom Python standard library with desired batteries. It would be possible to add metadata about which are pure python and which would need a different implementation for an alternative interpreter. [https://github.com/dholth/nonstdlib](https://github.com/dholth/nonstdlib) ---
**Author:** ![flavianhautbois](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/flavianhautbois/32/1219_2.png) [@flavianhautbois](https://discuss.python.org/u/flavianhautbois)\ **Post date:** [July 28, 2019, 5:00pm UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/2 "2019-07-28T17:00:27Z")
@dholth I really like this approach ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [July 28, 2019, 6:36pm UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/3 "2019-07-28T18:36:29Z")
Thanks. Suppose we get rid of the standard library entirely. `python --nonstd`. Gets you whatever Python requires to boot. Brutally minimal. Now distribute CPython with a default, named (virtual?) environment that includes the standard library. When you are running the Python interpreter as an application, rather than as an application runtime, you get that. Useful libraries an `import` statement away. Include individually wrapped (packaged) standard library modules. Wheels, long sys.path-style folder-per-module, importlib hooks so that they are not importable by default, whatever. Since the individually wrapped modules are distributed with CPython, they can be added to a new Python environment instantly without any of that pesky internet access. Start moving individually wrapped modules from the “importable by default” set to the “must be declared explicitly to be importable” set. As a bonus could it become possible to remove the `virtual` from `virtualenv` because there was no big, system-specific default environment to overcome? In this way a library that is a former- or soon-to-be- member of the standard library can be special without being importable by default. ---
**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\ **Post date:** [July 29, 2019, 9:01am UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/4 "2019-07-29T09:01:53Z")
I think this should be moved/split off to another thread, but I don’t know how to do that in Discourse. (Does it need an administrator?) > [@PEP 594: Removing dead batteries from the standard library](https://discuss.python.org/t/pep-594-removing-dead-batteries-from-the-standard-library/1704/57): > > I wrote a little thing that builds wheels for each top level name or directory in cpython’s Lib/ . It is simple but could be augmented with dependency declarations and more sophisticated splitting. Getting the dependency declarations right – and maintaining them – won’t be easy. For starters, it turns out we don’t currently have a canonical list of modules in the standard library. (Building Python and looking doesn’t count; you’d need to do it on all platforms/configurations. Docs don’t count either; there are plenty of undocumented internal modules like `_socket`.) If you’d like to help with creating such a list, let me know. It would help with PEP 534, which aims to split the stdlib into “mandatory” and “optional” parts. (That could also be a first step toward a more modular stdlib.) ---
**Author:** ![njs](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/njs/32/204_2.png) [@njs](https://discuss.python.org/u/njs)\ **Post date:** [July 29, 2019, 9:48am UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/5 "2019-07-29T09:48:10Z")
> [@encukou](#): > > I think this should be moved/split off to another thread, but I don’t know how to do that in Discourse. (Does it need an administrator?) I’m not sure what permissions are needed, but I split it off anyway. (If you do have permissions, then you do it by clicking the wrench icon on the right side of the thread view, then clicking “select posts”.) ---
**Author:** ![tiran](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tiran/32/48_2.png) [@tiran](https://discuss.python.org/u/tiran)\ **Post date:** [July 29, 2019, 9:58am UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/6 "2019-07-29T09:58:22Z")
Thanks, I requested a split-off by flagging the first post yesterday. I don’t have permission to split off the thread myself. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [July 29, 2019, 12:35pm UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/7 "2019-07-29T12:35:01Z")
So far the split is based on the names in the un-built Lib/ directory. The next step will be to generate dependencies between modules based on static analysis. [https://github.com/dhellmann/python-stdlib-dependencies](https://github.com/dhellmann/python-stdlib-dependencies) is promising. This kind of dependency metadata would be re-built automatically with SCons whenever Lib/ changed. I also considered subtracting sys.modules (before import x) from sys.modules (after import x). For the use case of subsetting Python the goal would be to generate the list of required dependencies instead of an exhaustive list. After that you’d want to keep a folder with diffs or additions to be combined with the automatically discovered metadata. For example a rule to package a module together with its internal underscore-prefixed implementation. Or a link to the docs. Longer term Lib/ itself could be a build product created from packages. If you wanted it to look as it does today, have an installer that puts the \*.dist-info in a separate metadata folder to keep track of what’s installed so far without adding clutter. We’ve found this similar effort in another discussion. They split the stdlib but not into wheels. Need a pull request with this against nonstdlib. [https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/python/python3/python3-manifest.json](https://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/python/python3/python3-manifest.json) ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [August 1, 2019, 6:59pm UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/8 "2019-08-01T18:59:07Z")
> [@Re-use of standard library across implementations](https://discuss.python.org/t/re-use-of-standard-library-across-implementations/2051/16): > > You’ll never get enough of the ecosystem to declare their dependencies of specific parts of the stdlib. I am sympathetic to your problem, but the solution should not require any changes for existing Python users (as long as they aren’t relying on the physical layout of the stdlib in the filesystem – games with the default sys.path are to some extent acceptable). And neither is a solution feasible that provides backwards compatibility for some time while deprecating current habits (again excluding reliance on filesystem layout). It’s a little embarrassing but we did get everyone to stop using the distutils features that don’t package very well in wheels; I guess it was easier for them than contributing to packaging; egg did a similar feat years before. So I think it is doable for better or for worse. A game developer will want to redistribute your library. They will submit a pull request “please include this bit of extra metadata”, and so it goes. The payoff would be better support for Python applications as opposed to using the Python interpreter as an application. I think most of the extra metadata would be the responsibility of the application developer instead of the library developers. Imagine something like py2exe but during the entire development process instead of at the end. When you run your application, the subset of Python you’re not planning to depend upon would be hidden. Then you would be continuously ready to redistribute however much of Python when it is time to share the program. The imagined tool would be optional, like virtualenv. Since you might already get a subset of the standard library if you are using a Linux distribution, it could be surprisingly helpful to make it official. ---
**Author:** ![guido](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/guido/32/21_2.png) [@guido](https://discuss.python.org/u/guido)\ **Post date:** [August 2, 2019, 4:45am UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/9 "2019-08-02T04:45:27Z")
I’m sorry, I find it hard to follow what you are writing, jumping from things that happened in the past (I don’t recall the details and you don’t provide them) to hypotheticals. But I realized I don’t mind as long you require the extra work only for apps who want to use your accelerated Python implementation. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [August 2, 2019, 10:57am UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/10 "2019-08-02T10:57:07Z")
Who will produce a version of pip that can operate in an environment that doesn’t contain the stdlib? It could add the stdlib to sys.path to run, but only search for dist-info on the original paths. ---
**Author:** ![njs](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/njs/32/204_2.png) [@njs](https://discuss.python.org/u/njs)\ **Post date:** [August 2, 2019, 8:26pm UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/11 "2019-08-02T20:26:27Z")
I like the energy and goals here, but I think a more incremental approach (like Ruby did) is more likely to be acceptable to other devs and users, and therefore be more effective in the end. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [August 2, 2019, 9:16pm UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/12 "2019-08-02T21:16:02Z")
I’m interested in the extreme case. The necessary artifacts will be useful for regular users too. A no-standard-lib pip is also a not-reinstalled-in-each-venv pip. ---
**Author:** ![pradyunsg](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pradyunsg/32/206_2.png) [@pradyunsg](https://discuss.python.org/u/pradyunsg)\ **Post date:** [August 24, 2019, 3:16am UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/13 "2019-08-24T03:16:51Z")
> [@dholth](#): > > A no-standard-lib pip is also a not-reinstalled-in-each-venv pip. Hmm… pip currently gets information about _where_ to install, from the running interpreter and environment. Changing this up, to get information from an independent environment, will likely require more work than just vendoring stdlib stuff in or whatnot. Unless, you’re suggesting having pip sit outside site-packages or something to that effect. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [November 4, 2019, 2:24pm UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/14 "2019-11-04T14:24:14Z")
poetry does it. In fact it seems to implement most of the ideas required for a bold new packaging vision. It presents as a standalone application that manages the virtualenv applications wind up in, and it manages its own dependencies in a novel way. ---
**Author:** ![uranusjr](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/uranusjr/32/103_2.png) [@uranusjr](https://discuss.python.org/u/uranusjr)\ **Post date:** [November 7, 2019, 5:47pm UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/15 "2019-11-07T17:47:58Z")
There’s nothing technically unviable to implement a package manager that runs outside of an environment, but the amount of modification to transform pip into that is so much that you’d basically rewrite the whole thing. I’d say it’s better to start fresh if you really want that (I would love such an implementation FWIW), rather than trying to reshape a square peg that is pip into the out-of-env round hole. ---
**Author:** ![dholth](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dholth/32/952_2.png) [@dholth](https://discuss.python.org/u/dholth)\ **Post date:** [November 7, 2019, 6:06pm UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/16 "2019-11-07T18:06:54Z")
You would need a stub that added itself to PYTHONPATH to run under the target environment. It could be as much as running the whole pip under the target Python but not leaving it there, or something that reported back only with the dependencies and paths in the target env. There’s already been a lot of work getting pip to do isolated builds. It might be easier than you think. ---
**Author:** ![Hasnat](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hasnat/32/1526_2.png) [@Hasnat](https://discuss.python.org/u/Hasnat)\ **Post date:** [December 26, 2019, 6:12pm UTC](https://discuss.python.org/t/moving-all-stdlib-packages-into-wheels/2056/17 "2019-12-26T18:12:24Z")
You are right. If it gets information from independent environment instead of running than it may create more issues for virtual env users.