Okay, I would like to submit a PEP for the idea of categorizing the standard library into subfolders. Is there a core developer who would like to sponsor me on this PEP?
First hi all , and pardon me if i mess up with post rules since iām new to this. Iām also a portability āextremistā so i donāt expect people to even like my position.
You can even go a bit further and say : does the the python standard library belong to the CPython implementation for supported platforms ?
cpython itself canāt even load its own library on android or wasm, an obvious example is āasyncioā pulling threading and multiprocessing that are forbidden on WASI/WASM
it can get not only hard to get a stdlib for a new VM but for porting cpython itself too ā¦
Thanks for all the past and future platforms what will never get decent cpython support then. (Pardon me but i will really laugh out loud if micropython node makes it in visual code before cpython)
indeed, thatās exactly why i love micropython VM attitude, C modules are just forbidden to enter core or platform support (called ports) until itās proven to be really not possible to have a pure python counterpart FIRST even a very slow one.
The stdlib - really - bored me both on Android kitkat and ASM.js - especially asyncio - ( and still is on WASM) and thatās why i embraced micropython and itās asyncio ālightā. For writing python tools on win32 i use MSYS2 cpython3 and canāt wait for midipix to be released since i donāt even want to hear of msvc stdlib āfeaturesā anymore when i just want to open ā¦ an UART in 2018!.
i donāt have time to lose for that i want to have fun, not go back in my old dayjob (useless) porting nightmares.
Alan since you want my opinion, iād say any stdlib not able to run on a WASI polyfill or strict POSIX ( including aio but without pthread ) is of very low interest for the future (yes iām looking at you win32 ).
aio:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/aio.h.html
wasi:
pure python stdlibs are wanted:
https://github.com/pmp-p/pydk/tree/master/sources.em uses a pure python stdlib and threadless asyncio.
We decided some time ago that CPython would require threading support from the OS. This eases maintenance a lot for us. You may take a different stance based on which platforms you are interested in, and itās fine to use (or even write) a different implementation for that purpose. But I think weāre unlikely to switch back to make threading support optional.
Hmm, so what? Personally I donāt care whether Visual Code (I assume youāre talking about the text editor) has a āmicropython nodeā or whatever else. I donāt think youāll find any CPython core developer who gets frustrated at the idea that other implementations may be selected for specific use casesā¦
Well, thatās your opinion about the future 
Other opinions are valid. For example, personally, I think a platform that doesnāt offer any kind of threading support isnāt very interesting as a general-purpose computing platform.
Well, a bunch of people have been saying that for many years, and it still seems to have trouble happening. Perhaps thereās a reason why. Perhaps itās not that easy, and thereās a scarcity of people willing to do the necessary hard work. But pressuring CPython developers to do it for you wonāt work either.
Sometimes we may rewrite C code as Python code when it helps make the code better-architected, more reasonable to maintain, evolve and think about (e.g. importlib or zipimport). However, in many cases, there is no such concern, and rewriting C code that works fine and thatās performant into pure Python is not our priority.
You may find other projects that have priorities more aligned with yours. Good, so why not use those projects and contribute to them? At the moment, you sound like someone who complains to the Linux developers that they didnāt choose the same priorities as NetBSDā¦
1 Like
Yeah, as i said at the top āi donāt expect people to even like my positionā.
But just to be clear as OP is talking of an STDLIB written in Python not cpython written in C and C dependants stdlibs bits.
Sorry but Python3 is Python3 to me i donāt care about VM implementation l leave that to core dev of rust / java / javascript / C / C++ / whatever the VM is.
for my few needs, i can (try to) write my own preemtible stackless vm instead of going complaining in bpo/ideas/else ( but i love to rant loud on IRC some french people know that already).
So far as iām concerned i use cpython3.7+ when i need it for Panda3d and it does its job to the perfection *
helloworld.c give same result on GNU/Linux or any BSD flavour or wasi platform/ internet browser, but yeah even with my poor C skills i know thatās more about stdio and not C stdlib.
So - and i mean no disrespect, itās just light mood criticism - please letās build more cpython toolchains and transpilers for each platform available to have the best optimized low level stdlibs ever with handcrafted bleeding edge Python opcodes pulling direct syscalls no other VM can afford paid dev. to implement.
indeed, iām aligned with people who want to write and SHARE Python portable code. not put cpython non portable code to the freezer**.
I love all pythons vm equally, but it would be really nice if they could live in the same stdlib.
- on supported platforms.
** except for supported platforms.
@pmp-p I edited your post to remove some inappropriate comparisons, without changing the meaning. Please keep in mind that this forum has a diverse international audience, and keep things professional. Thanks!
2 Likes
@njs, thanks for clarification iāll try to keep that in mind 
Usually, how is this writing effort coordinated? Do you recommend some online tool?
And: is ok to start writing or is advisable to wait for a core developer sponsorship before start a draft?
Personally, Iād like to see some more detail before sponsoring this. I think writing up a draft so people can understand your approach and its tradeoffs would be important in this case.
1 Like
People typically coordinate on GitHub via a fork of the peps repo.
Depends on what it takes to get a sponsor. Do make sure to read PEP 1 ā PEP Purpose and Guidelines | peps.python.org which outlines the PEP process.
No one has spoken up to say that they would use a pure Python _codecs module in their VM which was my key question, so for that module it isnāt sounding like itās worth the effort to do, especially as an initial effort to fill in more of the stdlib with pure Python code.
You tell me. This is more of a question to help scope the amount of work left to potentially go back and fill in the gaps of the stdlib where thereās no comparable pure Python code but it functionally could. Since Iām not going to be doing that work it really isnāt for me to answer whatās needed (I already wrote PEP 399 and importlib so I did my bit in trying to make this sort of stuff easier 
).
Thanks a lot for that.
In in the weekend I started hacking something to divide what modules are Pure and what ones are not. However, giving a module, I am still wondering how to force it to skip the C-based version and go for the Pure only. Some tip on this would be appreciated
I would use on Grumpy at least. This and everything Pure I could find. At least until got enough compatibility to have real useful stuff running, even very very slow.
And @freakboy3742 said they would use as a compliance test for optimized versions, IIUC.
My question is: If someone (like me) contributes a pure subset having what Guido outlined, would this be included in the stdlib and maintained as PEP399 described?
If you want to force imports of C code to fail then you can put None in sys.modules for a module and it will cause ModuleNotFoundError to be raised.
Thatās probably a steering council question unless you can get consensus around the idea from the other core devs as it is signing the core devs up for more work to maintain that new code.
But this would allow a Pure version of the same module to be loaded?
In the stdlib there isnāt a name clash for any of the modules that have Python and C code, so no, it wonāt prevent it from happening (for instance see cpython/Lib/datetime.py at cf9360e524acafdce99a8a1e48947fd7da06f3d4 Ā· python/cpython Ā· GitHub).
I see. Then I am getting inclined to produce that list of C-only modules, put here to understand what ones are consensual to be included and start by this ones.
Okā¦ There is no _datetime.py when a _datetime.so is produced. Thanks for the info.
I think the order of preference is in importlib._bootstrap_external._get_supported_file_loaders. If both a source and extensions loader is available, the extensions loader will be preferred.
There are two shortcomings for a pure-python _codecs module, I think these principles generalize to many of the c-extension stdlib modules. At some point it has to make system calls to unicodedb and locale-based encode-decode routines. And I think it also requires access to implementation-specific low-level routines to manipulate partially constructed PyUnicodeObjects (or equivalent).
1 Like
Preliminar summary of modules count by loader, after importing all the stdlib modules listed by stdlib_list, without filtering:
Modules by loader:
Items: 30 Loader: <class '_frozen_importlib.BuiltinImporter'>
Items: 6 Loader: <class '_frozen_importlib.FrozenImporter'>
Items: 411 Loader: <class '_frozen_importlib_external.SourceFileLoader'>
Items: 51 Loader: <class '_frozen_importlib_external.ExtensionFileLoader'>
Items: 8 Loader: <class 'AttributeError'>
This sums 30+6+51+8 = up to 95 modules non-pure, of 506 imported (minus the stdlib_list ones).
Excluding the ones raising AttributeError when trying to grab its loader and FrozenImporter that seems to be about the import system, we have 81 C-based modules:
Loader: <class '_frozen_importlib.BuiltinImporter'>
_abc
_ast
_codecs
_collections
_functools
_imp
_io
_locale
_operator
_signal
_sre
_stat
_string
_symtable
_thread
_tracemalloc
_warnings
_weakref
atexit
builtins
errno
faulthandler
gc
itertools
marshal
posix
pwd
sys
time
zipimport
Loader: <class '_frozen_importlib_external.ExtensionFileLoader'>
_asyncio
_bisect
_blake2
_bz2
_contextvars
_crypt
_csv
_ctypes
_curses
_curses_panel
_datetime
_dbm
_decimal
_elementtree
_hashlib
_heapq
_json
_lsprof
_lzma
_multiprocessing
_opcode
_pickle
_posixsubprocess
_queue
_random
_scproxy
_sha3
_socket
_sqlite3
_ssl
_struct
_tkinter
_uuid
array
audioop
binascii
cmath
fcntl
grp
math
mmap
nis
parser
pyexpat
readline
resource
select
syslog
termios
unicodedata
zlib
I can start digging what already have a pure version somewhere, what is ok to have a pure version and what shouldnāt/canāt have one.
1 Like
We had a discussion at the core dev sprint and I think we would be willing to entertain a PR that implements a pure Python implementation of a module to start. So if you want to propose parts of _codecs in pure Python we can have a look and see how this goes (if you want to propose a different module to start then let us know and we can discuss if that module makes sense). Obviously we are after something that is actually being used and thus of high-quality. Tests are also important. And another part of ābeing usedā is to do it for things that actually will have use, e.g. implementing a UTF-8 encoder is a bit silly since basically every platform has that natively implemented in some fashion.
1 Like