I’m all for it, but how would you write that? We could try to import all stdlib (sub)modules like my quick script did, but I’m not sure how to detect “side effects” programatically… what I saw manually was some text output (this) and program/tab opening (antigravity, idlelib.idle), do we can / want to detect that in a unit test?
I had the same concern when we hardcoded some submodules in the completer (those that could not be detected by pkgutil, see PR), I tend to think that both situations concern rare edge-cases that are not expected to happen in new code.
I would think it’s easier to go about it by maintaining two lists, modules that should be possible to import autocomplete, and modules with known side effects that shouldn’t be, and then have automation around detecting new modules that aren’t on one or the other (ideally, not blocking a PR for something like this), and sticking to only importing the known good rather than any not listed as having sideffects.
If we don’t test for this then we should at least write down guidelines on what to do when adding a new module to the stdlib, including avoiding side-effects upon import, otherwise add the module to the blocklist.
Any / arbitrary side effects is hard to test for. Ones which we commonly see such as “Does not print” should be more straightforward to do. There’s at least one test which imports every module to make sure “everything is importable” (cpython/Lib/test/test___all__.py at main · python/cpython · GitHub). There’s also tests that running the interpreter only outputs something specific that is expected.
Combining those should be able to test “There is no unexpected printing in any which autocomplete will list”. There may be some work to handle things like deprecation warnings. I suspect test___all__.py will be a good guide for that.
I don’t know how to make sure nothing opens a UI which I think would also be useful to check… In the test suite there is a gui resource which might help guide but that is more of a manual tag.
Thanks for the pointers! I just added a test leveraging test.test___all__, checking for prints to stdout/stderr, the subprocess.Popen audit event (covers webbrowser.open), and tkinter.Tk creation.