Pydoc -k usability (or lack thereof)

I can’t be the only one who’s noticed that pydoc -k has become near-unusable as of late, due to all of the tracebacks and warnings that various packages emit when they’re scanned. Is there anything that can be done to restore the tool’s usefulness?

Here’s what e.g. pydoc3 -k schema looks like on my current system (Fedora 32 with mostly distro-packaged modules installed in /usr/lib64/python3.8/site-packages/):

$ pydoc3 -k schema
PyInstaller.hooks.hook-jsonschema 
2596 INFO: NumExpr defaulting to 4 threads.
5740 INFO: ENTER ['pretty_getcwd']()
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib64/python3.8/logging/__init__.py", line 1081, in emit
    msg = self.format(record)
  File "/usr/lib64/python3.8/logging/__init__.py", line 925, in format
    return fmt.format(record)
  File "/usr/lib64/python3.8/logging/__init__.py", line 664, in format
    record.message = record.getMessage()
  File "/usr/lib64/python3.8/logging/__init__.py", line 369, in getMessage
    msg = msg % self.args
TypeError: not enough arguments for format string
Call stack:
  File "/usr/bin/pydoc3", line 5, in <module>
    pydoc.cli()
  File "/usr/lib64/python3.8/pydoc.py", line 2694, in cli
    apropos(val)
  File "/usr/lib64/python3.8/pydoc.py", line 2186, in apropos
    ModuleScanner().run(callback, key, onerror=onerror)
  File "/usr/lib64/python3.8/pydoc.py", line 2135, in run
    for importer, modname, ispkg in pkgutil.walk_packages(onerror=onerror):
  File "/usr/lib64/python3.8/pkgutil.py", line 92, in walk_packages
    __import__(info.name)
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3.8/site-packages/FedoraReview/__init__.py", line 24, in <module>
    from .mock import Mock
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3.8/site-packages/FedoraReview/mock.py", line 29, in <module>
    import mockbuild.util
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3.8/site-packages/mockbuild/util.py", line 1733, in <module>
    ORIGINAL_CWD = pretty_getcwd()
  File "/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py", line 101, in trace
    doLog(l2, logging.INFO, filename, lineno,
  File "/usr/lib/python3.8/site-packages/mockbuild/trace_decorator.py", line 39, in doLog
    logger.handle(logger.makeRecord(logger.name, level, *args, **kargs))
Message: 'LEAVE %s --> %s\n'
Arguments: ['pretty_getcwd', '/home/ferd']
5793 INFO: OpenGL_accelerate module loaded
INFO: OpenGL_accelerate module loaded
5800 INFO: Using accelerated ArrayDatatype
INFO: Using accelerated ArrayDatatype
5980 INFO: Failed to load library ( 'libgle.so.0' ): libgle.so.0: cannot open shared object file: No such file or directory
INFO: Failed to load library ( 'libgle.so.0' ): libgle.so.0: cannot open shared object file: No such file or directory
ZConfig - Structured, schema-driven configuration library.
ZConfig.components.logger - ZConfig schema component package for logging configuration.
ZConfig.info - Objects that can describe a ZConfig schema.
ZConfig.loader - Schema loader utility.
ZConfig.schema - Parser for ZConfig schemas.
ZConfig.schemaless - Support for working with ZConfig data without a schema.
7238 INFO: Running [1] lvm version ...
INFO: Running [1] lvm version ...
7258 INFO: stdout[1]:   LVM version:     2.03.09(2) (2020-03-26)
  Library version: 1.02.171 (2020-03-26)
  Configuration:   ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm --with-usrlibdir=/usr/lib64 --enable-fsadm --enable-write_install --with-user= --with-group= --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --enable-pkgconfig --enable-cmdlib --enable-dmeventd --enable-blkid_wiping --with-cluster=internal --enable-cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with-thin=internal --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm --enable-lvmlockd-dlmcontrol --enable-lvmlockd-sanlock --enable-dbus-service --enable-notify-dbus --enable-dmfilemapd --with-writecache=internal --with-vdo=internal --with-vdo-format=/usr/bin/vdoformat --disable-silent-rules

INFO: stdout[1]:   LVM version:     2.03.09(2) (2020-03-26)
  Library version: 1.02.171 (2020-03-26)
  Configuration:   ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm --with-usrlibdir=/usr/lib64 --enable-fsadm --enable-write_install --with-user= --with-group= --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --enable-pkgconfig --enable-cmdlib --enable-dmeventd --enable-blkid_wiping --with-cluster=internal --enable-cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with-thin=internal --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm --enable-lvmlockd-dlmcontrol --enable-lvmlockd-sanlock --enable-dbus-service --enable-notify-dbus --enable-dmfilemapd --with-writecache=internal --with-vdo=internal --with-vdo-format=/usr/bin/vdoformat --disable-silent-rules

7258 INFO: stderr[1]: /dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Incompatible libdevmapper 1.02.171 (2020-03-26) and kernel driver (unknown version).

INFO: stderr[1]: /dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Incompatible libdevmapper 1.02.171 (2020-03-26) and kernel driver (unknown version).

7258 INFO: ...done [1] (exit code: 0)
INFO: ...done [1] (exit code: 0)
7338 WARNING: /usr/lib/python3.8/site-packages/fedora/client/bodhi.py:47: DeprecationWarning: fedora.client.bodhi has been deprecated. Please use bodhi.client.bindings instead.
  warnings.warn('fedora.client.bodhi has been deprecated. Please use bodhi.client.bindings instead.',

WARNING: /usr/lib/python3.8/site-packages/fedora/client/bodhi.py:47: DeprecationWarning: fedora.client.bodhi has been deprecated. Please use bodhi.client.bindings instead.
  warnings.warn('fedora.client.bodhi has been deprecated. Please use bodhi.client.bindings instead.',

7463 WARNING: /usr/lib/python3.8/site-packages/caribou/settings/setting_types.py:14: PyGIDeprecationWarning: GObject.SIGNAL_RUN_FIRST is deprecated; use GObject.SignalFlags.RUN_FIRST instead
  (GObject.SIGNAL_RUN_FIRST,

WARNING: /usr/lib/python3.8/site-packages/caribou/settings/setting_types.py:14: PyGIDeprecationWarning: GObject.SIGNAL_RUN_FIRST is deprecated; use GObject.SignalFlags.RUN_FIRST instead
  (GObject.SIGNAL_RUN_FIRST,

7464 WARNING: /usr/lib/python3.8/site-packages/caribou/settings/setting_types.py:124: PyGIDeprecationWarning: GObject.G_MININT is deprecated; use GLib.MININT instead
  self.min = kwargs.pop('min', GObject.G_MININT)

WARNING: /usr/lib/python3.8/site-packages/caribou/settings/setting_types.py:124: PyGIDeprecationWarning: GObject.G_MININT is deprecated; use GLib.MININT instead
  self.min = kwargs.pop('min', GObject.G_MININT)

7465 WARNING: /usr/lib/python3.8/site-packages/caribou/settings/setting_types.py:125: PyGIDeprecationWarning: GObject.G_MAXINT is deprecated; use GLib.MAXINT instead
  self.max = kwargs.pop('max', GObject.G_MAXINT)

WARNING: /usr/lib/python3.8/site-packages/caribou/settings/setting_types.py:125: PyGIDeprecationWarning: GObject.G_MAXINT is deprecated; use GLib.MAXINT instead
  self.max = kwargs.pop('max', GObject.G_MAXINT)

7465 WARNING: /usr/lib/python3.8/site-packages/caribou/settings/setting_types.py:135: PyGIDeprecationWarning: GObject.G_MINFLOAT is deprecated; use GLib.MINFLOAT instead
  self.min = kwargs.pop('min', GObject.G_MINFLOAT)

WARNING: /usr/lib/python3.8/site-packages/caribou/settings/setting_types.py:135: PyGIDeprecationWarning: GObject.G_MINFLOAT is deprecated; use GLib.MINFLOAT instead
  self.min = kwargs.pop('min', GObject.G_MINFLOAT)

7465 WARNING: /usr/lib/python3.8/site-packages/caribou/settings/setting_types.py:136: PyGIDeprecationWarning: GObject.G_MAXFLOAT is deprecated; use GLib.MAXFLOAT instead
  self.max = kwargs.pop('max', GObject.G_MAXFLOAT)

WARNING: /usr/lib/python3.8/site-packages/caribou/settings/setting_types.py:136: PyGIDeprecationWarning: GObject.G_MAXFLOAT is deprecated; use GLib.MAXFLOAT instead
  self.max = kwargs.pop('max', GObject.G_MAXFLOAT)

cerberus.schema 
cerberus.tests.test_schema 
copr.client_v2.schemas 
8068 WARNING: /usr/lib64/python3.8/pkgutil.py:92: DeprecationWarning: please use 'debian' instead of 'debian_bundle'
  __import__(info.name)

WARNING: /usr/lib64/python3.8/pkgutil.py:92: DeprecationWarning: please use 'debian' instead of 'debian_bundle'
  __import__(info.name)

django.contrib.gis.db.backends.mysql.schema 
django.contrib.gis.db.backends.oracle.schema 
django.contrib.gis.db.backends.postgis.schema 
django.contrib.gis.db.backends.spatialite.schema 
django.db.backends.base.schema 
django.db.backends.mysql.schema 
django.db.backends.oracle.schema 
django.db.backends.postgresql.schema 
django.db.backends.sqlite3.schema 
9403 WARNING: /usr/lib/python3.8/site-packages/fedmsg/__init__.py:67: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  scrub = inspect.getargspec(func).args

WARNING: /usr/lib/python3.8/site-packages/fedmsg/__init__.py:67: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  scrub = inspect.getargspec(func).args

9448 WARNING: Cannot find qpid python module. Make sure you have python-qpid installed.
WARNING: Cannot find qpid python module. Make sure you have python-qpid installed.
9864 WARNING: /usr/lib/python3.8/site-packages/flask_wtf/recaptcha/widgets.py:5: DeprecationWarning: The import 'werkzeug.url_encode' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.urls import url_encode' instead.
  from werkzeug import url_encode

WARNING: /usr/lib/python3.8/site-packages/flask_wtf/recaptcha/widgets.py:5: DeprecationWarning: The import 'werkzeug.url_encode' is deprecated and will be removed in Werkzeug 1.0. Use 'from werkzeug.urls import url_encode' instead.
  from werkzeug import url_encode

9868 WARNING: /usr/lib/python3.8/site-packages/fontTools/misc/py23.py:11: DeprecationWarning: The py23 module has been deprecated and will be removed in the next release. Please update your code.
  warnings.warn(

WARNING: /usr/lib/python3.8/site-packages/fontTools/misc/py23.py:11: DeprecationWarning: The py23 module has been deprecated and will be removed in the next release. Please update your code.
  warnings.warn(

googleapiclient.schema - Schema processing for discovery based APIs
jsonschema - An implementation of JSON Schema for Python
jsonschema.__main__ 
jsonschema._format 
jsonschema._legacy_validators 
jsonschema._reflect - Standardized versions of various cool and/or strange things that you can do
jsonschema._types 
jsonschema._utils 
jsonschema._validators 
jsonschema.benchmarks - Benchmarks for validation.
jsonschema.benchmarks.issue232 - A performance benchmark using the example from issue #232.
jsonschema.benchmarks.json_schema_test_suite - A performance benchmark using the official test suite.
jsonschema.cli - The ``jsonschema`` command line.
jsonschema.compat - Python 2/3 compatibility helpers.
jsonschema.exceptions - Validation errors, and some surrounding helpers.
jsonschema.tests 
jsonschema.tests._helpers 
jsonschema.tests._suite - Python representations of the JSON Schema Test Suite tests.
jsonschema.tests.test_cli 
jsonschema.tests.test_exceptions 
jsonschema.tests.test_format - Tests for the parts of jsonschema related to the :validator:`format` property.
jsonschema.tests.test_jsonschema_test_suite - Test runner for the JSON Schema official test suite
jsonschema.tests.test_types - Tests on the new type interface. The actual correctness of the type checking
jsonschema.tests.test_validators 
jsonschema.validators - Creation and extension of validators, with implementations for existing drafts.
marshmallow.class_registry - A registry of :class:`Schema <marshmallow.Schema>` classes. This allows for string
marshmallow.decorators - Decorators for registering schema pre-processing and post-processing methods.
marshmallow.schema - The :class:`Schema` class, including its metaclass and options (class Meta).
pyroute2.ndb.schema 
rdflib.plugins.sparql.datatypes - Utility functions for supporting the XML Schema Datatypes hierarchy
lxml.isoschematron - The ``lxml.isoschematron`` package implements ISO Schematron support on top
pandas.io.json._table_schema - Table Schema builders
pandas.tests.io.json.test_json_table_schema - Tests for Table Schema integration.
playhouse.migrate - Lightweight schema migrations.
samba.ms_schema 
samba.ms_schema_markdown 
samba.netcmd.schema 
samba.schema - Functions for setting up a Samba Schema.
sqlalchemy.dialects.mssql.information_schema 
sqlalchemy.engine.reflection - Provides an abstraction for obtaining database schema information.
sqlalchemy.schema - Compatibility namespace for sqlalchemy.sql.schema and related.
sqlalchemy.sql.ddl - Provides the hierarchy of DDL-defining schema items as well as routines
sqlalchemy.sql.schema - The schema module provides the building blocks for database metadata.
sqlalchemy.testing.schema 

Now, obviously, the real fault is with those packages that insist on spewing noise the moment they’re imported. But is there anything more pydoc could be doing to suppress [more of] those messages, when it’s merely scanning the modules’ contents for keywords?

There is – for example it could for example scan for docstrings statically, instead of importing the code.
The issue is priority: is the problem worth the time it would take to solve it?

1 Like

Fair enough. Obviously, there’s always something that can be done, if you’re willing to put enough effort into it. So I guess that was really my question: Is there a solution that can be implemented easily enough that it’d be worth it?

Worth it for whom? On my TODO list it is way down in the “maybe/never” category. I can walk you through the options if you’d like to do the work.

The first step would be to open an issue at https://bugs.python.org/ – as fat as I can see, one isn’t open for this yet (though it’s hard to search for -k).

My 2c: let’s ignore broken packages.

Possibly also ignore deprecation warnings.