# AC: \`NULL\` defaults prevent correct signatures. Let's add \`inspect.unrepresentable\` to fix this **URL:** \ **Category:** Core Development\ **Created:** [August 1, 2023, 9:26am UTC](https://discuss.python.org/t/ac-null-defaults-prevent-correct-signatures-lets-add-inspect-unrepresentable-to-fix-this/30753 "2023-08-01T09:26:45Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![sobolevn](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sobolevn/32/9274_2.png) [@sobolevn](https://discuss.python.org/u/sobolevn)\ **Post date:** [August 1, 2023, 9:26am UTC](https://discuss.python.org/t/ac-null-defaults-prevent-correct-signatures-lets-add-inspect-unrepresentable-to-fix-this/30753/1 "2023-08-01T09:26:45Z")
## Problem While working on [https://github.com/python/cpython/pull/103132](https://github.com/python/cpython/pull/103132) I’ve noticed that `NULL` default is a big problem for current defaults in AC. Right now, `inspect.signature` will fail for any function with `NULL` as the default. Let’s take `builtins.iter` (on 3.12) as an example: ```python >>> iter. __text_signature__'($module, object, sentinel=, /)' >>> import inspect >>> inspect.signature(iter) Traceback (most recent call last): File "", line 1, in File "/Users/sobolev/Desktop/cpython/Lib/inspect.py", line 3362, in signature return Signature.from_callable(obj, follow_wrapped=follow_wrapped, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/sobolev/Desktop/cpython/Lib/inspect.py", line 3106, in from_callable return _signature_from_callable(obj, sigcls=cls, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/sobolev/Desktop/cpython/Lib/inspect.py", line 2599, in _signature_from_callable return _signature_from_builtin(sigcls, obj, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/sobolev/Desktop/cpython/Lib/inspect.py", line 2400, in _signature_from_builtin return _signature_fromstr(cls, func, s, skip_bound_arg) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/sobolev/Desktop/cpython/Lib/inspect.py", line 2261, in _signature_fromstr raise ValueError("{!r} builtin has invalid signature".format(obj)) ValueError: builtin has invalid signature ``` This is how AC converts `NULL` to be `` in ` __text_signature__ `: ```python /*[clinic input] test_str_converter a: str = NULL / [clinic start generated code]*/ PyDoc_STRVAR(test_str_converter __doc__ , "test_str_converter($module, a=)\n" "--\n" "\n"); ``` Right now we have ~52 files with `` signatures. There’s also a user-reported issue about `bytes.hex` having incorrect `inspect.Signature`: [inspect.signature(bytes.hex) raises ValueError "builtin has invalid signature" · Issue #87233 · python/cpython · GitHub](https://github.com/python/cpython/issues/87233) And incorrect signatures in `builtins` module: [Help text of builtin functions – missing signatures · Issue #107526 · python/cpython · GitHub](https://github.com/python/cpython/issues/107526) In `typeshed` we use `...` to specify default values. For example, here’s how `dir` is defined: [typeshed/stdlib/builtins.pyi at 1c0500a57050815102c702efd053e09770a5ee88 · python/typeshed · GitHub](https://github.com/python/typeshed/blob/1c0500a57050815102c702efd053e09770a5ee88/stdlib/builtins.pyi#L1320) ## Proposed solution I propose adding a special signleton value `inspect.unrepesentable` to be used instead. We can customize its ` __repr__ ` to be ``, `...`, or whatever. Here’s how a hypothetical patch would look like: ```diff diff --git Lib/inspect.py Lib/inspect.py index c8211833dd0..64e1b8f0839 100644 --- Lib/inspect.py +++ Lib/inspect.py @@ -2238,10 +2238,29 @@ def _signature_strip_non_python_syntax(signature): add(string) if (string == ','): add(' ') - clean_signature = ''.join(text).strip().replace("\n", "") + clean_signature = ''.join(text).strip().replace("\n", "").replace( + # Handle `NULL` defaults: + "", + " __unrepresentable__", + ) return clean_signature, self_parameter +class _Unrepresentable: + _instance = None + + def __new__ (cls): + if cls._instance is not None: + return cls._instance + cls._instance = super(). __new__ (cls) + return cls._instance + + def __repr__ (self): + return "" + +unrepresentable = _Unrepresentable() + + def _signature_fromstr(cls, obj, s, skip_bound_arg=True): """Private helper to parse content of ' __text_signature__' and return a Signature based on it. @@ -2309,6 +2328,8 @@ def visit_Attribute(self, node): def visit_Name(self, node): if not isinstance(node.ctx, ast.Load): raise ValueError() + if node.id == " __unrepresentable__": + return unrepresentable return wrap_value(node.id) def visit_BinOp(self, node): @@ -2331,7 +2352,10 @@ def p(name_node, default_node, default=empty): if default_node and default_node is not _empty: try: default_node = RewriteSymbolics().visit(default_node) - default = ast.literal_eval(default_node) + if default_node is unrepresentable: + default = unrepresentable + else: + default = ast.literal_eval(default_node) except ValueError: raise ValueError("{!r} builtin has invalid signature".format(obj)) from None parameters.append(Parameter(name, kind, default=default, annotation=empty)) ``` This will allow us to parse and inspect this signature: ```python >>> import inspect >>> sig = inspect.signature(iter) >>> sig.parameters mappingproxy(OrderedDict({'object': , 'sentinel': ">})) >>> sig.parameters['sentinel'] "> >>> sig.parameters['sentinel'].default >>> sig.parameters['sentinel'].default is inspect.unrepresentable True ``` Related: - [https://github.com/python/cpython/pull/13933](https://github.com/python/cpython/pull/13933) - [Issue 37206: Incorrect application of Argument Clinic to dict.pop() - Python tracker](https://bugs.python.org/issue37206) - [Signatures, a call to action - #34 by larry](https://discuss.python.org/t/signatures-a-call-to-action/23580/34) I others agree, I can submit my patch + tests + docs. CC @erlendaasland @storchaka ---
**Author:** ![erlendaasland](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/erlendaasland/32/4378_2.png) [@erlendaasland](https://discuss.python.org/u/erlendaasland)\ **Post date:** [August 1, 2023, 9:31am UTC](https://discuss.python.org/t/ac-null-defaults-prevent-correct-signatures-lets-add-inspect-unrepresentable-to-fix-this/30753/2 "2023-08-01T09:31:49Z")
Thanks for taking this on; +1 from me. I would prefer `...` instead of ``. ---
**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\ **Post date:** [August 1, 2023, 10:22am UTC](https://discuss.python.org/t/ac-null-defaults-prevent-correct-signatures-lets-add-inspect-unrepresentable-to-fix-this/30753/3 "2023-08-01T10:22:15Z")
It would be nice if the issue be so simple. But it is not. `iter(object)` and `iter(object, inspect.unrepresentable)` are different calls. For now, the only special value for `Parameter.default` is `Parameter.empty`, any code which builds `args` and `kwargs` by inspecting a signature will fail on `unrepresentable`. ---
**Author:** ![zware](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/zware/32/58_2.png) [@zware](https://discuss.python.org/u/zware)\ **Post date:** [August 1, 2023, 2:42pm UTC](https://discuss.python.org/t/ac-null-defaults-prevent-correct-signatures-lets-add-inspect-unrepresentable-to-fix-this/30753/4 "2023-08-01T14:42:28Z")
> [@sobolevn](#): > > We can customize its ` __repr__ ` to be ``, `...`, or whatever. `...` is out: ```python >>> ... is Ellipsis True ``` ---
**Author:** ![sobolevn](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sobolevn/32/9274_2.png) [@sobolevn](https://discuss.python.org/u/sobolevn)\ **Post date:** [August 1, 2023, 3:57pm UTC](https://discuss.python.org/t/ac-null-defaults-prevent-correct-signatures-lets-add-inspect-unrepresentable-to-fix-this/30753/5 "2023-08-01T15:57:29Z")
We can document that this is just our way of representing the unrepresentable. And passing `_Unrepresentable` instance would not work for calling these functions. It will only work for inspecting the signatures. I think that it is better than the current state: just no signature for `NULL`. ---
**Author:** ![sobolevn](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/sobolevn/32/9274_2.png) [@sobolevn](https://discuss.python.org/u/sobolevn)\ **Post date:** [August 10, 2023, 7:02am UTC](https://discuss.python.org/t/ac-null-defaults-prevent-correct-signatures-lets-add-inspect-unrepresentable-to-fix-this/30753/6 "2023-08-10T07:02:21Z")
I’ve done some research about generating signatures based on @storchaka feedback and I now agree that we cannot make this the default. But, adding `include_unrepresentable=False` flag to `inspect.signature` function seems like a simple and working solution. It will: - Allow us to use `inspect.signature(..., include_unrepresentable=True)` on all things inside (like in `pydoc`, `help()`, etc). And things that have `NULL` default will work correctly for our use-case - Not break anyone else’s code - Not complicate future potential `inspect.signatures` implementation Downsides: - We will leak our internal tooling (which I consider it to be) as a public API, so maybe `_include_unrepresentable`, to indicate that this is some very strange argument? ---
**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\ **Post date:** [August 10, 2023, 9:44am UTC](https://discuss.python.org/t/ac-null-defaults-prevent-correct-signatures-lets-add-inspect-unrepresentable-to-fix-this/30753/7 "2023-08-10T09:44:31Z")
See [Pydoc: fall back to \_\_text\_signature\_\_ if inspect.signature() fails · Issue #107782 · python/cpython · GitHub](https://github.com/python/cpython/issues/107782) which adds a workaround of this problem in `pydoc`. I believe that the correct general solution to this problem is to support multi-signatures. Instead of `(object, sentinel=, /)` you will get a union of `(object, /)` and `(object, sentinel, /)`. I looked at the corresponding `inspect` code, and it looks feasible. I think I can do it in the next few weeks or months. Maybe I’ll start today. ---
**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\ **Post date:** [August 10, 2023, 10:44am UTC](https://discuss.python.org/t/ac-null-defaults-prevent-correct-signatures-lets-add-inspect-unrepresentable-to-fix-this/30753/8 "2023-08-10T10:44:15Z")
See also the discussion: > [@Signatures, a call to action](https://discuss.python.org/t/signatures-a-call-to-action/23580/): > > I saw the other thread on math.log() and wanted to start a new one to shift the focus to what I think is the underlying problem than needs to be solved. If the playful story telling style doesn’t fit your tastes, please try and look past the style and focus on the substance of the post. I tried rewriting this a few times but found that the parallel construction form of comparison and contrast best communicated where work needs to be done. There once was little scripting language called Py… ---
**Author:** ![tjreedy](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tjreedy/32/137_2.png) [@tjreedy](https://discuss.python.org/u/tjreedy)\ **Post date:** [August 10, 2023, 6:22pm UTC](https://discuss.python.org/t/ac-null-defaults-prevent-correct-signatures-lets-add-inspect-unrepresentable-to-fix-this/30753/9 "2023-08-10T18:22:06Z")
Serhiy, thank you for the reference to Raymond’s Signatures thread. I always thought that marking optional parameters with no default with square brackets in the doc is just fine. Perfectly understandable to humans. I think that replacing bracket in the Library Buitin-functions chapter with two signatures is a regression for human readers. For me, there is one signature – pass something and maybe something else. At least reading that thread explains why the change was made even though I think it wrong. I hope that after you fix AC and inspect for tool use of signatures, pydoc can be changed to display the better form for humans. ---
**Author:** ![Jelle](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jelle/32/1049_2.png) [@Jelle](https://discuss.python.org/u/Jelle)\ **Post date:** [November 9, 2023, 4:31pm UTC](https://discuss.python.org/t/ac-null-defaults-prevent-correct-signatures-lets-add-inspect-unrepresentable-to-fix-this/30753/10 "2023-11-09T16:31:36Z")
I ran into this problem too because the tool we use to verify function signatures in typeshed relies on `inspect.signature`, which fails on these “``” defaults. I proposed [a hacky solution](https://github.com/python/mypy/pull/16433), but it would be better if the `inspect` module supported these signatures directly. I would favor Nikita’s suggested solution of adding a marker like `inspect.unrepresentable` (I’d vote for exposing it as `inspect.Parameter.unrepresentable`, similar to `.empty`). @storchaka objects that this might break tools that assume something like `iter(object, inspect.unrepresentable)` would work. But it’s not unexpected for such tools that inspect signatures to have to adapt to new features in new Python versions (e.g., positional-only parameters). I help maintain several tools that rely on `inspect.signature`, and I’d much prefer if they could support unrepresentable defaults, even if that means I have to make some changes to support Python 3.13. There is an alternative suggestion to support “multi-signatures”. That would be a good solution for some functions like `iter`, but for other cases like `bytes.hex`, I don’t think that solution is more elegant. It also adds significantly more complexity.