Misleading error message, shows the wrong code

Python Help
1

The code

a = [1, 2]
b = [3, 4]
diffs = filter(None, map(a, b))
next(diffs, len(a) - len(b))

leads to this error:

Traceback (most recent call last):
  File ".code.tio", line 6, in <module>
    next(diffs, len(a) - len(b))
TypeError: 'list' object is not callable

What list gets called there? I only call next and len there, and neither is a list.

The bug is actually in the line before (which should’ve done map(cmp, a, b), I forgot the compare function, so the list a gets called instead). It would be better if the actual buggy line were included in the traceback.

Is this a known problem? I don’t remember seeing this problem before, and I don’t see a GitHub issue about it, but I suspect someone must’ve encountered this already and perhaps it was decided that it’s ok to not include the buggy line or somehow too difficult to include it.

Tested on Python 3.8 and 3.10.2.

2 Likes
2

That is really bad that the traceback does not show that the exception comes from inside the map() function.

Is not this caused by the fact that the map function is not written in Python?

>>> hasattr(map, '__code__')
False
>>> import types
>>> isinstance(map, types.FunctionType)
False
3

Could be, yes. If I try the similar diffs = (a(x) for x in b), I get this:

Traceback (most recent call last):
  File ".code.tio", line 4, in <module>
    next(diffs, len(a) - len(b))
  File ".code.tio", line 3, in <genexpr>
    diffs = (a(x) for x in b)
TypeError: 'list' object is not callable
4

I think the delay in recognising the error is caused by the fact that both filter and map are lazy. map does not try to apply the non-function a to the first element of iterable b until next asks for the first element of the result. CPython correctly reports this is the line during which the error occurs, but it is not a very good guide to where the mistake is.

I would call this a strategic error in the way exceptions are reported from within built-in functions. The author of map could produce a slightly more meaningful message, but it’s a point solution with many parallels through the code base. I don’t immediately see a strategic solution. Maybe if built-ins produced a sort of minimal interpreter frame?

5

How hard is it for map to check that its arg is callable and raise an error at the call site?

6

Looks like an easy point solution, and I think only breaks arcane code where the first argument becomes callable later.

7

No, it does not matter if a function is a generator (as you say “lazy”). Normally the exception shows at the correct place. (Maybe you meant it is both a generator and not written in Python?)

Let’s reimplement map() in Python:

def map(function, *iterables):
    return (function(*items) for items in zip(*iterables))

a = [1, 2]
b = [3, 4]
diffs = filter(None, map(a, b))
next(diffs, len(a) - len(b))

Now the exception shows at the correct place:

Traceback (most recent call last):
  File "/home/vbrozik/tmp/python/test_generator_exception.py", line 7, in <module>
    next(diffs, len(a) - len(b))
  File "/home/vbrozik/tmp/python/test_generator_exception.py", line 2, in <genexpr>
    return (function(*items) for items in zip(*iterables))
TypeError: 'list' object is not callable
8

I wanted to see the nice exception message of Python 3.11. Here it is:

$ docker run --rm -v "$(pwd):/python:ro" python:3.11-rc-alpine python3 /python/test_generator_exception.py
Traceback (most recent call last):
  File "/python/test_generator_exception.py", line 7, in <module>
    next(diffs, len(a) - len(b))
  File "/python/test_generator_exception.py", line 2, in <genexpr>
    return (function(*items) for items in zip(*iterables))
            ^^^^^^^^^^^^^^^^
TypeError: 'list' object is not callable

With the builtin map() we have nothing new :sob:

$ docker run --rm -v "$(pwd):/python:ro" python:3.11-rc-alpine python3 /python/test_generator_exception_orig_map.py
Traceback (most recent call last):
  File "/python/test_generator_exception_orig_map.py", line 7, in <module>
    next(diffs, len(a) - len(b))
TypeError: 'list' object is not callable
9

Deftly done. Yes, recognition of the error is still delayed until line 7 is executed, but because map here is written in Python, there is a frame to locate it more precisely at line 2.

I was speculating that one could decouple being a Python code object from being a context for an error report, say by marking certain functions as a virtual frame. Barry’s specific fix is also good.