Are Import in classes not working when c implementation available?

Python Help
, , ,
1

I was playing with standard library module [heapq](heapq — Heap queue algorithm — Python 3.13.2 documentation, which Imho has a very weird API where you use a list as a heap… I didn´t really like that and for what I see I’m not the only one.

Soon I realized that I could patch everything that I don´t like with just one extra line:

class heapq(list):
    from heapq import heappush as hpush, heappop as hpop

Needless to say I was very proud of it because now I can use the heap just like a list, with his own type and methods, for me looks very elegant.

my_heap = heapq()
to_append =  (1,"task1") 
my_heap.push(to_append)

Buuut, I got:

TypeError: heappush expected 2 arguments got 1

Interesting because the definition of heappush is:

def heappush(heap, item):
    """Push item onto heap, maintaining the heap invariant."""
    heap.append(item)
    _siftdown(heap, 0, len(heap)-1)

So now ‘heap’ is the ‘self’ and therefore I’m passing both arguments 0.o, interesting…

then i realize these lines at the end of the library:

# If available, use C implementation
try:
    from _heapq import *
except ImportError:
    pass

I remove these and for my surprise everything works !!

But does this mean that my class don´t pass the instance if the method is coming from C? Is this a bug from python itself? anyone knows how does this work?

2

Hello,

Just note that technically, you have only entered one argument which is one tuple which contains two elements.

Have you tried instead:

my_heap.push( 1,"task1")  # two explicit arguments and not two in a tuple
3

Is intended, to a heap of this library you need to pass a tuple to heappush, but i change it so is more clear. But in short, that was not the issue.

4

Have you tried this tutorial:

From this example, if you would like to add an element to the heap, you enter the name of the list and the element value that you would like to append. Note that it is not in tuple form.

import heapq

# Creating an initial heap
h = [10, 20, 15, 30, 40]
heapq.heapify(h)

# Appending an element
heapq.heappush(h, 5)

print('New heap value: ', h)
5

This is a question about a python behavior when importing functions coming from C inside classes.

Is not a question about heapq, heapq appears to demonstrate the minimal reproducible example, I did read the docs of heapq and even the source code.

The example from the docs of heapq is:

h = []
heappush(h, (5, 'write code'))
heappush(h, (7, 'release product'))
heappush(h, (1, 'write spec'))
heappush(h, (3, 'create tests'))
heappop(h)
6

This is to do with how bound methods work and that the C implementation as a builtin function doesn’t work the same as a regular function defined in Python code.

Here you can see that the C defined heappush remains a built-in function when accessed from the class or the instance, while the python defined version becomes a bound method. This is what provides the initial argument and is why the C implementation behaves differently in this case.

>>> from heapq import _siftdown, heappush
>>> def py_heappush(heap, item):
...     heap.append(item)
...     _siftdown(heap, 0, len(heap) - 1)
...
>>> class MyHeap(list):
...     c_heappush = heappush
...     py_heappush = py_heappush
...
>>> MyHeap.c_heappush
<built-in function heappush>
>>> MyHeap.py_heappush
<function py_heappush at 0x109fd00e0>
>>> inst = MyHeap()
>>> inst.c_heappush
<built-in function heappush>
>>> inst.py_heappush
<bound method py_heappush of []>
2 Likes
7

Interesting, so basically python doesn´t bound methods that come from C functions right? and is this intended or just a bug in python?

8

btw, I ran your script as you have provided it except that I added the name of the instance:

class heapq(list):
    from heapq import heappush as hpush, heappop as hpop

my_heap = heapq()
to_append =  (1,"task1")
my_heap.hpush(my_heap, to_append)

print(my_heap)

or

my_heap = heapq()
to_append = ("task1")
my_heap.hpush(my_heap, to_append)

print(my_heap)

Elaborating further:

class heapq(list):
    from heapq import heappush as hpush, heappop as hpop

my_heap = heapq()
to_append =  ["task1", 'task2', 'task3', 'task4', 'task5']

for val in to_append:
    my_heap.hpush(my_heap, val)

print(my_heap)

my_heap.hpop(to_append)

print(to_append)
9

I believe it’s just that built-in functions don’t have a __get__ method which is how regular Python functions are turned into methods.

>>> py_heappush.__get__(None, MyHeap)
<function py_heappush at 0x109fd00e0>
>>> py_heappush.__get__(MyHeap(), MyHeap)
<bound method py_heappush of []>

>>> heappush.__get__
Traceback (most recent call last):
  File "<python-input-13>", line 1, in <module>
    heappush.__get__
AttributeError: 'builtin_function_or_method' object has no attribute '__get__'. Did you mean: '__ge__'?

I’m not sure I’d consider it a bug, but it is an observable difference in some functions implemented in C compared to the functions you can implement at the Python level.

10

I think you’re misunderstanding the issue here.

The C implementation of the function and the Python implementation are subtly different in behaviour when it comes to being accessed as an instance attribute.

The C implementation remains a basic function while the Python implementation is converted into a bound method which provides the instance as the first argument automatically.

This can be demonstrated with PyPy which relies on the Python implementation of heapq. Running your example:

CPython:

Python 3.11.11 (main, Jan 14 2025, 23:51:34) [Clang 19.1.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class heapq(list):
...     from heapq import heappush as hpush, heappop as hpop
...
>>> my_heap = heapq()
>>> to_append =  (1,"task1")
>>> my_heap.hpush(my_heap, to_append)
>>>
>>> print(my_heap)
[(1, 'task1')]

PyPy:

Python 3.11.11 (b38de282cead, Feb 05 2025, 18:10:31)
[PyPy 7.3.18 with GCC Apple LLVM 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>> class heapq(list):
....     from heapq import heappush as hpush, heappop as hpop
....
.... my_heap = heapq()
.... to_append =  (1,"task1")
.... my_heap.hpush(my_heap, to_append)
....
.... print(my_heap)
....
Traceback (most recent call last):
  File "<python-input-0>", line 6, in <module>
    my_heap.hpush(my_heap, to_append)
TypeError: heappush() takes 2 positional arguments but 3 were given. Did you forget 'self' in the function definition?
[]

In PyPy, as it’s using the Python implementation instead of the C implementation, the function is turned into a bound method when accessed from an instance.

>>>> my_heap.hpush(to_append)
>>>> print(my_heap)
[(1, 'task1')]
1 Like
11

I’m aware, to see the issue just define another function freely and import it to the class in the same way, you will see that it is used as a method, and you don´t need to pass the instance as a variable.

The question was never about how to use heapq, just about python funcitons against C ones, just happens that heapq gives you both python and C implementations.

12

Very very interesting thanks! also the pypy detail, I didn´t know any of this things.

It would be amazing to know if this behavior is intended, because is clearly very misleading, there is no warning and when having an error is very unexpected.

Specially concerning if you can also trigger it as you did with class attributes that are much more common than in-class imports.

14

Ahh, ok, I see the difference. Thank you very much for pointing that out.

15

I wouldn’t find it very concerning as this only happens if you happen to assign a built-in function as an attribute on a class, which is probably fairly rare?

The only thing that might be considered a bug would be the difference in behaviour if you get the Python or C implementations. However I’d almost consider this to be an internal implementation detail and not something you should rely on?

16

I once had a similar question.

17

oh i see 10 years ago, Probably 10 years from now someone will also find this issue and will still be here, but yeh, as David says is probably not so relevant after all.