Issues setting PyFunctionObject's vectorcall field in Python3.11+

I have some code that used to work that looked like:

void set_my_vectorcall(PyFunctionObject *func_obj){
   func_obj->vectorcall = myvectorcall; // myvectorcall preserves original behavior of func_obj->vectorcall
}

This works up through python3.10 as expected: func_obj->vectorcall is set and myvectorcall is called whenever func_obj is called. However as of 3.11, func_obj->vectorcall is set, but myvectorcall is not called whenever func_obj is called. I am aware of (and have tried) using PyFunction_SetVectorcall in 3.12, but that doesn’t help either. I’ve even confirmed “PyObject_Call(func_obj, PyTuple_New(0), NULL);” does indeed call myvectorcall after func_obj->vectorcall is set.

Any ideas why myvectorcall wouldn’t be called when func_obj is called in python3.11? Given it works when I manually invoke PyObject_Call, I’m really not sure what could be going on.

I can try and create a reproducer on request.

Thanks!

Here is a small reproducer, instructions at top of file:

// file custom.c
//
// PYVERSION=3.9 will print "my_vectorcallfunc", whereas PYVERSION=3.11 will not. This is the problem I am running into.
// To compile:
// PYVERSION=3.11; gcc -fPIC -shared -o custom.so -I /usr/include/python${PYVERSION}/ custom.c -L /usr/lib64/python${PYVERSION}/config-${PYVERSION}-x86_64-linux-gnu/ -lpython${PYVERSION} 
// To run:
// LD_LIBRARY_PATH=LD_LIBRARY_PATH:/usr/lib64/python${PYVERSION}/ python${PYVERSION} -c 'import custom; import os.path; custom.custom_function(os.path.realpath); print(os.path.realpath("abc/123"))'
#include <Python.h>

#include <stdio.h>

static PyObject *(*old_vectorcall)(PyObject *callable, PyObject *const *args, size_t nargsf, PyObject *kwnames);
static PyObject *my_vectorcallfunc(PyObject *callable, PyObject *const *args, size_t nargsf, PyObject *kwnames){
    printf("%s\n",__func__);
    return old_vectorcall(callable, args, nargsf, kwnames);
}

static PyObject* custom_function(PyObject* self, PyObject* args) {
    PyFunctionObject *args_func = (PyFunctionObject *) PyTuple_GET_ITEM(args, 0);
    old_vectorcall = args_func->vectorcall;
    args_func->vectorcall = (vectorcallfunc) my_vectorcallfunc;
    PyObject_Print(self, stdout, Py_PRINT_RAW);
    printf("\n1\n");
    PyObject_Print(args, stdout, Py_PRINT_RAW);
    printf("\n2\n");
    return args;
}

static PyMethodDef custom_methods[] = {
    {"custom_function", custom_function, METH_VARARGS, "Description of custom_function"},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef custommodule = {
    PyModuleDef_HEAD_INIT,
    "custom",
    "Module documentation",
    -1,
    custom_methods
};

PyMODINIT_FUNC PyInit_custom(void) {
    return PyModule_Create(&custommodule);
}

@encukou you’ve replied to my other posts. Do you have any ideas what’s going wrong, or how to work around it?

Another peculiarity is that with Python3.11, PyObject_CallObject does call my_vectorcallfunc, even though a normal call to os.path.realpath doesn’t:

    // before custom_function's "return args;"
    PyObject *tuple = PyTuple_Pack(1, PyUnicode_FromString("def/456"));
    PyObject_CallObject((PyObject*) args_func, tuple); // prints "my_vectorcallfunc"

Hm, I wasn’t aware PyFunction_SetVectorcall existed. The docs don’t really explain what its purpose is, but the issue that added it makes it clearer.

I’m not aware of any guarantee that vectorcall, rather than tp_call, will be used. But you should probably ask people involved in adding PyFunction_SetVectorcall.

@itamaro @markshannon any ideas?

See PEP 590 – Vectorcall: a fast calling protocol for CPython | peps.python.org

Specifically the part that says “must implement the tp_call function and make sure its behaviour is consistent with the vectorcallfunc function”
By implication, if tp_call must be consistent with vectorcallfunc, then vectorcallfunc must be consistent with tp_call, and because they must be consistent the VM is free to call whichever one it wants.

In practice, we assume that the whichever form fits the given arguments best will be the fastest, so that’s what gets called.

Try comparing the behaviour of the calls f(1,2) and f(*(1,2))

One other thing.
Directly setting fields of internal data structures is unsafe and unsupported. You may be breaking invariants that you are not aware of.
There is no such problem with calling an API function like PyFunction_SetVectorcall, which is why it was added.

1 Like

Thanks for the reply, @markshannon !

I had read that invariant, but forgot to accomodate for it when creating my small reproducer. The PEP says:

Setting tp_call to PyVectorcall_Call is sufficient.

But even if I add this when I’m setting vectorcallfunc:

    Py_TYPE(args_func)->tp_call = PyVectorcall_Call;
    PyType_Modified(Py_TYPE(args_func));

It appears to have no effect.

I did rerun my small reproducer with python3.12 and PyFunction_SetVectorcall and it works as expected, so that’s good.

I know setting vectorcallfunc for <python3.12 is not technically supported, but any reason you can think of as to why that in conjunction with setting tp_call to PyVectorcall_Call and calling PyType_Modified wouldn’t have the desired effect? I’ve also tried setting “func_version = 0” as done internally in python3.12’s PyFunction_SetVectorcall, but no luck.

PyFunction_SetVectorcall PR for context.

As I said before, “Directly setting fields of internal data structures is unsafe and unsupported.”
You can’t just set the tp_call field of PyFunction_Type.

You will need to create an entirely new class, setting tp_call and tp_vectorcall of that class when you create it.

OOI, why do you want to do this?

You will need to create an entirely new class, setting tp_call and tp_vectorcall of that class when you create it.

Yes, this did come to mind, but I’m not sure its suitable for what I’m trying to do. What I’m trying to do is, with the C Python API, wrap individual python functions while not altering the behavior of the code. The problem with creating a new class is that any type(wrapped_func) in python code changes.

Appreciate you taking a look and giving this some thought!