# Issues setting PyFunctionObject's vectorcall field in Python3.11+

**URL:** https://discuss.python.org/t/issues-setting-pyfunctionobjects-vectorcall-field-in-python3-11/39826
**Category:** Python Help
**Tags:** c-api
**Created:** [November 28, 2023, 11:16pm UTC](https://discuss.python.org/t/issues-setting-pyfunctionobjects-vectorcall-field-in-python3-11/39826 "2023-11-28T23:16:08Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![TannerFirl](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tannerfirl/32/11345_2.png) [@TannerFirl](https://discuss.python.org/u/TannerFirl)
#### Post date: [November 28, 2023, 11:16pm UTC](https://discuss.python.org/t/issues-setting-pyfunctionobjects-vectorcall-field-in-python3-11/39826/1 "2023-11-28T23:16:08Z")

</div>

I have some code that used to work that looked like:

```auto
void set_my_vectorcall(PyFunctionObject *func_obj){
   func_obj->vectorcall = myvectorcall; // myvectorcall preserves original behavior of func_obj->vectorcall
}

```

This works up through python3.10 as expected: func\_obj-\>vectorcall is set and myvectorcall is called whenever func\_obj is called. However as of 3.11, func\_obj-\>vectorcall is set, but myvectorcall is not called whenever func\_obj is called. I am aware of (and have tried) using PyFunction\_SetVectorcall in 3.12, but that doesn’t help either. I’ve even confirmed “PyObject\_Call(func\_obj, PyTuple\_New(0), NULL);” does indeed call myvectorcall after func\_obj-\>vectorcall is set.

Any ideas why myvectorcall wouldn’t be called when func\_obj is called in python3.11? Given it works when I manually invoke PyObject\_Call, I’m really not sure what could be going on.

I can try and create a reproducer on request.

Thanks!

---

<div class="post-metadata">

### Author: ![TannerFirl](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tannerfirl/32/11345_2.png) [@TannerFirl](https://discuss.python.org/u/TannerFirl)
#### Post date: [December 3, 2023, 3:51pm UTC](https://discuss.python.org/t/issues-setting-pyfunctionobjects-vectorcall-field-in-python3-11/39826/2 "2023-12-03T15:51:33Z")

</div>

Here is a small reproducer, instructions at top of file:

```auto
// file custom.c
//
// PYVERSION=3.9 will print "my_vectorcallfunc", whereas PYVERSION=3.11 will not. This is the problem I am running into.
// To compile:
// PYVERSION=3.11; gcc -fPIC -shared -o custom.so -I /usr/include/python${PYVERSION}/ custom.c -L /usr/lib64/python${PYVERSION}/config-${PYVERSION}-x86_64-linux-gnu/ -lpython${PYVERSION} 
// To run:
// LD_LIBRARY_PATH=LD_LIBRARY_PATH:/usr/lib64/python${PYVERSION}/ python${PYVERSION} -c 'import custom; import os.path; custom.custom_function(os.path.realpath); print(os.path.realpath("abc/123"))'
#include <Python.h>

#include <stdio.h>

static PyObject *(*old_vectorcall)(PyObject *callable, PyObject *const *args, size_t nargsf, PyObject *kwnames);
static PyObject *my_vectorcallfunc(PyObject *callable, PyObject *const *args, size_t nargsf, PyObject *kwnames){
    printf("%s\n", __func__ );
    return old_vectorcall(callable, args, nargsf, kwnames);
}

static PyObject* custom_function(PyObject* self, PyObject* args) {
    PyFunctionObject *args_func = (PyFunctionObject *) PyTuple_GET_ITEM(args, 0);
    old_vectorcall = args_func->vectorcall;
    args_func->vectorcall = (vectorcallfunc) my_vectorcallfunc;
    PyObject_Print(self, stdout, Py_PRINT_RAW);
    printf("\n1\n");
    PyObject_Print(args, stdout, Py_PRINT_RAW);
    printf("\n2\n");
    return args;
}

static PyMethodDef custom_methods[] = {
    {"custom_function", custom_function, METH_VARARGS, "Description of custom_function"},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef custommodule = {
    PyModuleDef_HEAD_INIT,
    "custom",
    "Module documentation",
    -1,
    custom_methods
};

PyMODINIT_FUNC PyInit_custom(void) {
    return PyModule_Create(&custommodule);
}

```

@encukou you’ve replied to my other posts. Do you have any ideas what’s going wrong, or how to work around it?

Another peculiarity is that with Python3.11, PyObject\_CallObject _does_ call my\_vectorcallfunc, even though a normal call to os.path.realpath doesn’t:

```auto
    // before custom_function's "return args;"
    PyObject *tuple = PyTuple_Pack(1, PyUnicode_FromString("def/456"));
    PyObject_CallObject((PyObject*) args_func, tuple); // prints "my_vectorcallfunc"

```

---

<div class="post-metadata">

### Author: ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)
#### Post date: [December 5, 2023, 1:14pm UTC](https://discuss.python.org/t/issues-setting-pyfunctionobjects-vectorcall-field-in-python3-11/39826/3 "2023-12-05T13:14:06Z")

</div>

Hm, I wasn’t aware `PyFunction_SetVectorcall` existed. The docs don’t really explain what its purpose is, but [the issue that added it](https://github.com/python/cpython/issues/91049) makes it clearer.

I’m not aware of any guarantee that vectorcall, rather than tp\_call, will be used. But you should probably ask people involved in adding `PyFunction_SetVectorcall`.

---

<div class="post-metadata">

### Author: ![TannerFirl](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tannerfirl/32/11345_2.png) [@TannerFirl](https://discuss.python.org/u/TannerFirl)
#### Post date: [December 5, 2023, 1:25pm UTC](https://discuss.python.org/t/issues-setting-pyfunctionobjects-vectorcall-field-in-python3-11/39826/4 "2023-12-05T13:25:22Z")

</div>

@itamaro @markshannon any ideas?

---

<div class="post-metadata">

### Author: ![markshannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/markshannon/32/72_2.png) [@markshannon](https://discuss.python.org/u/markshannon)
#### Post date: [December 6, 2023, 11:20am UTC](https://discuss.python.org/t/issues-setting-pyfunctionobjects-vectorcall-field-in-python3-11/39826/5 "2023-12-06T11:20:05Z")

</div>

See [PEP 590 – Vectorcall: a fast calling protocol for CPython | peps.python.org](https://peps.python.org/pep-0590/#third-party-extension-classes-using-vectorcall)

Specifically the part that says “must implement the `tp_call` function and make sure its behaviour is consistent with the `vectorcallfunc` function”  
By implication, if `tp_call` must be consistent with `vectorcallfunc`, then `vectorcallfunc` must be consistent with `tp_call`, and because they must be consistent the VM is free to call whichever one it wants.

In practice, we assume that the whichever form fits the given arguments best will be the fastest, so that’s what gets called.

Try comparing the behaviour of the calls `f(1,2)` and `f(*(1,2))`

---

<div class="post-metadata">

### Author: ![markshannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/markshannon/32/72_2.png) [@markshannon](https://discuss.python.org/u/markshannon)
#### Post date: [December 6, 2023, 11:23am UTC](https://discuss.python.org/t/issues-setting-pyfunctionobjects-vectorcall-field-in-python3-11/39826/6 "2023-12-06T11:23:38Z")

</div>

One other thing.  
Directly setting fields of internal data structures is unsafe and unsupported. You may be breaking invariants that you are not aware of.  
There is no such problem with calling an API function like `PyFunction_SetVectorcall`, which is why it was added.

---

<div class="post-metadata">

### Author: ![TannerFirl](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tannerfirl/32/11345_2.png) [@TannerFirl](https://discuss.python.org/u/TannerFirl)
#### Post date: [December 6, 2023, 2:54pm UTC](https://discuss.python.org/t/issues-setting-pyfunctionobjects-vectorcall-field-in-python3-11/39826/7 "2023-12-06T14:54:10Z")

</div>

Thanks for the reply, @markshannon !

I had read that invariant, but forgot to accomodate for it when creating my small reproducer. The PEP says:

```auto
Setting tp_call to PyVectorcall_Call is sufficient.

```

But even if I add this when I’m setting vectorcallfunc:

```auto
    Py_TYPE(args_func)->tp_call = PyVectorcall_Call;
    PyType_Modified(Py_TYPE(args_func));

```

It appears to have no effect.

I did rerun my small reproducer with python3.12 and PyFunction\_SetVectorcall and it works as expected, so that’s good.

I know setting vectorcallfunc for \<python3.12 is not technically supported, but any reason you can think of as to why that in conjunction with setting tp\_call to PyVectorcall\_Call and calling PyType\_Modified _wouldn’t_ have the desired effect? I’ve also tried setting “func\_version = 0” as done internally in python3.12’s PyFunction\_SetVectorcall, but no luck.

[PyFunction\_SetVectorcall PR](https://github.com/python/cpython/pull/92257/files#diff-57bc77178b3d6f1010dd924722c87522f224d93bc341f0e46c0945094124d8f2) for context.

---

<div class="post-metadata">

### Author: ![markshannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/markshannon/32/72_2.png) [@markshannon](https://discuss.python.org/u/markshannon)
#### Post date: [December 6, 2023, 4:34pm UTC](https://discuss.python.org/t/issues-setting-pyfunctionobjects-vectorcall-field-in-python3-11/39826/8 "2023-12-06T16:34:29Z")

</div>

As I said before, “Directly setting fields of internal data structures is unsafe and unsupported.”  
You can’t just set the `tp_call` field of `PyFunction_Type`.

You will need to create an entirely new class, setting `tp_call` _and_ `tp_vectorcall` of that class when you create it.

OOI, why do you want to do this?

---

<div class="post-metadata">

### Author: ![TannerFirl](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tannerfirl/32/11345_2.png) [@TannerFirl](https://discuss.python.org/u/TannerFirl)
#### Post date: [December 6, 2023, 4:47pm UTC](https://discuss.python.org/t/issues-setting-pyfunctionobjects-vectorcall-field-in-python3-11/39826/9 "2023-12-06T16:47:54Z")

</div>

> You will need to create an entirely new class, setting `tp_call` _and_ `tp_vectorcall` of that class when you create it.

Yes, this did come to mind, but I’m not sure its suitable for what I’m trying to do. What I’m trying to do is, with the C Python API, wrap individual python functions while not altering the behavior of the code. The problem with creating a new class is that any `type(wrapped_func)` in python code changes.

Appreciate you taking a look and giving this some thought!
