# Define module constants in PyModuleDef

**URL:** <https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749>\
**Category:** Core Development\
**Created:** [November 14, 2020, 4:09pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749 "2020-11-14T16:09:34Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![tiran](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tiran/32/48_2.png) [@tiran](https://discuss.python.org/u/tiran)\
**Post date:** [November 14, 2020, 4:09pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/1 "2020-11-14T16:09:34Z")

</div>

Over the past few releases the module system has seen great improvements. Kudos to everybody who was involved in the design and implementation of PyModuleDef and multiphase module implementation.

There is one issue, which has not been addressed yet and that has annoying me for years: module level constants. The module API has helpers like `PyModule_AddStringConstant` and `PyModule_AddStringMacro`, but they are tedious to use. A common problem is also the lack of error checking. Some stdlib modules don’t verify the return values yet.

I came up with a proof of concept for a simpler and more readable approach. The idea is inspired by structmember. Module constants are declared in a new PyModuleConstants\_Def array. The array is part of PyModuleDef and initialized by the module initialization framework. The PoC defines a couple of types (int, str, None, boolean, float, function call) and helper macros.

> <https://github.com/python/cpython/pull/23286>

```auto
typedef struct PyModuleConstants_Def {
    const char *name;
    int type;
    union {
        const char *m_str;
        long m_long;
        double m_double;
        PyObject* (*m_call)(void);
    } value;
} PyModuleConstants_Def;

```

posix module example:

```auto
static PyModuleConstants_Def _posix_constants[] = {
#ifdef F_OK
    PyMC_LongMacro(F_OK),
#endif
    ...
    {NULL, 0},
};

```

math module example:

```auto
static PyObject*
m_inf_o(void)
{
    return PyFloat_FromDouble(m_inf());
}

static PyModuleConstants_Def math_constants[] = {
    PyMC_Double("pi", Py_MATH_PI),
    PyMC_Double("e", Py_MATH_E),
    PyMC_Double("tau", Py_MATH_TAU), // 2pi
    PyMC_Call("inf", m_inf_o),
#if !defined(PY_NO_SHORT_FLOAT_REPR) || defined(Py_NAN)
    PyMC_Call("nan", m_nan_o),
#endif
    {NULL, 0},
};

```

What do you think about the approach?

---

<div class="post-metadata">

**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\
**Post date:** [November 14, 2020, 8:30pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/2 "2020-11-14T20:30:23Z")

</div>

I like it. Some modules expose myriads constant. You need to write

```
if (PyModule_AddIntConstant(m, "P_WAIT", _P_WAIT)) return -1;
if (PyModule_AddIntConstant(m, "P_NOWAIT", _P_NOWAIT)) return -1;
if (PyModule_AddIntConstant(m, "P_NOWAITO", _P_NOWAITO)) return -1;

```

Or use macros and define variations of the following macros in every file:

```
#define ADD_INT(name, value) \
    do { \
        if (PyModule_AddIntConstant(m, name, value) < 0) { \
            return -1; \
        } \
    } while (0)

```

In some file you need `return -1;`, in others `Py_DECREF(m); return NULL;`, and in others `goto error;`.

With this proposition we will have standard macros available in every file and do not bother error handling.

Some attributes will still be added manually using `PyModule_AddObject()` or better its fixed variant `PyModule_Add()` ([https://bugs.python.org/issue42327](https://bugs.python.org/issue42327)).

---

<div class="post-metadata">

**Author:** ![tiran](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tiran/32/48_2.png) [@tiran](https://discuss.python.org/u/tiran)\
**Post date:** [November 15, 2020, 10:03pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/3 "2020-11-15T22:03:36Z")

</div>

I identified two more common patterns in module initialization: types and exceptions. My PR contains two helper functions:

```auto
PyTypeObject *
PyModule_AddNewTypeFromSpec(PyObject *module, PyType_Spec *spec,
                            PyObject *base)
PyObject *
PyModule_AddNewException(PyObject *module, const char *name, const char *doc,
                         PyObject *base, PyObject *dict)

```

PyModule\_AddNewTypeFromSpec() is PyType\_FromModuleAndSpec() + PyModule\_AddType(). It also handles non-tuple bases like PyErr\_NewException(). In most cases the base argument is either NULL or a single-item tuple. The extra check gets rid of tuple packing in the caller.

PyModule\_AddNewException() is PyErr\_NewExceptionWithDoc() + PyModule\_AddObjectRef() with correct short name. PyErr\_NewException\*() takes a qualified name but PyModule\_Add\*() expects a short name.

Example code:

```auto
    _hashlibstate *state = get_hashlib_state(module);

    state->EVPtype = PyModule_AddNewTypeFromSpec(
        module, &EVPtype_spec, NULL);
    return state->EVPtype == NULL ? -1 : 0;

```

```auto
    PySSLErrorObject = (PyObject *)PyModule_AddNewTypeFromSpec(
        m, &sslerror_type_spec, PyExc_OSError);
   ...
    PySSLZeroReturnErrorObject = PyModule_AddNewException(
        m, "ssl.SSLZeroReturnError", SSLZeroReturnError_doc,
        PySSLErrorObject, NULL);

```

---

<div class="post-metadata">

**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\
**Post date:** [November 16, 2020, 9:15am UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/4 "2020-11-16T09:15:32Z")

</div>

Thanks for taking this one! Generally looks OK. But:

- `PyModuleDef` is part of the stable ABI; we can’t just add a new member to it and expect it’ll be NULL if unset. Please add a `PyModule_AddConstants(PyObject *module, PyModuleConstants_Def *consts)` instead. (Extending/versioning `PyModuleDef` is a can of worms we’ll have to open eventually, but I don’t think it’s necessary _right_ now.)
- Could we use e.g. `PyModuleConst_` rather than `PyModuleConstants_`/`PyMC_`?

This week, my CPython day unfortunately falls on a national holiday. Next week I should have time to review more thoroughly.

---

<div class="post-metadata">

**Author:** ![tiran](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tiran/32/48_2.png) [@tiran](https://discuss.python.org/u/tiran)\
**Post date:** [November 16, 2020, 10:26am UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/5 "2020-11-16T10:26:43Z")

</div>

Thanks Petr!

I have renamed new API functions and constants to use `PyModuleConst_` prefix.

I see, stable ABI makes this tricky. How can we safely handle new struct members then? Or should I exclude the new feature from the stable ABI for now?

---

<div class="post-metadata">

**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\
**Post date:** [November 16, 2020, 10:46am UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/6 "2020-11-16T10:46:47Z")

</div>

Add a `PyModule_AddConstants` function. It will be useful even if/when `PyModuleDef` is extended – we already have functions like `PyModule_AddFunctions` that let you add only specific parts of the `PyModuleDef`.  
I don’t see a good way of excluding the feature from the stable ABI.

To „extend“ `PyModuleDef`, IMO we’ll need to add a new struct with a version number + some macro(s) to comfortably/automatically fill that version number, and new function(s) that use that new struct.

---

<div class="post-metadata">

**Author:** ![tiran](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tiran/32/48_2.png) [@tiran](https://discuss.python.org/u/tiran)\
**Post date:** [November 16, 2020, 11:10am UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/7 "2020-11-16T11:10:31Z")

</div>

I have another idea. There is no need to extend the `PyModuleDef` struct. We could also use `PyModuleDef_Slot` with a new slot type `Py_mod_constants`. This avoids an ABI mismatches.

My PoC PR now contains `PyModuleConst_` prefix, the requested `PyModule_AddConstants` function, and `Py_mod_constants` instead of `PyModuleDef.m_constants`.

---

<div class="post-metadata">

**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\
**Post date:** [November 16, 2020, 11:35am UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/8 "2020-11-16T11:35:26Z")

</div>

Please don’t. `PyModuleDef_Slot` is for functions. There are a few non-function slots, but in the long run I’d like to deprecate them. Mixing data pointers and function pointers is not valid standard C.

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [November 16, 2020, 2:40pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/9 "2020-11-16T14:40:12Z")

</div>

You can’t deprecate them until we deprecate the entire limited API, at which point it can all change anyway.

Until then, we’re stuck with only supporting platforms that can mix data and function pointers, so we may as well use it.

---

<div class="post-metadata">

**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\
**Post date:** [November 16, 2020, 4:10pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/10 "2020-11-16T16:10:15Z")

</div>

> [@steve.dower](#):
>
> You can’t deprecate them until we deprecate the entire limited API,

I think I can :‌) Expect a PEP by end of year. Sorry I’m slow – not only is there a lot of parts to think about, but also writing in English takes me a lot of time.  
The main idea is that it is the _ABI_ that is stable; pieces of the limited _API_ could be deprecated as long as the ABI still works. But of course, the devil is in the details.

---

<div class="post-metadata">

**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\
**Post date:** [November 16, 2020, 4:16pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/11 "2020-11-16T16:16:00Z")

</div>

Oh, and in case you’re interested, I do have a brainstorming repository for my PEP-to-be. Issues [#9](https://github.com/encukou/abi3/issues/9) and [#14](https://github.com/encukou/abi3/issues/14) there are relevant.

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [November 16, 2020, 11:15pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/12 "2020-11-16T23:15:31Z")

</div>

I mean, in as much as you can simply label it deprecated, sure. But that doesn’t change that you can’t do anything with how it behaves. So since we’re stuck with it, why make the rest of the API unnecessarily complicated?

---

<div class="post-metadata">

**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\
**Post date:** [November 17, 2020, 7:18pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/13 "2020-11-17T19:18:54Z")

</div>

Hm. I don’t think it makes the rest of the API necessarily complicated.  
If you disagree, add the slot – it’s not my place to forbid that. I’m just telling you is that it doesn’t fit in my long-term plans; if I stick around long enough, I’ll want to remove it again :‌)

---

<div class="post-metadata">

**Author:** ![tiran](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tiran/32/48_2.png) [@tiran](https://discuss.python.org/u/tiran)\
**Post date:** [November 17, 2020, 7:34pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/14 "2020-11-17T19:34:03Z")

</div>

Let’s postpone the decision. The new APIs are useful and convenient without the slot. I have updated the PR yesterday and removed the slot.

---

<div class="post-metadata">

**Author:** ![steve.dower](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steve.dower/32/56_2.png) [@steve.dower](https://discuss.python.org/u/steve.dower)\
**Post date:** [November 17, 2020, 8:56pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/15 "2020-11-17T20:56:04Z")

</div>

> [@encukou](#):
>
> I’m just telling you is that it doesn’t fit in my long-term plans; if I stick around long enough, I’ll want to remove it again :‌)

That’s totally fine, I want to remove them too (before we’ve added them) 🙂  
But we’re not going to be able to until we break the API completely, so I don’t see why we’d make things more complicated in the meantime (in this case, “more complicated” mean “some data objects provided one way, others provided in another way, even though they look like the same kind of thing”).

---

<div class="post-metadata">

**Author:** ![tiran](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tiran/32/48_2.png) [@tiran](https://discuss.python.org/u/tiran)\
**Post date:** [November 18, 2020, 11:39am UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/16 "2020-11-18T11:39:19Z")

</div>

“More complicated” == one additional one-line function per module

I can live with that trade-off if it makes Petr happy. ❤

---

<div class="post-metadata">

**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\
**Post date:** [November 18, 2020, 5:44pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/17 "2020-11-18T17:44:29Z")

</div>

Actually 3-4 lines with error handling. But I am fine with this.

---

<div class="post-metadata">

**Author:** ![tiran](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tiran/32/48_2.png) [@tiran](https://discuss.python.org/u/tiran)\
**Post date:** [November 18, 2020, 6:01pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/18 "2020-11-18T18:01:22Z")

</div>

No error handling required:

```auto
static int
math_init_constants(PyObject *module)
{
    return PyModule_AddConstants(module, math_constants);
}

```

```auto
    {Py_mod_exec, math_init_constants},

```

---

<div class="post-metadata">

**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\
**Post date:** [November 21, 2020, 10:37am UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/19 "2020-11-21T10:37:50Z")

</div>

Accepting both a tuple and a single class without packing it into a 1-tuple in `PyType_FromModuleAndSpec()` is a good idea. Opened [https://bugs.python.org/issue42423](https://bugs.python.org/issue42423) for it.

---

<div class="post-metadata">

**Author:** ![encukou](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/encukou/32/2461_2.png) [@encukou](https://discuss.python.org/u/encukou)\
**Post date:** [November 24, 2020, 4:14pm UTC](https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749/20 "2020-11-24T16:14:06Z")

</div>

I’m sure there are other viewpoints, but I generally consider the ergonomics of the C API less important than correctness. For ease of use, I’d rather have people switch to Cython :‍)  
The API added here is good mainly because it encourages more correct code: it eliminates lots of boilerplate error checking, which is often missing/wrong in the stdlib.