[SOLVED] Porting Python module to CPython: how to translate global scope?

Python Help
1

Hello,

First post woo!

Let me start by pointing out that beyond very simple examples, I’m jumping off the deep end with C. I have some C# & Swift experience prior (also consider myself ~master but shy of Doctorate in pure Python) which has all helped me pick up on this pretty quickly, and I’ve already ported several functions (surprised myself with how fast I picked it up even). However, I’m stuck on this one issue pretty hard and not quite sure even how to google it (obviously have tried).

My module/extension needs some global dictionaries avaliable both in the module, and in Python.

To be very specific, from Bitbucket, I’m trying to express lines 36, 40, 44, and 47 via the CPython API. However, just putting e.g.:

PyObject* worldBlocks = PyDict_New();

in the global scope of my C does not work. The compiler throws:

world_test.c:17:25: error: initializer element is not constant

If I put the same code in a function that returns void, I get the same error. If I put it in a function that returns PyObject, then it works. This behavior baffles me, probably due to my awkward language background and overall lack of knowledge in C. Actually, I more or less understand why the above doesn’t work in C global scope, but not quite sure about why it doesn’t in a generic void function (discovered that on accident)… Anyway, still not sure how to get the dicts I need into the this mystical global land that certainly has to exist…

I’m lost on this one, which feels weird with how far I’ve come. I ported my terrain generator (Bitbucket) with relative ease!

Thanks for reading and for any insight!

2

Indeed. C is very restrictive about what you can initialize a global variable with. Basically you can only use a compile-time constant as initializer.

I doubt it. Please post the full code of that function.

3

Sorry, I guess you’re right - Popping those lines into a function that returns void works perfectly fine. Got my wires crossed somehow there, or perhaps made another error last night while I was trying things out.

My bad! I’ve tried a few more things with that in mind, but not getting anywhere. Tried adding a get function for each global dictionary, but it segfaults after a couple of calls. I was burned out so haven’t looked into that any further yet.

Still feels like I’m overlooking something. Much more googling lead me here, which may have the solution but it’s late and I’m tired so will read up on that tomorrow.

Cheers.

4

Unless you’re a C expert and have a special need for writing your extension module directly in C, I would recommend Cython.

5

If you want to create module-level globals, do so during module initialization.

In your init function, create the dict, then set it as a module attribute with PyModule_AddObject. This is how the os.environ dictionary is added to the os module for example (albeit that the PyDict_New() call is made in a helper function).

None of this requires a C global.

6

Thanks for the replies. I need access to the dictionaries in BOTH Python and C.

I did try Cython initially but I spent a week just to break even on performance. It became clear I’d need to not only learn many C concepts but also Cython syntax… Picking up C has been quite a bit easier and the returns on my time way better.

I think I"m on the right track, defining my dicts as C globals, initializing them in my PyMODINIT_FUNC function, then writing “get” functions (that also has to increment the reference counter) get them into Python.

The weird thing now, though, is that my test.py segfaults, but if I run the interpreter via gdb, I can’t reproduce the issue and everything seems to work perfectly.

7

Yes, which is why I pointed you to a real-world example. The os.environ object is exactly such a dictionary, it is created in a C-level module and accessible from Python too.

8

Sorry, I was on my phone when I replied, and overlooked that. I figured it out anyway and just came back to update that I was able to implement it. Edit: Thought it solved my segfault, but a bit more testing brought it back.

Thanks again!!

9

A quick look through your repository [https://bitbucket.org/experimentfailed/testcraft/src/working/world_c_test/world_test.c] you are missing several Py_INCREF calls which would more than likely be the cause of your segfaults.

Every call to PyTuple_SetItem steals a reference from the item that is set:

PyTuple_SetItem(return_tuple, 0, Py_None) // must incref Py_None

Also, the returned values from PyArg_ParseTuple are borrowed references. They too will need incref’ing when set.

PyTuple_GetItem also returns borrowed references.

10

Thank you… I had a sneaking suspicion it might have something to do with the reference counters, but since I couldn’t reproduce the issue with gdb, was feeling a bit helpless to solve it (I had actually already solved one such issue using gdb so thought it would show me where I’m making rookie mistakes like that).

I will go over everything again with that info in mind.

Update: Success! Martijn Pieters’ tip helped me get the global dicts setup properly and Jeremy Kloth’s tip solved my segfault. This is probably not the best way to do this, but I basically just printed before and after reference counts throughout the code, to look for unexpected results. Turns out, a PyListObject I was adding to a dictionary was losing reference, and need an Py_INCREF. For now, seems like the ref counts stay correct (actually, maybe some areas to Py_DECREF)!

Cheers and many thanks to all who replied…Everyone taught me something valuable!