3.14.0a2 rebuild needed for 0x03070000 limited api c extension

I maintain the rl-accel · PyPI module which is an optional accelerator for the reportlab toolkit.

WHile testing latest Python-3.14.0a2 release I noticed a test failure involving reference counts for one of the functions in this extension.

It is built as a limited api with value define_macros=[(“Py_LIMITED_API”, 0x03070000),

A round of experiments shows that the released extension behaves correctly for Pythons 3.9 - 3.13, but I see refcount differences in 3.14.0a2.

However if I rebuild the extension with python 3.14.0a2 then it works OK in 3.9 - 3.14.

Should I expect this and what constructs might be causing it?

You should probably file an issue. Mark and/or Brandt might know more.

But honestly, expecting specific refcounts is not going to work in the future anyway, due to changes in GC implementation and free threading.

3 Likes

It’s likely due to object immortalization. Objects marked as immortal won’t have their refcounts adjusted by modern Py_INCREF/DECREF macros, but the older Py_INCREF embedded from the 3.9 headers will still modify them (just hopefully not by enough to cause the object to be deallocated). So the refcounts will only be adjusted by your code, but not the runtime.

Building with the 3.14 (stable ABI) headers is going to use the exported Py_IncRef function, which will be slower across the board but will get the counting correct. (Without stable ABI you’ll get a fast macro that does the right thing.) This particular case works backwards by a few versions, but the official guidance is that you should build with the earliest headers that you want to support and so set the Py_LIMITED_API macro to match Py_HEX_VERSION.[1]

So most likely the answer is that you need to be more selective with refcount tests and avoid counting built-in objects entirely. I’m sure sooner or later we’ll let people immortalize arbitrary objects[2] which will make things even more complicated for these kinds of tests, but until then you should be able to be more targeted. This will require some kind of test helper extension that is version-specific, as there’s (deliberately) no stable ABI way to get the precise refcount, but it’s totally doable.


  1. Which, for the record, I think is unfortunate and we didn’t have to do it this way, but it’s done now. ↩︎

  2. FTR, also not a fan of this. ↩︎

I think you are right, but exactly how to determine if I should deal with simple string attributes is going to be a problem this produces different results for some common values

if __name__=='__main__':
	from sys import getrefcount
	class A:
		def __init__(self,encName='utf8'):
			self.encName = encName
			print(f'getrefcount(self.encName={self.encName!r})=={hex(getrefcount(self.encName))}')
	a=A('PDF')
	a=A('utf8')
	a=A('utf-8')
$ python311 tmp/trc.py 
getrefcount(self.encName='PDF')==0x7
getrefcount(self.encName='utf8')==0x9
getrefcount(self.encName='utf-8')==0x7
 
$ python312 tmp/trc.py 
getrefcount(self.encName='PDF')==0x7
getrefcount(self.encName='utf8')==0xffffffff
getrefcount(self.encName='utf-8')==0x7

$ python313 tmp/trc.py 
getrefcount(self.encName='PDF')==0x7
getrefcount(self.encName='utf8')==0xffffffff
getrefcount(self.encName='utf-8')==0x7

$ python314 tmp/trc.py 
getrefcount(self.encName='PDF')==0x7
getrefcount(self.encName='utf8')==0xc0000000
getrefcount(self.encName='utf-8')==0x7

My problem came with the ‘utf8’ value so it seems that the older stable builds cannot cope with a modified refcount definition. I assume ‘utf8’ is special because of the codecs. Makes a nonsense of the original limited stable api promise though.

I think a later python build will cover things for my specific test until the next ‘optimisation’. The 3.14.0a2 build works OK for 3.9-3.13.

I wonder how to check that a C extension is not leaking in this modern era?

I don’t think the limited API ever promised that reference counting would be reproducible and the same.

I don’t think there’s a formal way of telling that an object is immortal from Python. Practically though, if the refcount is greater than 1 << 30 then you can assume it is. In this case you should treat any reference count changes as meaningless.

3 Likes

Well I think I must have misread the documentation in the 3.7 version I see this

become hidden from the extension module; in return, a module is built that works on any 3.x version (x>=2) without recompilation.

however in 3.13 I see this

Limited API, is compatible across several minor releases.

although later in the 3.13 doc I see this

The extension will work without recompilation with all Python 3 releases from the specified one onward, and can use Limited API introduced up to that version.

I guess what’s not being said is that some effects may still be noticed when the minor versions change. The extension doesn’t crash and appears to work, but some differences are visible (particularly if they are internal details).

The failing test was an attempt to to ensure that the C version of some code did not differ in refcount behaviour from an equivalent python code. I suppose I have to bite the poisoned apple and check for overlarge refcounts and assume thos ought not to matter.

As Guido says above it’s probably stupid to try and check reference numbers etc etc.

Well, that’s part for the course with any Python version bump. Error messages may be improved, reference counts may change, repr’s of standard library types may become more informative… these things are not part of the cross-version compatibility guarantees, and users should not rely on such details in their code.

Edit:

Ah, sure. Well, such low-levels tests probably need to be implemented more subtly then…
For example, skip the test if the object being tested with turns out to be immortal?

Right, I modified the implementation of functions related to reference counting to implement these functions as “opaque function calls”: Py_TYPE() and Py_REFCNT() are opaque function calls in limited C API 3.14.

Would it be a reasonable workaround for you to build your C extension on a recent Python version?

I agree that this wording is unfortunate and misleading.

I opened issue #127253 to clarify that the stable ABI is about ABI compatibility. Behaviour changes are still governed by PEP-387 (and of course by the judgement of the core devs, who should try to not break users).

1 Like

Not sure what happens if I start releasing a cp314-abi3 module before python 3.14, but it’s not a big deal for reportlab now as the C extensions that we build are being deprecated.

My experiments suggest cp314-abi3 works in earlier pythons so probably I should dive back into the cibuildwheel again?

I can fix the cp37-abi3 test by just ignoring refcounts&0xc0000000 != 0.

If we do go back to C what’s the recipe for detecting errors in Py_INC/DECREF, presumably no changes will be observed in immortals and others have to match a python equivalent.

FYI last month, immortal reference count changed: commit.