Choice of complex buffer protocol format intentional break with PEP?

[seberg (Sebastian Berg)]

@tim.one

I agree it’s easy to live with minor glitches for the single type raised in this topic. But it’s not “one and done”. An unexpected offshoot of the rise of machine learning apps is a proliferation of new floating-point formats, with ever less precision. There are two 16-bit floating types in wide use in that application area now, and NVIDIA has newer HW support for an 8(!)-bit floating type too.

Right, to me this is the discussion in: Buffer protocol and arbitrary (data) types - #13 by seberg, or the bfloat16 (or a new one).
This is important and I care about it a lot (I have to catch up on the bfloat16 one)!

For me, this thread was about a specific unintentional divergence for complex floating types and the constraints when changing the format specification.
From my perspective (I am not quite sure we all agree on it), adding D and or additionally asking to stop using Zd is an API divergence or break. Doesn’t mean much, except that it should be done with that awareness^[1].

Now, I don’t care too much about breaking with some parts of the PEP. The points about the inadequacy of both the PEP specificity and the also parts NumPy implementations aren’t wrong. So we have to try and use them and make progress from there.

I am a friend of trying to guess the impact:

• E.g. the X{} function pointer syntax is possibly used by no-one. You can get away with just saying it is invalid. Just in case, I would try to avoid re-using it with a different meaning, anyway (if mostly to avoid having to search if anyone might be using it after all).
• Weirder things like subarray alignment detail? Yeah, sorry we didn’t discuss it at the time… This is pretty niche in practice: It would be good to make sure there is a way to export any dtype in a way that is understood by both new and old NumPy behavior if we push for a change. But I expect it is niche enough that we could think about a DeprecationWarning, even if that isn’t technically a smooth transition, and to just accept if there is a longer divergence between NumPy and Python.
  • I.e. needs a bit of thought, but besides avoiding to use the same syntax for something different, I suspect we are in the clear.
  • Things in this category (probably): & (because I don’t really see it used as well),
• Zd is hugely adopted. So let’s just not break it or even “deprecate” it (until there is confidence practically everyone also supports D which will take years). And because it is used so much, I think it is best if Python itself also accepts it to avoid anyone having to choose which version is “right”.^[2]
  • g/Zg: I would start in this category, but it is likely niche enough to consider it the above.
  • t: I honestly think we would have to do a search to see how much it is used if we want to use it for something else.

(I personally don’t like to say “this is a bug” even if it clearly is, but sure, it can be used to tilt that towards “let’s just break it”. – except of course if others really need the bug to be fixed)

EDIT: sorry, if this is all obvious, I don’t mind to discuss other things, I suppose I am just looking for a bit of closure :).

---

1. I think Python ideally would have realized it may be that. Others may say that Antoine, I, or someone else should have seen the ctypes change or PEP note go through… It doesn’t matter, the only action item here is to amend the docs so that they don’t make it sound like a format change/extension is clearly unproblematic.] ↩︎

2. Unless someone looks at it, shrugs, and says: Yeah, we can just use Zd in Python core libraries as well, not a big deal. But that doesn’t sound likely – and I have neither thought it through to make that case, nor do I care all that much about it. ↩︎