Document BINARY_OP_ opcodes?

Documentation
1

Hey there, I just spent a while trying to distinguish the difference between BINARY_OP_ADAPTIVE and
BINARY_OP_MULTIPLY_FLOAT. The dis docs only have BINARY_OP:

I did find BINARY_OP_MULTIPLY_FLOAT in the CPython codebase, but I couldn’t find BINARY_OP_ADAPTIVE in the CPython codebase at all, so I presume its name is dynamically concatenated.

What do you think of documenting the various BINARY_OP_ opcodes? Are there too many?

Similarly wondering about LOAD_CONST__LOAD_FAST and RESUME_QUICK.

2

I wasn’t aware any opcodes were documented outside of the source code. That said, there seems to be an error in the BINARY_OP documentation, namely the introduction version. That opcode (ignoring the variants you mentioned) has been around for as long as I’ve been messing around with Python (1.0 timeframe). It was probably part of the first version Guidon released. My guess is that one or both of the variants you mentioned were added in 3.11.

I say go for it. I’m sure a PR would be welcome. That said, if the dust from major virtual machine rework hadn’t settled yet, you might well be playing a game of whack-a-mole for a while.

3

Where did you search? In the 3.11 sources, BINARY_OP_ADAPTIVE (and many other _ADAPTIVE) opcodes exist; in main (to become 3.12) they no longer exist.

All the opcodes you mention are “specialized” opcodes, and will never be seen in the co_code attribute of code objects when requested from Python code. They are inserted dynamically in Python/specialize.c. The numeric values of these are dynamically calculated in code that is intentionally hard to find (but the values are revealed in Include/opcode.h). You might find a little more background information about this topic in PEP 659 (Specializing Adaptive Interpreter).

Basically, adaptive opcodes are a stepping stone towards specialization. LOAD_CONST__LOAD_FAST and friends are “super-instructions” that replace a pair (here, LOAD_CONST + LOAD_FAST) but save a little of the dispatch effort.

I’d be reluctant to document any of these, since there’s such a small audience for the documentation, and things are changing quickly in 3.12. I’m curious what led you to find these in the first place?

4 Likes
4

Actually, what has been around through 3.10 was separate opcodes BINARY_OP_ADD, BINARY_OP_MULTIPLY, and so on. In 3.11 these were combined into BINARY_OP, with the oparg indicating which operation to perform. So I think the version mentioned in the docs is correct.

5

Yup; the opcodes are all documented in the 3.10 docs, and only removed in 3.11, as we documented in the opcodes section of the What’s New in 3.11. So all appears to be in order here.

6

Doh, thanks for the gentle correction. LBYL would have been appropriate here. My synapses are wearing out. I ask forgiveness. :wink:

1 Like
7

I feel you – it’s the same for me.

8

Ah, good point, I only checked main for the opcode names, I didn’t check the 3.11 tree.

I discovered them as I made a little tool (https://www.dis-this.com/) for running dis online with Pyodide, and I added 3.11 support this week with a toggle to try the specializing adaptive interpreter (adaptive=True). My tool hyperlinks the opcodes to the reference, and that’s when I realized the specializing opcodes weren’t in the reference.

I realize my tool is very niche - it was inspired from how I would sometimes run dis in my Python class when students wanted to understand the difference between two syntaxes that achieved the same result. I mostly use it now when I am just curious. It’s like seeing into the Matrix.

Anyway, if it’s not appropriate to document those opcodes due to their high likelihood to change, that’s fine. I just wanted to make sure the lack of documentation was intentional. Thanks!

2 Likes
9

Yeah, it’s definitely intentional (though we can argue about it).

Maybe your tool can link to the source for the specialized opcodes. :slight_smile:

Pretty cool tool BTW. Feature request: select the Python version.

2 Likes
10

Good idea! You can now change the Python version: https://www.dis-this.com/
I have to reload the whole page to change the version, since Pyodide leaks into the window globals and it’s too messy to load a new Pyodide version after. Also, notably, Pyodide isn’t designed to stably support multiple versions (see discussion) - but hey, it works. :slight_smile:

As suggested, I also made it so the links go out to ceval.c for the undocumented opcodes in 3.11.
If you’re curious, here’s a list of undocumented opcodes:

['BINARY_OP_ADAPTIVE', 'BINARY_OP_ADD_FLOAT', 'BINARY_OP_ADD_INT', 'BINARY_OP_ADD_UNICODE', 'BINARY_OP_INPLACE_ADD_UNICODE', 'BINARY_OP_MULTIPLY_FLOAT', 'BINARY_OP_MULTIPLY_INT', 'BINARY_OP_SUBTRACT_FLOAT', 'BINARY_OP_SUBTRACT_INT', 'BINARY_SUBSCR_ADAPTIVE', 'BINARY_SUBSCR_DICT', 'BINARY_SUBSCR_GETITEM', 'BINARY_SUBSCR_LIST_INT', 'BINARY_SUBSCR_TUPLE_INT', 'CALL_ADAPTIVE', 'CALL_PY_EXACT_ARGS', 'CALL_PY_WITH_DEFAULTS', 'COMPARE_OP_ADAPTIVE', 'COMPARE_OP_FLOAT_JUMP', 'COMPARE_OP_INT_JUMP', 'COMPARE_OP_STR_JUMP', 'EXTENDED_ARG_QUICK', 'JUMP_BACKWARD_QUICK', 'LOAD_ATTR_ADAPTIVE', 'LOAD_ATTR_INSTANCE_VALUE', 'LOAD_ATTR_MODULE', 'LOAD_ATTR_SLOT', 'LOAD_ATTR_WITH_HINT', 'LOAD_CONST__LOAD_FAST', 'LOAD_FAST__LOAD_CONST', 'LOAD_FAST__LOAD_FAST', 'LOAD_GLOBAL_ADAPTIVE', 'LOAD_GLOBAL_BUILTIN', 'LOAD_GLOBAL_MODULE', 'LOAD_METHOD_ADAPTIVE', 'LOAD_METHOD_CLASS', 'LOAD_METHOD_MODULE', 'LOAD_METHOD_NO_DICT', 'LOAD_METHOD_WITH_DICT', 'LOAD_METHOD_WITH_VALUES', 'PRECALL_ADAPTIVE', 'PRECALL_BOUND_METHOD', 'PRECALL_BUILTIN_CLASS', 'PRECALL_BUILTIN_FAST_WITH_KEYWORDS', 'PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS', 'PRECALL_NO_KW_BUILTIN_FAST', 'PRECALL_NO_KW_BUILTIN_O', 'PRECALL_NO_KW_ISINSTANCE', 'PRECALL_NO_KW_LEN', 'PRECALL_NO_KW_LIST_APPEND', 'PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST', 'PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS', 'PRECALL_NO_KW_METHOD_DESCRIPTOR_O', 'PRECALL_NO_KW_STR_1', 'PRECALL_NO_KW_TUPLE_1', 'PRECALL_NO_KW_TYPE_1', 'PRECALL_PYFUNC', 'RESUME_QUICK', 'STORE_ATTR_ADAPTIVE', 'STORE_ATTR_INSTANCE_VALUE', 'STORE_ATTR_SLOT', 'STORE_ATTR_WITH_HINT', 'STORE_FAST__LOAD_FAST', 'STORE_FAST__STORE_FAST', 'STORE_SUBSCR_ADAPTIVE', 'STORE_SUBSCR_DICT', 'STORE_SUBSCR_LIST_INT', 'UNPACK_SEQUENCE_ADAPTIVE', 'UNPACK_SEQUENCE_LIST', 'UNPACK_SEQUENCE_TUPLE', 'UNPACK_SEQUENCE_TWO_TUPLE', 'DO_TRACING', 'HAS_CONST(op)', 'NB_ADD', 'NB_AND', 'NB_FLOOR_DIVIDE', 'NB_LSHIFT', 'NB_MATRIX_MULTIPLY', 'NB_MULTIPLY', 'NB_REMAINDER', 'NB_OR', 'NB_POWER', 'NB_RSHIFT', 'NB_SUBTRACT', 'NB_TRUE_DIVIDE', 'NB_XOR', 'NB_INPLACE_ADD', 'NB_INPLACE_AND', 'NB_INPLACE_FLOOR_DIVIDE', 'NB_INPLACE_LSHIFT', 'NB_INPLACE_MATRIX_MULTIPLY', 'NB_INPLACE_MULTIPLY', 'NB_INPLACE_REMAINDER', 'NB_INPLACE_OR', 'NB_INPLACE_POWER', 'NB_INPLACE_RSHIFT', 'NB_INPLACE_SUBTRACT', 'NB_INPLACE_TRUE_DIVIDE', 'NB_INPLACE_XOR', 'HAS_ARG(op)', 'IS_ARTIFICIAL(op)']

I think at least one of those, IS_ARTIFICIAL, will never show up in a disassembly.

11

Most of that looks expected. How did you compute this data? It looks like you scanned opcode.h. HAS_ARG(op) also looks like a macro you accidentally picked up from there.

Also, I had no idea you were using Pyodide! That’s amazing. Of course, now I am getting greedy and I want the nightly 3.12 build too (I don’t even know if we have that, the latest alpha would do) but I suggest that you don’t put any effort in that, there’s only literally a handful of folks who care (mostly the Faster CPython team at Microsoft). It’s already very useful to be able to compare 3.10 to 3.11.

12

Yeah, I just computed it with a little script:

The script didn’t distinguish between opcodes and macros properly.

I don’t think Pyodide has any 3.12 support yet, from my scanning of the repo, so I wouldn’t be able to add that. I’ll watch the repo to see when they add that. Currently the 3.11 support isn’t even a numbered Pyodide release, it’s the “dev” branch, but 3.11 should be released as 0.23.0 of Pyodide soon.