I believe this is pip specific terminology and pip specific CLI options (though other tools do copy pip’s interface).
So I think it would best be discussed on the pip GitHub issue page.
And while I do agree that no-binary and only-binary do occasionally cause confusion, the difference here is wheels vs sdists, not about whether a package is pure Python or not, and pip needs very good reason to make any breaking changes as it’s such a critically relied on tool.
I think you can rely on --no-binary not downloading wheels. As you mentioned, ‘binary’ here means ‘built’, and in practice means wheels, as pip never supported any of the other built package formats, and there’s not much reason to add support now.
I’d guess that what is cached and when it is used is an implementation detail of pip, so you shouldn’t rely on it reusing locally built wheels.