Most of your arguments are against float (IEEE 754 binary64) itself. The limitations of floats are well-known, in Python and other languages, and aren’t limited to TOML.
TOML made the choice to use IEEE binary64. Despite all its shortcomings, it’s a useful implementation of floating-point numbers, and it’s nearly universally available in programming languages. Since TOML is designed for interoperability, binary64 makes sense.
The fact that some builds of CPython may not have binary64 floats is the only other reason to keep parse_float. But it’s not a very practical reason: those builds are extremely rare.
Without those quotes, I stay convinced that parse_float is second-guessing the TOML designers, and it’s not something the stdlib should do. (But it is of course perfectly fine for tomli to do it – I can imagine the “interface between a non-tech-savvy person and an application/developer” argument carries a lot more weight there.)