The shared venv is out of the way of scripts that the distro provides, and which the distro maintainers consider to be part of the operating system. Even if you only did user-level installs, something you installed could for example shadow a name that a system script wants to import, and cause that script to break (hence “break system packages”). And of course if you don’t want a --user install then you’ll need root privileges in order to unpack the files, and that means setup.py could run as root, too.
This can be quite serious - e.g. the system package manager itself may have Python components (I know this is true for Debian’s apt, and I think I saw in the PEP that it’s true of Fedora’s dnf too).
On Linux, Python doesn’t come with Linux to save you the step of installing it like Windows users have to do. It comes with Linux because parts of Linux (well, GNU, really) need it.