The solution you suggest does not work well for sharing objects between libraries, I give an example here:
Python packaging works really well for the use case of allowing third party libraries (like numpy) to provide foundational basis and be the lingua franca for other third party libraries, installing multiple versions of the same library would break that.