FWIW, this is actually how Spack environments work.
Each package is installed in its own prefix with a unique hash, and packages “in” and environment are symlinked into a common prefix for that environment. There will be only as many installs of each package as you have unique configurations for it.
We also do the virtualenv trick, where we copy the python interpreter (and a few other things) into place so that python thinks it lives in the env, but everything else is just pointers back to the original packages. Practically speaking, that just means you can use pip in a spack env if you want to mix the two.