I think I’ve managed to track down what’s going on here. First of all, the reason that this works with pip is because pip is actually still using distutils, at least for Python 3.9 and older. See here where that decision is made. So this explains why pip works and my solution doesn’t.
Pip does basically the same thing that Spack does, and installer is planning on doing. The main method of interest is get_prefixed_libs, which calls either _distutils.get_prefixed_libs or _sysconfig.get_prefixed_libs depending on Python version. The _sysconfig method queries the purelib/platlib attributes and replaces base and platbase with the prefix of interest. However, the _distutils method calls get_python_lib which is hard-coded to provide the proper prefix.
The pip source code actually alludes to the exact issue I reported above, that the Python shipped with XCode is broken and doesn’t work with sysconfig, so distutils is currently the only way to get this working. I’ll likely try to find a workaround for this in Spack.