At the end of the day you will end up following in the footsteps of the conda, Fedora, Debian et al. Pick your poison.
The only thing you have to decide is what trade offs you will accept and what features you cannot do without.
If the only things that you package in pure python then you lose.
It you will not package C, C++, fortran, rust then you cannot win.
My experience is with packaging extensions like pysvn. That needs lots
of non-python dependencies built. I have the skills to do this, and I’m well
aware that other people do not have the time to learn these skills. They
just want to ship their cool python package.