If you distribute wheels, you can distribute them with the compiled extension where it works, and a pure python wheel that will be used as a fallback when there’s no platform-specific wheel available.
Since you’re using setuptools, it is possible for you to override the build_ext command, such that if there’s a failure, it is ignored. I’m not a 100% sure if that’s a good fit for what you want to do here.