Agreed, I think you could make this work with extras. @dustin @pradyunsg @ofek do you see environment markers (or some modifications to build tags) as a more targeted solution here?
Poetry frequently runs into users struggling with ML packages; we have felt for a long time there is a strong case for GPGPU wheel tags.