The discussion here on “default extras” is probably relevant, too. But it’s essentially arguing for some sort of “extras that remove dependencies” feature, and one of the reasons I find the discussion confusing is precisely that “extras removing things” feels weird to me.
So +1 on this proposal specifically. Also I’d be in favour of a more general principle that “extras only add stuff”. People looking for a more general “manipulate dependencies at install time” facility should come up with a design that isn’t constrained by how extras currently work. Such a design may well (probably should) make extras obsolete. @steve.dower’s selector packages idea may be worth reviewing in that context.