I don’t disagree, and my usage of ‘us’ in the OP does not preclude the vendors from participating; after they all they are part of the community too. My purpose in starting this topic was to at least inspire a discussion, and get some sort of ‘interest group’ formed who could define the parameters and expected results of what this sort of testing would need to be. If people who are not vendors choose to test and report results, that’s fine, but if the vendors do it that’s even better, and if we have some of both, that’s still a win in my book.
One reasonable outcome of such an ‘interest group’ could be indicating that vendors of such tools need to specify that they are “PEP 503” compliant, and not just “support Python package repositories”, and that they’ve used some sort of community-blessed test suite to verify their compliance.