I think we may want to start with something generic and use Pipelines for coordinating the environment itself.
Ideally we’ll have some collection of scripts that orchestrate the integration tests, and then the configuration files for Azure that specifies which of them gets run on what platforms.
I do agree that we almost certainly need to be closer to the metal than tox will give us, though. For one thing I kinda think tox should be one of the things under test, but also I predict we’re going to have a lot of trouble making sure that we’re actually using the right versions of everything here. Ideally, we’re going to want a pyproject.toml that says ["setuptools"] in it to pull the master version of setuptools, for example. Same with virtualenv, etc. It will probably be harder to make sure that’s happening with more layers of indirection around the build environments.
I agree, but I think “making sure that stuff runs in tox” is one of the tests that we should be doing. In other words, I agree we would probably be better not using tox as our test harness, but “projects running their CI under tox” is a big use case, and I think we should test that.
@pf_moore is right as the scope of the integration testing is to avoid breaking most users. is not to test what is convenient and is more about trying to cover most use-cases found in the wild.
I will be happy to help with integrations testing, especially with stuff like adding a travis job. I could even try to add some Zuul jobs definintions that would use OpenStack infrastructure to test it but I am not sure yet if I will get the approvals for that.
Any update on this (e.g. was this discussed at the PyCon packaging mini summit)?
The pypa/integration-test repo has not seen any action since this discussion petered out…