Make one Windows CI job mandatory

Previously, Python had a voting (mandatory) Azure Pipelines job. I asked to disable it since all of its jobs are now available as GitHub actions which seem better integrated into GitHub, and each job was run twice (once in Azure Pipelines, once as GH Action). Also, Azure Pipelines CI was the slowest CI, there is no way to re-schedule a failed job (only workaround: close/reopen a PR which re-schedule all CIs, not only Azure Pipelines), and it started to fail randomly.

Brett made Azure Pipelines CI non-voting:

Then Python had no more voting CI :frowning: So I asked to make Travis CI and Windows (64 bits) (GH action) voting:

The Windows GH action is smart and is not scheduled if a PR is a “documentation only” change. Problem: GitHub still requires the Windows job to pass even if it’s not scheduled… which never happens and so a documentation PR can no longer be merged :frowning: It seems like the Windows job should be made non-voting again.

Brett Cannon explained the problem:

Actually, I just realized we can’t make these status checks required because they don’t always run. :slight_smile: Our Actions are smart enough to not run when they aren’t necessary, i.e. doc changes don’t run the rest of the checks. And so making the OS-specific tests required would block doc tests.

Basically we would either have to waste runs on things that aren’t really necessary but then be able to require runs, or we have to just rely on people paying attention to failures.

I’m personally for the latter.

I asked if we could modify the Windows job to implement the “doc only” change in the job, rather in the GH action scheduler: as done by .travis.yml file.

Steve Dower replied:

It would make the job definition significantly more complicated, and I don’t want to do that just to work around an issue with github until we’ve got positive confirmation that the behaviour is intentional and won’t change.

The current state is that there is no more voting Windows job on PR. Core developers should pay attention to CI failures and not merge a PR if a Windows job fails.

One issue is that technically, it remains possible to merge a CI even if it breaks Python on Windows. I can happen if a core dev clicks too quickly, or… intentionally.

Well, one practical solution is to change GH Actions to run alwats run Windows jobs: run them even for “documentation only” changes. It will waste a few resources, but it’s easy to implement and will prevent to break Windows by mistake.

That can be enhanced later if needed.

1 Like

@steve.dower, @brettcannon: What do you think of this short term solution?

Better than not requiring the check at all.

Make sure you explain to python-dev why the change has happened.

Does someone know how to always run the Windows GitHub action?

Delete But that is not Windows-specific; it will apply to all CI runs for builds.

For reference, there is an issue on the Github support forums tracking this and so far it seems like the response from 6 months ago was

Though I can’t guarantee anything or share a timeline for this, I can tell you that it’s been shared with the appropriate teams for consideration.

So I think we can safely assume that Github won’t be fixing this themselves any time soon.

There is a workaround someone posted here that might be worth looking at:

1 Like

That workaround looks roughly like what is in the Azure Pipelines files. Someone just needs the time to go and migrate it.

Oh thanks, I didn’t know how to do that. I created:

I would appreciate a review for my CI configuration PR :wink:

FYI I’m buried in work ATM so someone else will need to be the driver on this.

I merged my PR to always run GH Action build jobs. Then Filipe Laíns found a way to again skip them for documentation only changes. In short, we don’t use more nor less CI resources than before, but it became possible to make a GH Action job mandatory!

@Mariatta proposed to wait 2 weeks to see if the Windows x64 job is stable and backport these GitHub Action workflow configuration to 3.7 and 3.8 in the meanwhile (done for 3.8, on-going for 3.7). I’m fine with this plan, it’s reasonable :slight_smile: