Pinning GNU Autotools using the GitHub container registry

TL;DR

@corona10 and I want to manage our GNU Autotools dependency using a custom container hosted by GitHub’s container registry.

Background

Until recently, we used Christian’s Podman/Docker image to regenerate our configure script. The container image was hosted on quay.io. The reason for this, was that we had standardised on a particular distribution of GNU Autotools[1]. After we migrated from GNU Autoconf 2.69 to 2.72[2], we decided to use the stock Ubuntu 22.04 GNU Autotools, mostly in order to reduce our CI dependency on an external web service (quay.io). @vstinner created a wrapper script[3] (based on Christan’s container), which is what we’re currently using.

Limitations with status quo

Currently, we fire up a stock Ubuntu 22.04 container, run some apt commands to install GNU Autotools, and then we finally regenerate the configure file every time we run the regen script. If we want to upgrade the GNU Autoconf or autoconf-archive version, we have to fetch sources, configure, build, and install the packages, and at last regenerate the configure file every time we run the regen script. In practice: if we want a more recent GNU Autoconf, our regen script becomes larger and slower, as we have to do more things for each invocation.

Proposal

Donghee and I propose to go back to Christian’s approach of using a custom container, but instead of quay.io, host the repo on the python GitHub org[4] and use GitHub’s container registry for hosting the container. This should make configure regeneration faster, and it will be easier to pin and upgrade the various parts of the GNU Autotools suite.

Donghee has a proof-of-concept image, based off of Christian’s container image. Try it out now:

$ podman run --rm -v$(pwd):/src ghcr.io/corona10/cpython_autoconf:272-final

  1. earlier, the configure script would be generated by whatever people had installed in their development environments ↩︎

  2. in the 3.12 alpha phase IIRC ↩︎

  3. cpython/Tools/build/regen-configure.sh at main · python/cpython · GitHub ↩︎

  4. python/autotools? python/buildtools? ↩︎

6 Likes

You beat me to this idea! :slightly_smiling_face: I was going to propose we start having dev-oriented containers under some repo so we could have them for general development of CPython (i.e. cpython/.devcontainer/Dockerfile at main · python/cpython · GitHub as its own container). And then we could use that container in cpython/.devcontainer/devcontainer.json at main · python/cpython · GitHub so it wasn’t in any way confined to GitHub codespaces. We could even go so far as to use the container in CI or anywhere else where we needed an easy way to make sure we had (at least) a container with all the necessary bits to build and test everything under Linux.

And as part of this I wanted to do the autotools container as well. So I’m totally supportive of this and having other containers that are dev-specific in some repo where we automatically publish the images for people to use.

3 Likes

You’re taking it to the next level: devcontainers :smile:

A general container repo sounds like a good idea.

1 Like

Bikeshedding :art: :bike: :

  • python/containers
  • python/devcontainers (I think I prefer this one)

Any better?

We could resurrect the “devinabox” name. “cpython-dev”, “cpython-devcontainers” also work. My concern of the more generic names is it won’t be self-describing for your fork (e.g. what’s brettcannon/devcontainers?).

1 Like

cpython-devcontainers sounds good to me.

A container to run Python tests in a CI and a container to develop Python sound like two very different use cases, no?

1 Like

And through the magic of containers they can stack on top of each other! :wink:

Seriously, it really depends on what you put in the containers as to whether they are massively different or one just adds extras on to the other. I mean how many times do we need to specify a C compiler and make?

1 Like

It is fine to use a container with the correctly pinned GNU Autotools for multiple use cases :wink:

1 Like

It seems there is no pushback on this, so let’s get it done. IIRC, we’ll need to open an issue on the SC tracker in order to get permission to create a new repo under the python GitHub org.

Any better suggestions?

1 Like

I’d probably avoid the cpython- prefix, many repos under the org relate to CPython but aren’t prefixed, and you can always rename your fork if you like to avoid ambiguity.

If prefixing, a hyphen usually works better than an underscore (for ctrl+left and suchlike) and looks like more repos at Python · GitHub have hyphens than underscores.

Might there be other containers in general than devcontainers?

2 Likes

I proposed cpython-devcontainers on the SC repo ticket.

BTW, the SC approves.

2 Likes