Add support for devcontainers to facilitate GitHub Codespaces usage?

As announced earlier this month, every GitHub user now gets 120 free hours of Codespaces compute (GitHub’s hosted compute solution). That means you can get:

  • 60 hours with a 2-core instance
  • 30 hours with a 4-core instance
  • 15 hours with 8 cores
  • ~7 hours with 16 cores

The interesting thing here is that the 2-core instance is definitely enough for a week of sprinting and not run out of quota! Because of this, I would like to propose we make using Codespaces easy by adding the appropriate devcontainer configuration files to GitHub - python/cpython: The Python programming language .

First, a disclaimer: I do work for Microsoft on the VS Code team. That means I work for the company providing the free compute and on the team that defines the devcontainer spec. I do not work on Codespaces, but the VS Code team itself does work with them closely (not me personally, though). But I am doing this because I think it would be a benefit for us, not to sell anyone anything.

With that out of the way, what I’m proposing is that we (at least) add a devcontainer.json to the repo that specifies how to build a container to develop for CPython. The nice thing about doing this is we can make sure that contributors are set up from the start with the right tools to develop for CPython. This can even include setting up autoconf appropriately for even seasoned core developers. :wink: We can even get toolchains like for WebAssembly set up in the container so that even that is supported. The key thing here is it would let us make sure a Linux environment is always available that people can use to develop for Python without having to figure out what to install. And this works both locally via Docker or in Codespaces, so a potential win even for those not wanting to use hosted compute from GitHub. And the devcontainers.json can pretty much just point to a Dockerfile that we maintain, so there isn’t any vendor lock-in here.

After that we can consider setting up pre-builds for Codespaces. What that does is it re-builds the container image via a GitHub Action whenever we want; we can have it do it after every commit, once a day, whatever. But the key thing is we can make it such that building CPython from a pre-built devcontainer is extremely fast (if not instantaneous if we pre-build after every commit so there’s actually nothing to build until you edit the code). Much like making sure new contributors have the tools they need to build CPython with devcontainer.json, this would help facilitate new contributors by getting them going even faster and knowing that the build was already successful and usable.

Finally, and this might be the most controversial, we could configure the devcontainer for VS Code / https://vscode.dev . Since devcontainers can contain tool-specific settings we can add VS Code settings and extension recommendations so that folks have appropriate settings for things like indentation, etc. The only reason I’m suggesting this is that with https::/vscode.dev / https://github.dev it would allow folks to develop entirely via a browser during sprints using Codespaces, no local setup required. To be clear, though, this is not required and you can even use Codespaces with other editors like PyCharm. But as I said, this would let someone to show up with a Chromebook at a sprint and be able to get work done quickly.

So, what do people think? I know we have a general policy of not putting tool-specific stuff in the repo, but I think the benefits of all of this are great enough to consider at least the devcontainer.json bit and probably the pre-builds. Obviously I would love to also add the stuff to make using https://vscode.dev work, but I understand if people feel that’s too tool-specific.

8 Likes

I like it.

FYI, Microsoft provides devcontainer config for cpython now. So we can start using Codespace for cpython already.
Adding devcontainer config to our repository means we maintain it instead of Microsoft.

This is devcontainer config for cpython.

2 Likes

Correct, we can make changes faster and they can be appropriate to the Python version in case we change things, e.g. what version of autoconf we want to use.

2 Likes

I talked this idea over with the SC to make sure there wasn’t any inherent bias on my part. In the end, the discussion came to the conclusion that this was a reasonable idea to follow through with for the benefit of sprints and people wanting to make quick edits.

3 Likes

I have a branch at cpython/.devcontainer at dev-container · brettcannon/cpython · GitHub configured. I am just testing pre-builds to make sure they do what I expect them to do, and then I will open a PR.

3 Likes

Is it Codespace-only? Or are you testing general devcontainer environment?

As far as I know, workspaceFolder is mandatory for general devcontainer, but not for Codespace.

General.

It all works fine for me locally via Docker, so I don’t think it’s necessary.

Opened the PR at GH-102973: add a dev container by brettcannon · Pull Request #102975 · python/cpython · GitHub (which also has an accompanying issue).

2 Likes

I merged the PR and I just turned on prebuilds!

6 Likes

Where do we stand on documentation on how to use this for newbies in the devguide / getting started?

3 Likes

Wherever you want if that gets you to write a PR. :wink:

At this point I am either waiting for one or two last tweaks or someone to try it out and tell me the current setup worked for them.

FWIW, apparently at least several people I talked to used it at the PyCon sprints and found it helpful for a quick setup, though apparently at least some of the documentation-related dependencies were missing from the default image if I recall correctly from the person reporting an issue with that.

Is there a specific issue tracking that? The image literally does make --directory Doc venv html, so if that doesn’t work then it’s a bigger issue than the dev container.

I’m not aware of one, sorry, nor can I recall the exact details, unfortunately—at the time I was helping the contributor with the issue, it was nearing the end of the sprints and I was heavily task-saturated and pretty exhausted after the nearly two-week-long PyCon trip. If I remember correctly, it was with an optional dependency that was not required to build the docs normally but something else related. I believe it might have been @TysonClugg who ran into the issue so if I’m remembering right, perhaps he could provide some more feedback here?

Step back a bit: Assume I literally don’t even know how to open a github codespace. That’s what I mean by documentation.

4 Likes

I know you did, but that doesn’t stop others who know about it to try it out and provide feedback before me (or anyone else) sinks the time into documenting how to use it.

I’ve used the codespaces as someone with minimal experience working with the CPython repo, and minimal experience with codespaces. I’ve found it to be really convenient to use, so thanks!

One thing i’m a bit confused about is whether I should be creating the codespace off my fork or on the main repo. Creating it off my own fork feels more intuitive since that’s what I’d be pushing to, but this doesn’t seem to allow you to use the prebuilds as far as I can tell, which made the creation pretty slow.

After the container was created things worked pretty nicely, the devguide was able to take over from there. I installed a couple of extensions, and also added the GitHub CLI tool since i’m used to working with it.

Documenting how to view the docs would probably be helpful. I ran ../python -m http.server -d build/html from the Doc/ directory, which opened a prompt to forward the port and open it in the browser and that seemed to work perfectly. I also tried building the docs with make html and all seemed to work fine.

1 Like

As someone with absolutely no experience of codespaces (although I have a local CPython dev environment, so it’s not something I actually expect to have a need for myself) I don’t even know what it means to say “create a codespace”.

I’m sure there’s overview documents I could go and look at, but if this is targeting someone who simply wants to get up to speed working on the CPython codebase as fast as possible, I’d want to start with:

  1. Do I need to install anything on my PC for this (or is it a web-based environment - I’m willing to assume a browser as a starting point).
  2. Presumably as it’s a github thing, my starting point is the github site, logged in as me. Is that right?

I went to my github account and there’s a “Codespaces” link there. Trying the “blank” template I get to what looks like a web-based version of VS Code. Which I’m comfortable with, so I guess that’s me up and running.

I guess, therefore, that the docs should be essentially telling me something along the lines of where I go to get to a link I can click on that does what clicking on “blank template”, but leaving me in a CPython build environment. Plus, whatever preparatory work I might need to do to set things up so that link works for me. Is that close?

Beyond that basic level of fumbling around, I didn’t see any links in this thread that might have led me to something more detailed, and I don’t know where to go next (or what to expect beyond this).

@brettcannon hopefully that’s the sort of “try it out and provide feedback” comments you had in mind? If there was information I should have found that gave me a better starting point, I didn’t see it (I did check the devguide quickly, but there’s nothing in there that I can see).

3 Likes

The main repo.

Think about where you would press the pencil icon to edit something in the repo; would you do it in the main repo or the fork? Besides, GitHub isn’t about to let you push to anything you shouldn’t, so don’t worry about messing anything up.

You honestly don’t even need that; the Live Preview extension should get installed, so you can just open the index.html file that way.

It’s all in the browser (if you want it to be).

Yes.

You get fancier stuff if you do it from the CPython repo directly. If you click the green “Code” button then you will see a tab called “Codespaces” which gets you everything you need already installed.

If you did it from the CPython repo, you don’t need anything else as at that point the devguide takes over to tell you how to e.g., build CPython.

Sure. :wink: I was specifically after feedback from coaches at sprints who could show folks how to launch a Codespace to identify gaps in the Codespace itself, but this also helps set baseline knowledge and what to potentially say in the docs on how to get a Codespace launched.

That’s what everyone is asking for someone (i.e., probably me) to write. :sweat_smile:

2 Likes

Ironically, this is the last place I’d have ever looked for the feature. But new folks probably see it and know about it.

What I want is when I am reviewing a PR made on a branch on someone else’s fork… For the “secret” , comma key to open a new tab codespace setup for cpython dev checked out into with my vscode settings on their branch so I can build + test and push changes to it, with zero further user intervention to get that going required on my part.

Instead what happens when I do that and click “create new codespace” from the (unnecessary intermediate implementation detail UX) , screen, I get a 500 error from GitHub. :poop:

So codespaces isn’t solving my core dev need, even though I hope it’s useful for others.

2 Likes