I was asked in an email how to work on various GitHub release branches at the same time. For example, working with the master branch and a 3.6 branch. I have read snippets in other messages about how some individuals manage this, but I thought it might be helpful ask in a central location to establish best practices. Hopefully this will generate some dialogue and we’ll all learn something.
Please note that I’m not asking about IDEs, editors, git workflow in general, etc. I’m just asking how you run multiple versions of Python (from source) simultaneously. If you want to talk about how you also manage multiple downloaded versions (or pre-installed versions) in addition to builds, then I think that can be helpful as well. OS might be helpful in this discussion.
I like to use git worktree for this (for anyone familiar with hg share, it’s basically the same thing). It essentially allows several checkouts of the same repository to share the object store, but no two worktrees are allowed to have the same branch checked out at the same time (though the same commit can be checked out under different branch names). In my view, this is the gold standard for running multiple versions from source at once, and is completely platform independent; I do the same on macOS, Linux, and Windows.
For example, I tend to check out my code from cd ~/code/github.com/python/cpython; the initial clone is in ./master (git clone https://github.com/python/cpython.git master), then from cd ./master I can run for branch in 3.7 3.6 2.7; do git worktree add ../$branch; done (though I would probably just do each individually as I need it).
On the other hand, there’s not nearly as much need to keep individual checkouts for each branch anymore, unlike in our old hg workflow. Now it’s simple enough to run make distclean (or git clean -fdx if there’s nothing untracked that you want to keep), check out the branch you need, and rebuild afresh on the occasions when a non-master build is needed. This doesn’t help with actually running two versions simultaneously if that’s what you need, though.
Many people don’t realize that python’s build system on Unix allows seperating the build directory from the source directory. I always do builds in a separate directory, both for general cleanliness (keeps source files and build artifacts separate; makes it harder to accidentally mix artifacts from different builds), and because it lets you have multiple independent builds simultaneously.
For example:
$ mkdir my-build-dir
$ cd my-build-dir
$ ../configure <options here>
$ make
The trick is that whatever directory you’re in when you run configure becomes the build dir, which is where you run make and where all the build artifacts end up. This doesn’t have to be the source directory; it can be any directory.
(This actually works for most projects that use the classic configure; make pattern, not just python.)
The build dir does refer back to the source tree when building; this doesn’t give you independent copies of the source files The ideal case is when you have multiple builds with different configure settings (e.g. different compiler flags). If you want to use this to work on multiple branches simultaneously, it can be a bit awkward – you have to make sure you only run make in your py36 build dir when you have the py36 branch checked out in the source tree, but there’s nothing that enforces this. @zware’s solution is probably less error-prone for this case. But it can work, and you can also combine the two approaches.
I haven’t really gone beyond the multiple build dirs approach myself. I regularly wipe out build dirs if I’m unaware of their state as compile times are so fast today. The git worktree sounds intriguing, will try! Thanks for the idea.
I like to one a different directory per Python version to benefit of incremental compilation (make doesn’t have to rebuild everything) when I switch to a different brank. As Zachary, I use git worktree:
I have a prog/python/update.sh script to update my checkout and move back to the “right” branch (for example, if I was in a local branch in master/, it goes back to master branch):
$ cat update.sh
set -x -e
cd ~/prog/python/master
git fetch upstream --tags --prune
git fetch origin --prune
for branch in master 3.7 3.6 2.7; do
cd ~/prog/python/$branch
git checkout $branch
git merge --ff
done
I also have a prog/python/all_make.sh script to recompile all Pythons:
set -e -x
(cd master && make) || exit $?
for version in 3.7 3.6 2.7; do
(cd $version && cp -f Modules/Setup.dist Modules/Setup && make) || exit $?
done
Example to manually cherry-pick a change into 3.7:
$ cd 3.7
$ git checkout -b myfix37
Switched to a new branch 'myfix37'
$ git cherry-pick -x SHA1
$ ...
$ make && ./python -m test ...
$ gh_pr.sh
gh_pr.sh is another hackish script to create a PR for Python:
set -e -x
if [[ "$1" = "-f" ]]; then
force=-f
else
force=
fi
local_branch=$(git name-rev --name-only HEAD)
project="$(basename $PWD)"
ref_branch=master
case "$project" in
[23].[0-9])
ref_branch=$project
project=cpython
;;
master)
ref_branch=$project
project=cpython
;;
esac
echo "branches: $local_branch -> $ref_branch"
git push origin HEAD $force
URL="https://github.com/python/$project/compare/$ref_branch...vstinner:$local_branch?expand=1"
python3 -m webbrowser $URL
gh_pr.sh requires that the Git “origin” remote is my fork:
I used that in the past, but when I switch between Python branches, it changes the modification time and so requires to re-compile some files just because of that. I don’t have this issue with git worktree. Well, use whatever works for you, I don’t think that one workflow is better than another
