Providing binary packages for Apple M1

Packaging
1

Hello,

as the psycopg2 maintainer I would like to provide binary packages for Apple M1. I understand that this machine can switch between two platform (intel/arm) and that the current toolchain only targets intel.

The current build script is here, here is a sample build run.

Are the packaging tag and the build tools already able to create arm packages? If so does my build script need tweaking?

Thank you very much and sorry if my questions are naive: I am not familiar with these machines. Any help is appreciated! Cheers!

– Daniele

1 Like
2

That’s going to depend on your build tool, but in general the answer is yes if you have an M1 Mac.

I can’t answer the “how”, though, unfortunately.

1 Like
3

The cibuildwheel project supports cross-compiling to universal2 and arm64 architectures from an x86_64 architecture: https://cibuildwheel.readthedocs.io/en/stable/faq/#apple-silicon

You might want to try using that instead to build the wheels?

1 Like
4

@dustin I was considering using cibuildwheel for psycopg3 but I didn’t want to change excessively the build toolchain for psycopg2. However I can give it a try, or to peek at its source to see what they do.

1 Like
5

In general, you can build binaries for M1 Macs on Intel Macs when using current Apple-supplied build tools (Xcode or Command Line Tools) on the current Mac operating systems releases (macOS 11 Big Sur or later). (The reverse should also be true.) As @dustin notes, it is possible to build so-called universal2 fat binaries that contain executables for both Intel and arm64 Macs in the same files. That is generally easier and should be buildable on either hardware type. By building universal2 you tend to avoid bootstrap issues, like running a Python being built. If it is necessary to have single arch binaries, the universal2 files can be separated into single arch binaries using the lipo utility but that should rarely be needed. macOS automatically chooses the appropriate architecture to execute under when a process is launched although that can be overridden. Of course for this to work, all necessary third-party libraries (C or C++) must also be built as universal2 fat binaries. Because the details of building for multiple archs are handled under the covers by the Apple-supplied tool chain, many packages build universal simply by supplying appropriate configure arguments, like -arch arm64 -arch x86_64 added to CFLAGS. Or they may have a specific configuration option like Python does:

  --with-universal-archs=ARCH
                          specify the kind of macOS universal binary that
                          should be created. This option is only valid when
                          --enable-universalsdk is set; options are:
                          ("universal2", "intel-64", "intel-32", "intel",
                          "32-bit", "64-bit", "3-way", or "all") see
                          Mac/README.rst

For third-party packages built with Distutils or Setuptools, building with a current Python that was itself built as a universal2 binary should automatically build any extension modules as universal2 binaries. The universal2 installer variants for macOS downloadable from python.org work this way. Also, some distributors of third-party open source projects for macOS, like the MacPorts project, support building universal variants.

1 Like
6

Hello @nad. This seems helpful, thank you very much.

Github Actions at the moment only offer Mac OS 10.15, with Mac OS 11 in limited preview and at the moment not available. I guess these packages will have to wait for when the Mac OS 11 workers are available.

1 Like
7

There are ways to do it with macOS 10.15, but it requires a bit more setup (at least based on my experience of trying to do this for a Rust project).

2 Likes
8

Yes, for macOS 10.15 Catalina, you would need to ensure you have installed (and use for your builds) the most recent Xcode supported for it, currently Xcode 12.5. But it is likely easier for things to go wrong (and building on 10.15 for use on 11 is not a configuration that we test for cPython itself), so probably best to build on macOS 11 Big Sur.

[EDIT: Sorry, I misremembered. I see that the current Xcode 12.5 is not supported on macOS 10.15 Catalina; the most recent and likely last Xcode release supported on 10.15 was Xcode 12.4. As I recall, there were a few issues related to Big Sur and M1 support that were resolved in Xcode 12.5, so that would be one more reason to build on 11 Big Sur and not use 10.15 Catalina.]

1 Like
9

This is what it took to add a cibuildwheel build to a non-cibuildwheel project, just for M1’s: ci: add Python 3.8 universal build by henryiii · Pull Request #106 · pganssle/zoneinfo · GitHub

Though I should note that there are several tricks to distributing highly compatible wheels that cibuildwheel gets right, so that package would benefit from moving at least the macOS builds to cibuildwheel. :wink:

You are going to be stuck with Xcode 12.4 on the currently available CI runners, but so far it’s worked fine for quite a few cibuildwheel packages, include mypy.

2 Likes
10

Hello folks,
I tried to cross-compile psycopg2-binary from an Intel Big Sur machine, however, it did not work.

This Gist works on a an Arm64 Big Sur machine but not on Intel:

It does not work with either cibuildwheel or pure delocate usage.

The PR I’ve been working from is this.

The execution boils down to:

export PACKAGE_NAME=psycopg2-binary
export ARCHFLAGS="-arch arm64"
export _PYTHON_HOST_PLATFORM="macosx-11.0-arm64"
export SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk"
# More set up steps
pip wheel --use-feature=in-tree-build --wheel-dir /tmp/psycopg2.91366/built_wheel --no-deps .
delocate-listdeps /tmp/psycopg2.91366/built_wheel/psycopg2_binary-2.9.1-cp38-cp38-macosx_11_0_arm64.whl
delocate-wheel --require-archs arm64 -w /tmp/psycopg2.91366/repaired_wheel /tmp/psycopg2.91366/built_wheel/psycopg2_binary-2.9.1-cp38-cp38-macosx_11_0_arm64.whl

delocate-listdeps does not list these on the Intel host while it does for the Arm64 host (Homebrew uses /usr/local/Cellar on Intel hosts):

/opt/homebrew/Cellar/openssl@1.1/1.1.1l/lib/libcrypto.1.1.dylib
/opt/homebrew/Cellar/openssl@1.1/1.1.1l/lib/libssl.1.1.dylib
/opt/homebrew/Cellar/postgresql/13.4/lib/libpq.5.13.dylib

I assume things don’t work because we need the arm64 version of the libraries:

❯ lipo -info /usr/local/Cellar/openssl@1.1/1.1.1l/lib/libssl.1.1.dylib
openssl@1.1/1.1.1l/lib/libssl.1.1.dylib
Non-fat file: /usr/local/Cellar/openssl@1.1/1.1.1l/lib/libssl.1.1.dylib is architecture: x86_64
❯ lipo -info /usr/local/Cellar/postgresql/13.4/lib/libpq.5.13.dylib
/usr/local/Cellar/postgresql/13.4/lib/libpq.5.13.dylib
Non-fat file: /usr/local/Cellar/postgresql/13.4/lib/libpq.5.13.dylib is architecture: x86_64

Does this mean that we cannot cross-compile to arm64 to an Intel host when a Python package needs to include an arm64 library?
Can the arm64 libraries that the Arm64 host has be packaged and used in the Intel host during the packaging process?

1 Like