Mid-February, I was able to get a PR merged upstream into Crystal for aarch64-apple-darwin support (i.e. Apple Silicon) based on some great work by Romain Franceschini to get the libc bindings and a lot of the foundational stuff ready. This PR also shipped with Crystal 1.0.0, and I was hoping that the cross-compilation story would get a lot easier, so I put off writing a post on targeting arm64. Unfortunately, there's been some speed-bumps I won't get into here, but suffice to say it's not quite possible to
brew install crystal via Homebrew yet or run via an official M1 binary build either.
However, I wanted to revisit the process for cross-compiling your copy of the Crystal compiler on the Apple M1 for those that are daring and not willing to wait. My hope is to keep this somewhat updated, so that as things evolve and change, eventually this'll be a pretty simple set of steps. Already, writing this in early/mid April, things have gotten a little bit easier.
bdw-gc in Homebrew now ships with the necessary multi-threading patch for Crystal. In addition, I landed another PR to rewrite the Crystal segfault handler in Crystal, instead of C, making it easier to cross-compile (this will ship in 1.1.0.)
We'll need two Homebrews, one for each CPU architecture, in order to cross-compile Crystal on the same machine. This is because we need to bootstrap the compiler from somewhere, and what better way than to bootstrap it from the x86_64 compiler via Rosetta 2, which we can also run on the machine.
Install x86_64 Homebrew (aka. Intel Homebrew)
Run the following to install Intel Homebrew, which we will occasionally refer to as
ibrew throughout this post:
$ arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Once this finishes, you should have a working Homebrew install in the default x86_64 prefix of
ibrew calls specified later will be in reference to this Intel Homebrew install.
Add an alias for Intel Homebrew
In your shell or your shell's config (
.zshrc, etc.), add an alias for
alias ibrew="arch -x86_64 /usr/local/bin/brew"
Install arm64 Homebrew (aka. ARM Homebrew)
Run the same line again, this time removing the explicit arch call in front of it. It should run under arm64 by default, unless your Terminal is setup to run under Rosetta (Stop doing that now, if you are using your Terminal in Rosetta mode. It'll make the rest so much harder/impossible, and I cannot help you.)
$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
This should prompt for installing into
/opt/homebrew, the default prefix for Apple Silicon Homebrew.
brew calls specified later will be in reference to ARM Homebrew.
Add ARM homebrew to your PATH
In your shell or your shell's config, add ARM brew to your path:
$ export PATH="/opt/homebrew/bin:/usr/local/bin:$PATH"
This is to ensure ARM software from Homebrew runs before Intel software.
Install Crystal dependencies
LLVM 11+ is required for targeting Apple Silicon, so we need to make sure we install LLVM on both installations of Homebrew, and all the other dependent libraries needed, including build-time dependencies.
$ ibrew install llvm bdw-gc gmp libevent libyaml [email protected] pcre pkg-config $ brew install llvm bdw-gc gmp libevent libyaml [email protected] pcre pkg-config
Grab an official macOS binary build of Crystal for x86_64
We'll need this for later, when compiling the compiler the first time. You could also install with
ibrew as well.
$ mkdir -p ~/crystal-prebuilt $ curl -fsSL https://github.com/crystal-lang/crystal/releases/download/1.0.0/crystal-1.0.0-1-darwin-x86_64.tar.gz > ~/crystal-prebuilt/crystal-1.0.0-1-darwin-x86_64.tar.gz $ cd ~/crystal-prebuilt $ tar -xf crystal-1.0.0-1-darwin-x86_64.tar.gz
Compiling Crystal for ARM64
Setup compile environment variables
Since we'll be targeting two different CPU architectures, we need to be extra careful about what build environment we specify. Intel Homebrew lives in a higher priority directory for most build tools, so it is often checked first and will be used instead of ARM Homebrew's directories unless we specify ARM Homebrew's build tools instead. We'll need to do this a lot, so lets specify some environment variables up-front:
$ export INTEL_BREW_PREFIX="$(ibrew --prefix)" $ export INTEL_LLVM_ROOT="$INTEL_BREW_PREFIX/opt/llvm" $ export INTEL_LLVM_CONFIG="$INTEL_LLVM_ROOT/bin/llvm-config" $ export ARM_BREW_PREFIX="$(brew --prefix)" $ export ARM_LLVM_ROOT="$ARM_BREW_PREFIX/opt/llvm" $ export ARM_LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config"
Compile for x86_64
The first step is to compile for x8664. Yep, x8664. We need to do this, because the official binary builds of the Crystal compiler ship compiled against [email protected] and without the ability to cross-compile for AArch64/ARM64 on Darwin, so we need to re-compile the compiler against LLVM 11 so that we can have the ability to cross-compile for arm64. In the future, this may not be necessary but for now it is.
$ LLVM_CONFIG="$INTEL_LLVM_CONFIG" \ LDFLAGS="-L$INTEL_LLVM_ROOT/lib" \ CPPFLAGS="-I$INTEL_LLVM_ROOT/include" \ CC="$INTEL_LLVM_ROOT/bin/clang" \ AR="$INTEL_LLVM_ROOT/bin/llvm-ar" \ CRYSTAL="$HOME/crystal-prebuilt/crystal-1.0.0-1/bin/crystal" \ arch -x86_64 make
Cross-compile to arm64
The next step is to take our x86_64 compiler and use it to cross-compile for the arm64 target triple (aarch64-apple-darwin).
$ LLVM_CONFIG="$INTEL_LLVM_CONFIG" \ LDFLAGS="-L$INTEL_LLVM_ROOT/lib" \ CRYSTAL_CONFIG_TARGET=aarch64-apple-darwin \ CRYSTAL_CONFIG_LIBRARY_PATH="$ARM_BREW_PREFIX/lib" \ arch -x86_64 ./bin/crystal build src/compiler/crystal.cr --cross-compile --target aarch64-apple-darwin -Dwithout_playground
Next, we need to compile the LLVM extension libraries for arm64:
$ LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \ LDFLAGS="-L$ARM_LLVM_ROOT/lib" \ $ARM_LLVM_ROOT/bin/clang -I$ARM_LLVM_ROOT/include -c -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc
Finally, we need to link our cross-compiled compiler object (
crystal.o) with our arm64
llvm_ext.o and the rest of our arm64 libraries, in order to get a final arm64 executable for the Crystal compiler.
$ LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \ LDFLAGS="-L$ARM_LLVM_ROOT/lib" \ CPPFLAGS="-I$ARM_LLVM_ROOT/include" \ $ARM_LLVM_ROOT/bin/clang crystal.o -o .build/crystal -rdynamic -L$ARM_BREW_PREFIX/lib src/llvm/ext/llvm_ext.o `"$ARM_LLVM_ROOT/bin/llvm-config" --libs --system-libs --ldflags 2> /dev/null` -lstdc++ -lpcre -lgc -lpthread -L$(brew --prefix libevent)/lib -levent -liconv -ldl
In this final step, it's super easy for x8664 libraries to get picked up instead of or in addition to the arm64 libraries. If you get a working compiler at the end but have a bunch of warnings about the linker trying to link against an x8664 library, you can safely ignore those. If you don't get a working compiler at the end, it's possible you're missing an arm64 version of a required library, and you'll need to specify a more direct path via the linker arguments.
Verifying the compiled compiler
Now, at the end, we should be able to verify that we have a working arm64 crystal compiler:
$ file .build/crystal .build/crystal: Mach-O 64-bit executable arm64
(Optional): Running compiler and stdlib test suite
This step is probably optional if you just want to compile some of your Crystal programs for arm64, but if you want to be extra sure the compiler works okay or are working on Crystal compiler or standard library development, you may want to run the test suite.
We're doing this directly to avoid triggering unnecessary rebuilds via the Makefile, but it's likely possible to just call
make spec with the same environment variables specified and have the right thing happen.
Compiling and running the compiler specs
$ CRYSTAL_PATH=$(pwd)/src \ CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \ CC="$ARM_LLVM_ROOT/bin/clang" \ LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \ LDFLAGS="-L$ARM_LLVM_ROOT/lib" \ CPPFLAGS="-I$ARM_LLVM_ROOT/include" \ PKG_CONFIG_PATH="$(brew --prefix [email protected])/lib/pkgconfig" \ ./bin/crystal build -o spec/compiler_spec spec/compiler_spec.cr -Di_know_what_im_doing $ CRYSTAL_PATH=$(pwd)/src \ CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \ CC="$ARM_LLVM_ROOT/bin/clang" \ LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \ LDFLAGS="-L$ARM_LLVM_ROOT/lib" \ CPPFLAGS="-I$ARM_LLVM_ROOT/include" \ PKG_CONFIG_PATH="$(brew --prefix [email protected])/lib/pkgconfig" \ spec/compiler_spec
Compiling and running the stdlib specs
$ CRYSTAL_PATH=$(pwd)/src \ CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \ CC="$ARM_LLVM_ROOT/bin/clang" \ LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \ LDFLAGS="-L$ARM_LLVM_ROOT/lib" \ CPPFLAGS="-I$ARM_LLVM_ROOT/include" \ PKG_CONFIG_PATH="$(brew --prefix [email protected])/lib/pkgconfig" \ ./bin/crystal build -o spec/std_spec spec/std_spec.cr --target aarch64-apple-darwin -Di_know_what_im_doing $ CRYSTAL_PATH=$(pwd)/src \ CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \ CC="$ARM_LLVM_ROOT/bin/clang" \ LLVM_CONFIG="$ARM_LLVM_ROOT/bin/llvm-config" \ LDFLAGS="-L$ARM_LLVM_ROOT/lib" \ CFLAGS="--target aarch64-apple-darwin" \ CPPFLAGS="-I$ARM_LLVM_ROOT/include" \ PKG_CONFIG_PATH="$(brew --prefix [email protected])/lib/pkgconfig" spec/std_spec
Compiling Crystal programs on arm64
Now that we've got a working Crystal compiler for arm64, we can go about compiling our arm64 binaries.
Or we could. But we probably won't get very far yet. We still need to compile
shards for arm64 so that you can install dependencies. However, compiling
shards is a fairly good demo for how to compile any Crystal program for arm64, as it is basically just another Crystal program.
$ git clone [email protected]:crystal-lang/shards.git $ cd shards # If $HOME/src/crystal isn't where Crystal is cloned for you, update the path in CRYSTAL_PATH and CRYSTAL $ CRYSTAL_PATH=lib:$HOME/src/crystal/src \ CRYSTAL_LIBRARY_PATH=$ARM_BREW_PREFIX/lib \ CRYSTAL=$HOME/src/crystal/.build/crystal \ make install
This should install it into
/usr/local/bin, which we can verify by running the below command:
$ which shards /usr/local/bin/shards $ file $(which shards) /usr/local/bin/shards: Mach-O 64-bit executable arm64
Compiling other programs
In general, you can probably get most of the way there with your programs by providing
CRYSTAL as specified above if you have a Makefile-type setup where
CRYSTAL can be provided. Otherwise, you can invoke the arm64-compiled
crystal binary directly while providing
CRYSTAL_LIBRARY_PATH. These two values are needed, because we're not using an omnibus build where these values would already be set by a wrapper script. Instead, we have to provide them and be explicit so that the compiler knows where to look up Crystal code and linkable libraries, respectively.
Depending on the libraries used in your application, you may need to supply other typical environment variables that you might be expected to supply anyone, the most common being for OpenSSL 1.1, e.g.
PKG_CONFIG_PATH="$(brew --prefix [email protected])/lib/pkgconfig"
HACK: Mimicking the Omnibus Crystal wrapper
Instead of passing the
CRYSTAL_LIBRARY_PATH all over, we can also add a thin shell wrapper to provide those as part of the
crystal command. This is similar to what the official omnibus builds do, where there's a top-level
crystal shell script which sets the environment properly and passes off to the actual
Add this script to your
PATH somewhere (don't forget to make it executable):
#!/bin/sh # Again, if you installed somewhere other than $HOME/src/crystal, change it: export CRYSTAL_PATH=lib:$HOME/src/crystal/src # This should be ARM homebrew's lib export CRYSTAL_LIBRARY_PATH=$(brew --prefix)/lib # Mad temporary. You might want to copy this build else where and point it to that less-temporary location export CRYSTAL=$HOME/src/crystal/.build/crystal exec "$CRYSTAL" "[email protected]"
This is a bit of a temporary improvement to ergonomics, but should reduce the amount of pain when working in Crystal regularly on an M1.
In future, I will be looking at trying to get aarch64/arm64 macOS support into the omnibus setup, but long-term this will require having Apple Silicon in CI, which would require some involvement from the core team.
In the meantime, I'm may try and get an initial version of the omnibus package working, that maybe I can provide builds for in a lagging fashion until it's possible to either cross-compile for arm64 in Homebrew via bootstrapping from the official x86_64 build (will require someone figuring out the LLVM 11 issues in release mode) or the core team is able to provide official builds.