Running arm32 agents on arm64 machines

Drone Support
1

I wanted to quickly document how I am using an arm64 server as an arm32 agent with Drone. Please note that not all arm64 servers are capable of running arm32 binaries. If you do not have access to such a server, I recommend using a Scaleway C1 ARMv7 server.

Installation and Configuration

The main challenge is forcing Docker to run in arm32 mode and pull arm32 images. By default it will detect the host OS is arm64 and will default to arm64 images. We can override this behavior by starting the daemon with setarch.

--- docker.service	2018-11-17 07:54:15.843925035 +0000
+++ /lib/systemd/system/docker.service	2018-11-17 07:53:42.595769259 +0000
@@ -10,7 +10,7 @@
 # the default is not to use systemd for cgroups because the delegate issues still
 # exists and systemd currently does not support the cgroup feature set required
 # for containers run by docker
-ExecStart=/usr/bin/dockerd -H unix://
+ExecStart=/usr/bin/setarch linux32 -B  /usr/bin/dockerd -H unix://
 ExecReload=/bin/kill -s HUP $MAINPID
 TimeoutSec=0
 RestartSec=2

The second step it to replace the arm64 docker binary with the arm32 Docker binary. I feel like this step shouldn’t be required, but I couldn’t get it working otherwise:

$ systemctl stop docker

$ curl https://download.docker.com/linux/static/stable/armhf/docker-18.06.1-ce.tgz --output docker-18.06.1-ce.tgz
$ tar xzvf docker-18.06.1-ce.tgz
$ cp docker/* /usr/bin/

The final step is to restart Docker:

$ systemctl daemon-reload
$ systemctl start docker

Configuration

Verify that the docker daemon and client are using the arm32 binaries:

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:54 2018
 OS/Arch:           linux/arm
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:38:12 2018
  OS/Arch:          linux/arm
  Experimental:     false

Pull the latest alpine image and the latest arm32v6 alpine image and confirm they are the same exact image, proving that Docker is pulling the correct images:

$ docker pull alpine:latest
$ docker pull arm32v6/alpine:latest
$ docker images
REPOSITORY          TAG                 IMAGE ID
arm32v6/alpine      latest              8e425eb51aaf
alpine              latest              8e425eb51aaf

Start an alpine container and run uname. In the below example we see armv8l which indicates the arm64 server is running in arm32 mode:

$ docker run alpine uname -a
armv8l Linux

Because I was skeptical I also decided to pull the golang image and then run a small Go program to see which operating system it reported.

$ cat <<EOF > main.go
package main

import (
  "fmt"
  "runtime"
)

func main() {
  fmt.Println(runtime.GOOS)
  fmt.Println(runtime.GOARCH)
}
EOF

$ docker run -v $(pwd):/go golang go run main.go
linux
arm

In Conclusion

It works …

2

Thank you so much for taking the time to put this together. I was able to follow this fairly closely to get armv8l image building running on AWS Graviton2 instance (aarch64). The only thing I had to add (in case anyone else is so inclined) was applying the /usr/bin/setarch linux32 -B treatment to the containerd daemon. One note of caution - the default docker that’s installed via yum on Amazon Linux 2 is version 19 or some such … I was not able to get the binaries for that version to work with this - kept getting Bus Error … but grabbing 18.x binaries mentioned here worked fine on top of the otherwise 19.x yum install. YMMV.

1 Like
3

There is a bit easier method, that will also work with the latest Docker 19.x releases. Here is the cloud-init I use for this setup. You could find more details in a blog post.

#cloud-config

apt_reboot_if_required: false
package_update: true
package_upgrade: true

bootcmd:
  - [ dpkg, --add-architecture, armhf  ]

apt:
  sources:
    docker.list:
      source: deb [arch=armhf] https://download.docker.com/linux/ubuntu $RELEASE stable
      keyid: 0EBFCD88

packages:
  - 'docker-ce:armhf'

write_files:
  - path: /etc/systemd/system/docker.service.d/override.conf
    content: |
      [Service]
      ExecStart=
      ExecStart=/usr/bin/setarch linux32 -B /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

  - path: /etc/systemd/system/containerd.service.d/override.conf
    content: |
      [Service]
      ExecStart=
      ExecStart=/usr/bin/setarch linux32 -B /usr/bin/containerd

runcmd:
  - [ systemctl, daemon-reload ]
  - [ systemctl, restart, docker ]
1 Like