Bugzilla – Bug 1143725
qemu-aarch64-binfmt, from qemu-linux-user, not working (no output)
Last modified: 2022-10-27 18:38:36 UTC
Context: I want to run an ARM64 (aarch64) container on an x86 host. This is possible using qemu Linux user emulation and Linux binfmt. It is supposed to work like this: //Fetching an ARM container on x86 (long story) $ skopeo --override-arch arm64 copy docker://opensuse/leap docker-daemon:opensuse-arm64/leap:latest //Install qemu Linux user, and setup binfmt for aarch64 # zypper install qemu-linux-user $ cp /usr/bin/qemu-aarch64-binfmt . # echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64 # echo ':qemu-aarch64:M:0:\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-binfmt:POCF' > /proc/sys/fs/binfmt_misc/register //Build our container $ cat << EOF > Dockerfile FROM opensuse-arm64/leap COPY qemu-aarch64-binfmt /usr/bin/qemu-aarch64-binfmt EOF docker build -t qemu-leap . //Now I should be able to run the ARM container, like this: # docker run -it qemu-leap-aarch64-binfmt /bin/sh But, basically, nothing happens. No output at all, and I'm definitely not inside an ARM container. Actually, it appears the container exited: $ docker ps --all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ea2252caf90a qemu-leap "/bin/sh" 4 seconds ago Exited (255) 3 seconds ago relaxed_goodall Now, if I, instead of qemu-aarch64-binfmt, use a qemu-aarch64-static binary copied from a Debian unstable box I have in my home network, i.e., this one: # ./qemu-aarch64-static -version qemu-aarch64 version 3.1.0 (Debian 1:3.1+dfsg-8) Things work! In fact: $ docker run -it --rm qemu-leap uname -a Linux bf12752f6a51 5.1.16-1-default #1 SMP Wed Jul 3 12:37:47 UTC 2019 (2af8a22) aarch64 aarch64 aarch64 GNU/Linux Note the *aarch64*. Of course, we need to change the aarch64 interpreter file too, in /proc/sys/fs/binfmt So, basically, it looks like the procedure is correct, or at least can work. But our qemu-linux-user packages contain binaries that are not adequate for being used for this use case. Things/differences to check: - QEMU version (the debian stolen from Debian, that makes things works is from QEMU 3.1.0, while we ship 4.0.0 - our patch queue - our configuration and compile options
Note that is doesn't look like it could be a "container specific" thing. In fact, I tried to setup a "traditional" ARM64 chroot, on the x86 host, again using qemu-linux-user and binfmt (basically, how it is described here: https://en.opensuse.org/HCL:Chroot) and I run into the same issue. BTW, adding brogers@suse.com to Cc, as this is about QEMU.
And I've just checked that building (statically) qemu-aarch64, from the upstream git tree, works as well. $ cat /proc/sys/fs/binfmt_misc/qemu-aarch64 enabled interpreter /usr/bin/qemu-aarch64-upstream flags: OCF offset 0 magic 7f454c460201010000000000000000000200b700 mask ffffffffffffff00fffffffffffffffffeffffff $ /usr/bin/qemu-aarch64-upstream --version qemu-aarch64 version 4.0.93 (v4.1.0-rc3-dirty) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers $ docker run -it --rm qemu-leap /bin/bash :/ # uname -a Linux cf5f2385946e 5.1.16-1-default #1 SMP Wed Jul 3 12:37:47 UTC 2019 (2af8a22) aarch64 aarch64 aarch64 GNU/Linux :/ # /usr/bin/qemu-aarch64-upstream --version qemu-aarch64 version 4.0.93 (v4.1.0-rc3-dirty) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers I think this is a step toward ruling out QEMU version issues. Now our patch-queue and our config options remain to be checked.
does running a simple AArch64 binary with qemu user mode work? (no container..?)
(In reply to Claudio Fontana from comment #3) > does running a simple AArch64 binary with qemu user mode work? (no > container..?) > It does. In fact, using the stock qemu-aarch64-binfmt interpreter, as provided by our qemu-linux-user package: $ cat pippo.c #include <stdio.h> int main() { int a=7; printf("a=%d\n", a); return a++; } $ uname -a Linux Palanthas 5.1.16-1-default #1 SMP Wed Jul 3 12:37:47 UTC 2019 (2af8a22) x86_64 x86_64 x86_64 GNU/Linux $ file pippo-static pippo-static: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=f0839e4cd8142d9b85cb41d3cac9afc4b82403ed, for GNU/Linux 3.7.0, with debug_info, not stripped $ ./pippo-static ; echo "pippo-static returned $?" a=7 pippo-static returned 7
(In reply to Dario Faggioli from comment #1) > Note that is doesn't look like it could be a "container specific" thing. In > fact, I tried to setup a "traditional" ARM64 chroot, on the x86 host, again > using qemu-linux-user and binfmt (basically, how it is described here: > https://en.opensuse.org/HCL:Chroot) and I run into the same issue. > > BTW, adding brogers@suse.com to Cc, as this is about QEMU. > And I was wrong here. A "traditional chroot", works: $ cat /proc/sys/fs/binfmt_misc/qemu-aarch64 enabled interpreter /usr/bin/qemu-aarch64-binfmt flags: P offset 0 magic 7f454c460201010000000000000000000200b700 mask ffffffffffffff00fffffffffffffffffeffffff # chroot src/rootfs/ARM/openSUSE/Tumbleweed/ /bin/bash Palanthas:/ # uname -a Linux Palanthas 5.1.16-1-default #1 SMP Wed Jul 3 12:37:47 UTC 2019 (2af8a22) aarch64 aarch64 aarch64 GNU/Linux So, this, together with Comment 4 , seems to suggest that it is something related to the interaction of qemu-aarch64-binfmt (from qemu-linux-user) and docker (or some other container-related component)
(qemu-linux-user 7.0.0-55.1) I had the same issue with binaries exiting with code 255. I found that using qemu-aarch64 instead works perfectly: $ echo ':qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:POCF' > /proc/sys/fs/binfmt_misc/register $ cat /proc/sys/fs/binfmt_misc/qemu-aarch64 enabled interpreter /usr/bin/qemu-aarch64 flags: POCF offset 0 magic 7f454c460201010000000000000000000200b700 mask ffffffffffffff00fffffffffffffffffeffffff Now Docker containers are able to run aarch64 binaries just fine. I did not test with a chroot to see if it behaves differently.
(In reply to Arnav Singh from comment #6) > (qemu-linux-user 7.0.0-55.1) > > I had the same issue with binaries exiting with code 255. I found that > using qemu-aarch64 instead works perfectly: > > $ echo > ':qemu-aarch64:M:: > \x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00: > \xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\ > xff:/usr/bin/qemu-aarch64:POCF' > /proc/sys/fs/binfmt_misc/register > > $ cat /proc/sys/fs/binfmt_misc/qemu-aarch64 > > enabled > interpreter /usr/bin/qemu-aarch64 > flags: POCF > offset 0 > magic 7f454c460201010000000000000000000200b700 > mask ffffffffffffff00fffffffffffffffffeffffff > > Now Docker containers are able to run aarch64 binaries just fine. > Yes! And thanks for the report. In fact, we've been working on removing some [open]SUSE specific handling of binfmt, which made things simpler, back then where they were introduced, but were instead complicating them a lot, now that we have containers! In fact, this works well for me as well now, just out of the box: > $ podman pull --arch=arm64 docker.io/library/httpd > $ sudo qemu-binfmt-conf.sh --systemd aarch64 --persistent yes > $ sudo systemctl daemon-reload > $ sudo systemctl start systemd-binfmt.service > $ podman run --rm -it docker.io/library/httpd uname -a > WARNING: image platform ({arm64 linux [] v8}) does not match the expected platform ({amd64 linux [] }) > Linux 827190b5981e 5.18.11-1-default #1 SMP PREEMPT_DYNAMIC Fri Jul 15 05:36:11 UTC 2022 (4fcb983) aarch64 GNU/Linux > I did not test with a chroot to see if it behaves differently. > Yeah, no need to do that. I'll get rid of the qemu-binfmt binary, and of all the symlinks, from the package ASAP. They're no longer needed and are only confusing people!
(In reply to Dario Faggioli from comment #7) > Yeah, no need to do that. I'll get rid of the qemu-binfmt binary, and of all > the symlinks, from the package ASAP. They're no longer needed and are only > confusing people! > And done: https://build.opensuse.org/request/show/1008185 Will submit to Factory later. For now, we can close this bug.
This is an autogenerated message for OBS integration: This bug (1143725) was mentioned in https://build.opensuse.org/request/show/1008827 Factory / qemu