Bug 1170063 - podman 1.9.0 is unable to download opensuse/leap image for aarch64
podman 1.9.0 is unable to download opensuse/leap image for aarch64
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Containers
Current
aarch64 Other
: P5 - None : Normal (vote)
: ---
Assigned To: Containers Team
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2020-04-21 12:06 UTC by Guillaume GARDET
Modified: 2021-10-19 08:49 UTC (History)
5 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Guillaume GARDET 2020-04-21 12:06:23 UTC
Since podman update from 1.8.2 to 1.9.0 [0], podman tests in openQA for Kubic/MicroOS [1] are unable to download the opensuse/leap image for aarch64 with the following error:
  no image found in manifest list for architecture arm64, variant v8, OS Linux

But "hello-world" and "Alpine:3.6" images are fine.

[0]: https://build.opensuse.org/request/show/794479
[1]: https://openqa.opensuse.org/tests/1241153#step/podman/11
Comment 1 Sascha Grunert 2020-04-21 14:32:23 UTC
Hey, the issue seems not related to podman. It looks like that we have no arm v8 image for leap available inside the docker registry:

I can see only those:
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 634,
         "digest": "sha256:61d9fed5fd5976c1b42e0e90ffa83392a067b822092656ff6b11cce1a8eb6afa",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 634,
         "digest": "sha256:f7e0ee9ff8e7755dccccb1745122952ef71cf7d25ffc372383b4b01f1f31247c",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 634,
         "digest": "sha256:2c7eda1052ef65d2a49b5d3446d766d8549c16b9700eab281597716e5077c18a",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 634,
         "digest": "sha256:0d96a71498af54c40726e9378bc18668cb7752ebc80801ac74b8654feba9e6b8",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      }
   ]
}
Comment 2 Guillaume GARDET 2020-04-22 06:44:35 UTC
(In reply to Sascha Grunert from comment #1)
> Hey, the issue seems not related to podman. It looks like that we have no
> arm v8 image for leap available inside the docker registry:

I think this is related since previous version worked fine.
I think we should add:
  "variant": "v8"

to arm64 architecture.

Who could add it?
Comment 3 Sascha Grunert 2020-04-22 07:34:39 UTC
(In reply to Guillaume GARDET from comment #2)
> (In reply to Sascha Grunert from comment #1)
> > Hey, the issue seems not related to podman. It looks like that we have no
> > arm v8 image for leap available inside the docker registry:
> 
> I think this is related since previous version worked fine.
> I think we should add:
>   "variant": "v8"
> 
> to arm64 architecture.
> 
> Who could add it?

I'm not completely sure how the images are pushed into the docker hub, but maybe someone from the openSUSE colleagues could help you out there.
Comment 4 Fabian Vogt 2020-04-22 08:04:38 UTC
That sounds like a podman bug. Does arm64 require that variant is also set?
Neither older podman nor docker do.
Comment 5 Guillaume GARDET 2020-04-22 08:26:52 UTC
(In reply to Fabian Vogt from comment #4)
> That sounds like a podman bug. Does arm64 require that variant is also set?
> Neither older podman nor docker do.

"hello-world" and "Alpine:3.6" images have this additional flag set, and it looks like lots of images have this flag set. So, it is probably a good idea to add it anyway.

But, podman should have a safe fallback when this flag is not set.
Comment 6 Sascha Grunert 2020-04-22 09:03:49 UTC
Probably related: https://github.com/containers/image/issues/898

I guess the logic can be a bit relaxed, at least for some arm architectures.
Comment 7 Guillaume GARDET 2020-04-22 09:18:43 UTC
(In reply to Sascha Grunert from comment #6)
> Probably related: https://github.com/containers/image/issues/898

It looks like this is the same problem.

> I guess the logic can be a bit relaxed, at least for some arm architectures.

Currently, only v8 variant is available for aarch64/arm64.
Please note that:
* arm,v6 can be run on arm,v7
* arm,v6/arm,v7 (32-bit arm) can be run on _some_ aarch64 (arm64,v8, 64-bit arm) system since 64-bit arm SoC are not all 32-bit capable.
Comment 8 Fabian Vogt 2020-04-22 10:27:29 UTC
As a workaround, I mapped aarch64 to ("arm64", "v8") now. Images on docker hub got updated already.
Comment 9 Sascha Grunert 2020-04-23 17:34:31 UTC
Thanks Fabian, a fix is incoming but it might take a release to land in Podman. CRI-O and friends are affected as well. 

https://github.com/containers/image/pull/904
Comment 10 Guillaume GARDET 2021-10-19 08:49:36 UTC
Fixed since a while.