Bug 1183717 - u-boot fails to read btrfs
u-boot fails to read btrfs
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Bootloader
Current
All openSUSE Tumbleweed
: P5 - None : Normal (vote)
: ---
Assigned To: Wenruo Qu
E-mail List
:
Depends on:
Blocks: 1184947
  Show dependency treegraph
 
Reported: 2021-03-18 14:24 UTC by Matwey Kornilov
Modified: 2021-10-28 20:35 UTC (History)
7 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
btrfs ins dump-tree output (2.56 MB, application/x-xz)
2021-04-02 06:20 UTC, Matwey Kornilov
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Matwey Kornilov 2021-03-18 14:24:13 UTC
Hello,

I am running openSUSE-Tumbleweed-ARM-JeOS-rock64.aarch64-2021.03.18-Build5.2.raw.xz image on Rock64 SBC with U-Boot 2021.01.

I see the following issue:

u-boot cannot load the board dtb file. The file itself is present in the proper place at the btrfs partition of the sdcard. I checked that the file can be successfully read from PC.

=> ls mmc 1:3 /@/.snapshots/1/snapshot/boot/dtb/
<SYMLINK>         48  Fri Mar 12 08:12:08 2021  px30-engicam-px30-core-ctouch2-of10.dtb -> rockchip/px30-engicam-px30-core-ctouch2-of10.dtb
<SYMLINK>         43  Fri Mar 12 08:12:08 2021  px30-engicam-px30-core-ctouch2.dtb -> rockchip/px30-engicam-px30-core-ctouch2.dtb
<SYMLINK>         44  Fri Mar 12 08:12:08 2021  px30-engicam-px30-core-edimm2.2.dtb -> rockchip/px30-engicam-px30-core-edimm2.2.dtb
<SYMLINK>         21  Fri Mar 12 08:12:08 2021  px30-evb.dtb -> rockchip/px30-evb.dtb
<SYMLINK>         23  Fri Mar 12 08:12:08 2021  rk3308-evb.dtb -> rockchip/rk3308-evb.dtb
<SYMLINK>         26  Fri Mar 12 08:12:08 2021  rk3308-roc-cc.dtb -> rockchip/rk3308-roc-cc.dtb
<SYMLINK>         27  Fri Mar 12 08:12:08 2021  rk3318-a95x-z2.dtb -> rockchip/rk3318-a95x-z2.dtb
<SYMLINK>         30  Fri Mar 12 08:12:08 2021  rk3326-odroid-go2.dtb -> rockchip/rk3326-odroid-go2.dtb
<SYMLINK>         22  Fri Mar 12 08:12:08 2021  rk3328-a1.dtb -> rockchip/rk3328-a1.dtb
<SYMLINK>         23  Fri Mar 12 08:12:09 2021  rk3328-evb.dtb -> rockchip/rk3328-evb.dtb
<SYMLINK>         30  Fri Mar 12 08:12:09 2021  rk3328-nanopi-r2s.dtb -> rockchip/rk3328-nanopi-r2s.dtb
<SYMLINK>         26  Fri Mar 12 08:12:09 2021  rk3328-roc-cc.dtb -> rockchip/rk3328-roc-cc.dtb
<SYMLINK>         26  Fri Mar 12 08:12:09 2021  rk3328-rock64.dtb -> rockchip/rk3328-rock64.dtb
<SYMLINK>         31  Fri Mar 12 08:12:09 2021  rk3368-evb-act8846.dtb -> rockchip/rk3368-evb-act8846.dtb
<SYMLINK>         27  Fri Mar 12 08:12:09 2021  rk3368-geekbox.dtb -> rockchip/rk3368-geekbox.dtb
<SYMLINK>         31  Fri Mar 12 08:12:09 2021  rk3368-lion-haikou.dtb -> rockchip/rk3368-lion-haikou.dtb
<SYMLINK>         34  Fri Mar 12 08:12:09 2021  rk3368-orion-r68-meta.dtb -> rockchip/rk3368-orion-r68-meta.dtb
<SYMLINK>         27  Fri Mar 12 08:12:09 2021  rk3368-px5-evb.dtb -> rockchip/rk3368-px5-evb.dtb
<SYMLINK>         23  Fri Mar 12 08:12:09 2021  rk3368-r88.dtb -> rockchip/rk3368-r88.dtb
<SYMLINK>         23  Fri Mar 12 08:12:09 2021  rk3399-evb.dtb -> rockchip/rk3399-evb.dtb
<SYMLINK>         25  Fri Mar 12 08:12:09 2021  rk3399-ficus.dtb -> rockchip/rk3399-ficus.dtb
<SYMLINK>         27  Fri Mar 12 08:12:09 2021  rk3399-firefly.dtb -> rockchip/rk3399-firefly.dtb
<SYMLINK>         27  Fri Mar 12 08:12:09 2021  rk3399-gru-bob.dtb -> rockchip/rk3399-gru-bob.dtb
<SYMLINK>         29  Fri Mar 12 08:12:09 2021  rk3399-gru-kevin.dtb -> rockchip/rk3399-gru-kevin.dtb
<SYMLINK>         35  Fri Mar 12 08:12:09 2021  rk3399-gru-scarlet-inx.dtb -> rockchip/rk3399-gru-scarlet-inx.dtb
<SYMLINK>         34  Fri Mar 12 08:12:09 2021  rk3399-gru-scarlet-kd.dtb -> rockchip/rk3399-gru-scarlet-kd.dtb
<SYMLINK>         30  Fri Mar 12 08:12:09 2021  rk3399-hugsun-x99.dtb -> rockchip/rk3399-hugsun-x99.dtb
<SYMLINK>         39  Fri Mar 12 08:12:09 2021  rk3399-khadas-edge-captain.dtb -> rockchip/rk3399-khadas-edge-captain.dtb
<SYMLINK>         33  Fri Mar 12 08:12:09 2021  rk3399-khadas-edge-v.dtb -> rockchip/rk3399-khadas-edge-v.dtb
<SYMLINK>         31  Fri Mar 12 08:12:09 2021  rk3399-khadas-edge.dtb -> rockchip/rk3399-khadas-edge.dtb
<SYMLINK>         34  Fri Mar 12 08:12:09 2021  rk3399-kobol-helios64.dtb -> rockchip/rk3399-kobol-helios64.dtb
<SYMLINK>         29  Fri Mar 12 08:12:09 2021  rk3399-leez-p710.dtb -> rockchip/rk3399-leez-p710.dtb
<SYMLINK>         29  Fri Mar 12 08:12:09 2021  rk3399-nanopc-t4.dtb -> rockchip/rk3399-nanopc-t4.dtb
<SYMLINK>         29  Fri Mar 12 08:12:09 2021  rk3399-nanopi-m4.dtb -> rockchip/rk3399-nanopi-m4.dtb
<SYMLINK>         31  Fri Mar 12 08:12:09 2021  rk3399-nanopi-neo4.dtb -> rockchip/rk3399-nanopi-neo4.dtb
<SYMLINK>         28  Fri Mar 12 08:12:09 2021  rk3399-orangepi.dtb -> rockchip/rk3399-orangepi.dtb
<SYMLINK>         32  Fri Mar 12 08:12:09 2021  rk3399-pinebook-pro.dtb -> rockchip/rk3399-pinebook-pro.dtb
<SYMLINK>         31  Fri Mar 12 08:12:09 2021  rk3399-puma-haikou.dtb -> rockchip/rk3399-puma-haikou.dtb
<SYMLINK>         36  Fri Mar 12 08:12:09 2021  rk3399-roc-pc-mezzanine.dtb -> rockchip/rk3399-roc-pc-mezzanine.dtb
<SYMLINK>         26  Fri Mar 12 08:12:09 2021  rk3399-roc-pc.dtb -> rockchip/rk3399-roc-pc.dtb
<SYMLINK>         30  Fri Mar 12 08:12:09 2021  rk3399-rock-pi-4a.dtb -> rockchip/rk3399-rock-pi-4a.dtb
<SYMLINK>         30  Fri Mar 12 08:12:09 2021  rk3399-rock-pi-4b.dtb -> rockchip/rk3399-rock-pi-4b.dtb
<SYMLINK>         30  Fri Mar 12 08:12:09 2021  rk3399-rock-pi-4c.dtb -> rockchip/rk3399-rock-pi-4c.dtb
<SYMLINK>         27  Fri Mar 12 08:12:09 2021  rk3399-rock960.dtb -> rockchip/rk3399-rock960.dtb
<SYMLINK>         32  Fri Mar 12 08:12:09 2021  rk3399-rockpro64-v2.dtb -> rockchip/rk3399-rockpro64-v2.dtb
<SYMLINK>         29  Fri Mar 12 08:12:09 2021  rk3399-rockpro64.dtb -> rockchip/rk3399-rockpro64.dtb
<SYMLINK>         38  Fri Mar 12 08:12:09 2021  rk3399-sapphire-excavator.dtb -> rockchip/rk3399-sapphire-excavator.dtb
<SYMLINK>         28  Fri Mar 12 08:12:09 2021  rk3399-sapphire.dtb -> rockchip/rk3399-sapphire.dtb
<SYMLINK>         34  Fri Mar 12 08:12:09 2021  rk3399pro-rock-pi-n10.dtb -> rockchip/rk3399pro-rock-pi-n10.dtb
<DIR>       2074  Thu Mar 18 10:56:07 2021  rockchip
=> load mmc 1:3 ${fdt_addr_r} /@/.snapshots/1/snapshot/boot/dtb/rk3328-rock64.dtb
BTRFS: An error occurred while reading file /@/.snapshots/1/snapshot/boot/dtb/rk3328-rock64.dtb
Failed to load '/@/.snapshots/1/snapshot/boot/dtb/rk3328-rock64.dtb'


I suppose that this behavior is not platform specific and is related to btrfs mount options (specific compression type or checksum type which are not supported by u-boot).
Comment 1 Guillaume GARDET 2021-03-19 13:13:19 UTC
@Richard, this may be the same problem on the pinebook pro.
Comment 2 Matwey Kornilov 2021-04-01 12:56:56 UTC
I've found the point of the issue.

fs/btrfs/inode.c function btrfs_read_extent_reg()

        ret = btrfs_decompress(btrfs_file_extent_compression(leaf, fi), cbuf,
                               csize, dbuf, dsize);
        if (ret != dsize) {
                ret = -EIO;
                goto out;
        }

ret is not equal to dsize here. ret seems to be the correct uncompressed file size (76074), while dsize is 1750 in my case. Note, that decompress_lzo() is used.

However, I am not sure what does it mean.
Comment 3 Matwey Kornilov 2021-04-01 13:13:51 UTC
I am sorry, dsize is 77824.
Comment 4 Matwey Kornilov 2021-04-01 16:09:56 UTC
I've found that both num_bytes and ram_bytes are 77824, while the file size (when mounting the filesystem at x86_64 PC) is 76074.

Unfortunately, I am not sure how it supposed to be.

However, when the check 

        if (ret != dsize) {
                ret = -EIO;
                goto out;
        }

is disabled, the file can be loaded successfully.
Comment 5 Wenruo Qu 2021-04-01 23:42:43 UTC
Would you please provide the "btrfs ins dump-tree <device>" out put of your fs?

It may be a little large, but it contains all the needed info for us to debug.
Comment 6 Matwey Kornilov 2021-04-02 06:20:45 UTC
Created attachment 847935 [details]
btrfs ins dump-tree output

Attached here is the output of btrfs dump-tree.
The file of interest is meson-sm1-odroid-c4.dtb (I switched to another board with the same issue.)
Comment 7 OBSbugzilla Bot 2021-04-12 13:50:03 UTC
This is an autogenerated message for OBS integration:
This bug (1183717) was mentioned in
https://build.opensuse.org/request/show/884671 Factory:ARM / JeOS
Comment 8 Guillaume GARDET 2021-04-16 16:36:02 UTC
I have a similar problem with JeOS-efi image on Jetson Nano board.
Comment 9 Guillaume GARDET 2021-04-16 16:37:53 UTC
(In reply to Guillaume GARDET from comment #8)
> I have a similar problem with JeOS-efi image on Jetson Nano board.

As a workaround, I use grub to load the DTB along kernel/initrd:
 devicetree /boot/dtb/nvidia/tegra210-p3450-0000.dtb
Comment 10 Matwey Kornilov 2021-04-16 17:07:27 UTC
(In reply to Guillaume GARDET from comment #9)
> (In reply to Guillaume GARDET from comment #8)
> > I have a similar problem with JeOS-efi image on Jetson Nano board.
> 
> As a workaround, I use grub to load the DTB along kernel/initrd:
>  devicetree /boot/dtb/nvidia/tegra210-p3450-0000.dtb

I see that btrfs code in u-boot has not been changed, nor JeOS btrfs mount options. What triggered this issue? This may be a key for the solution.
Comment 11 Wenruo Qu 2021-04-17 02:37:47 UTC
(In reply to Matwey Kornilov from comment #6)
> Created attachment 847935 [details]
> btrfs ins dump-tree output
> 
> Attached here is the output of btrfs dump-tree.
> The file of interest is meson-sm1-odroid-c4.dtb (I switched to another board
> with the same issue.)

The involved data extent is below:

	item 124 key (891 EXTENT_DATA 0) itemoff 5989 itemsize 53
		generation 18 type 1 (regular)
		extent data disk byte 199008256 nr 28672
		extent data offset 0 nr 77824 ram 77824
		extent compression 2 (lzo)

Which looks completely fine.

I'm wondering if it's specific to lzo compression code.

Could you retry with the same fs, but defrag the file with compression=zlib mount option?
Comment 12 Matwey Kornilov 2021-04-17 08:23:48 UTC
I've retried with zlib compression, and the behavior is still the same.

I hope that I recompressed the file correctly:

# compsize meson-sm1-odroid-c4.dtb
Processed 1 file, 1 regular extents (1 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       26%       20K          76K          76K
zlib        26%       20K          76K          76K
Comment 13 Matwey Kornilov 2021-04-17 10:35:09 UTC
(In reply to Guillaume GARDET from comment #8)
> I have a similar problem with JeOS-efi image on Jetson Nano board.

I think this is a common issue for all boards. Some boards may silently fallback to buildin FDT that is capable to booting the kernel. Then the boot happens as usual. Some boards have the FDT incompatible with the kernel (like rk3328), then we see failures.
Comment 14 Wenruo Qu 2021-04-17 12:11:55 UTC
(In reply to Matwey Kornilov from comment #13)
> (In reply to Guillaume GARDET from comment #8)
> > I have a similar problem with JeOS-efi image on Jetson Nano board.
> 
> I think this is a common issue for all boards. Some boards may silently
> fallback to buildin FDT that is capable to booting the kernel. Then the boot
> happens as usual. Some boards have the FDT incompatible with the kernel
> (like rk3328), then we see failures.

The problem is definitely in the uboot code of btrfs.

But I still can't figure out why the decompression code could return value smaller than the full extent size.

In LZO decompression, we're doing per-lzo-header decompression, with some zero padding at page boundary.
And for each decompressed range, we increase the res by the decompressed data size.
Note that, in kernel code, we compress the full page with zero padding, never bothering the inode size, thus we should always have page aligned decompressed size.

BTW, what's the return value in your case? Is that (u32)-1 or some value smaller than dlen?

(I really hope I got a better way to test Uboot in userspace other than booting my VM to test it as bootloader payload.)
Comment 15 Matwey Kornilov 2021-04-17 12:23:44 UTC
(In reply to Wenruo Qu from comment #14)
> (In reply to Matwey Kornilov from comment #13)
> > (In reply to Guillaume GARDET from comment #8)
> > > I have a similar problem with JeOS-efi image on Jetson Nano board.
> > 
> > I think this is a common issue for all boards. Some boards may silently
> > fallback to buildin FDT that is capable to booting the kernel. Then the boot
> > happens as usual. Some boards have the FDT incompatible with the kernel
> > (like rk3328), then we see failures.
> 
> The problem is definitely in the uboot code of btrfs.
> 
> But I still can't figure out why the decompression code could return value
> smaller than the full extent size.
> 
> In LZO decompression, we're doing per-lzo-header decompression, with some
> zero padding at page boundary.
> And for each decompressed range, we increase the res by the decompressed
> data size.
> Note that, in kernel code, we compress the full page with zero padding,
> never bothering the inode size, thus we should always have page aligned
> decompressed size.
> 
> BTW, what's the return value in your case? Is that (u32)-1 or some value
> smaller than dlen?
> 
> (I really hope I got a better way to test Uboot in userspace other than
> booting my VM to test it as bootloader payload.)

Do you mean return of btrfs_decompress()? It returns the file size: 76074 bytes.
Comment 16 Wenruo Qu 2021-04-17 12:31:16 UTC
(In reply to Matwey Kornilov from comment #15)
> (In reply to Wenruo Qu from comment #14)
> > 
> > (I really hope I got a better way to test Uboot in userspace other than
> > booting my VM to test it as bootloader payload.)
> 
> Do you mean return of btrfs_decompress()? It returns the file size: 76074
> bytes.

OK, it turns out that, even in kernel, we can get range smaller than dlen, as we just completely skip the padding zeros:

        /*
         * btrfs_getblock is doing a zero on the tail of the page too,
         * but this will cover anything missing from the decompressed
         * data.
         */
        if (bytes < destlen)
                memset(kaddr+bytes, 0, destlen-bytes);
        kunmap_local(kaddr);
out:

I'll fix it in U-boot soon.

Thanks very much for the report!
Comment 17 Matwey Kornilov 2021-04-17 12:41:40 UTC
(In reply to Wenruo Qu from comment #16)
> (In reply to Matwey Kornilov from comment #15)
> > (In reply to Wenruo Qu from comment #14)
> > > 
> > > (I really hope I got a better way to test Uboot in userspace other than
> > > booting my VM to test it as bootloader payload.)
> > 
> > Do you mean return of btrfs_decompress()? It returns the file size: 76074
> > bytes.
> 
> OK, it turns out that, even in kernel, we can get range smaller than dlen,
> as we just completely skip the padding zeros:
> 
>         /*
>          * btrfs_getblock is doing a zero on the tail of the page too,
>          * but this will cover anything missing from the decompressed
>          * data.
>          */
>         if (bytes < destlen)
>                 memset(kaddr+bytes, 0, destlen-bytes);
>         kunmap_local(kaddr);
> out:
> 
> I'll fix it in U-boot soon.
> 
> Thanks very much for the report!

Thanks. Feel free to ask me to test the patches.
Comment 18 Wenruo Qu 2021-04-17 12:54:33 UTC
(In reply to Matwey Kornilov from comment #17)
> (In reply to Wenruo Qu from comment #16)
> > (In reply to Matwey Kornilov from comment #15)
> > > (In reply to Wenruo Qu from comment #14)
> > > > 
> > > > (I really hope I got a better way to test Uboot in userspace other than
> > > > booting my VM to test it as bootloader payload.)
> > > 
> > > Do you mean return of btrfs_decompress()? It returns the file size: 76074
> > > bytes.
> > 
> > OK, it turns out that, even in kernel, we can get range smaller than dlen,
> > as we just completely skip the padding zeros:
> > 
> >         /*
> >          * btrfs_getblock is doing a zero on the tail of the page too,
> >          * but this will cover anything missing from the decompressed
> >          * data.
> >          */
> >         if (bytes < destlen)
> >                 memset(kaddr+bytes, 0, destlen-bytes);
> >         kunmap_local(kaddr);
> > out:
> > 
> > I'll fix it in U-boot soon.
> > 
> > Thanks very much for the report!
> 
> Thanks. Feel free to ask me to test the patches.

This time I won't bother you, as I just found how to do U-boot sandbox testing.

Now the v2 patch should fix the problem properly. (The v1 just fixes the inline read, but not regular extents read).

Anyway, thank you for the detailed report again!
Comment 21 Guillaume GARDET 2021-04-19 06:48:41 UTC
Great we have a patch. Would you mind to post the link here, please?
That way, we could backport it to fix the current Leap and Tumbleweed u-boot packages.
Comment 22 Wenruo Qu 2021-04-19 07:09:57 UTC
(In reply to Guillaume GARDET from comment #21)
> Great we have a patch. Would you mind to post the link here, please?
> That way, we could backport it to fix the current Leap and Tumbleweed u-boot
> packages.

Oh, I should reply in public.

Here is the patch:
https://patchwork.ozlabs.org/project/uboot/patch/20210417125213.132066-1-wqu@suse.com/
Comment 23 Guillaume GARDET 2021-04-19 07:23:39 UTC
Submission to hardware:boot:staging/u-boot: https://build.opensuse.org/request/show/886583

@Matthias, could you take care of the SLE package, please?
Comment 24 Matwey Kornilov 2021-04-19 07:46:41 UTC
And also not to forget to revert btrfs for rockchip based JeOSes.
Comment 25 Matthias Brugger 2021-04-19 09:46:03 UTC
(In reply to Guillaume GARDET from comment #23)
> Submission to hardware:boot:staging/u-boot:
> https://build.opensuse.org/request/show/886583
> 
> @Matthias, could you take care of the SLE package, please?

Will do. See bsc#1184947
Comment 26 Matthias Brugger 2021-04-19 09:48:36 UTC
(In reply to Matwey Kornilov from comment #24)
> And also not to forget to revert btrfs for rockchip based JeOSes.

I suppose you mean reverting https://build.opensuse.org/request/show/884671
Correct? Guillaume can you take care of this?
Comment 27 Matwey Kornilov 2021-04-19 09:50:21 UTC
(In reply to Matthias Brugger from comment #26)
> (In reply to Matwey Kornilov from comment #24)
> > And also not to forget to revert btrfs for rockchip based JeOSes.
> 
> I suppose you mean reverting https://build.opensuse.org/request/show/884671
> Correct? Guillaume can you take care of this?

Yes, since we have the correct fix. The workaround is not required anymore.
Comment 28 Guillaume GARDET 2021-04-19 09:52:35 UTC
(In reply to Matthias Brugger from comment #26)
> (In reply to Matwey Kornilov from comment #24)
> > And also not to forget to revert btrfs for rockchip based JeOSes.
> 
> I suppose you mean reverting https://build.opensuse.org/request/show/884671
> Correct? Guillaume can you take care of this?

I will, once u-boot update will be in Factory.
Comment 29 OBSbugzilla Bot 2021-04-27 07:40:38 UTC
This is an autogenerated message for OBS integration:
This bug (1183717) was mentioned in
https://build.opensuse.org/request/show/888681 Factory:ARM / JeOS
Comment 30 Matthias Brugger 2021-05-05 11:51:38 UTC
(In reply to Guillaume GARDET from comment #28)
> (In reply to Matthias Brugger from comment #26)
> > (In reply to Matwey Kornilov from comment #24)
> > > And also not to forget to revert btrfs for rockchip based JeOSes.
> > 
> > I suppose you mean reverting https://build.opensuse.org/request/show/884671
> > Correct? Guillaume can you take care of this?
> 
> I will, once u-boot update will be in Factory.

U-Boot got updated. I'll close this bug. Guillaume will you take care to revert the BTRFS workaround in JeOS?
Comment 31 Guillaume GARDET 2021-05-05 12:25:41 UTC
> U-Boot got updated. I'll close this bug. Guillaume will you take care to
> revert the BTRFS workaround in JeOS?

No, because the DTB is not loaded properly. I suspect the @ and .snapshots paths, specific to btrfs, to break it. We should have the default path to use latest version of the file.
Comment 32 Matthias Brugger 2021-05-05 13:15:52 UTC
(In reply to Guillaume GARDET from comment #31)
> > U-Boot got updated. I'll close this bug. Guillaume will you take care to
> > revert the BTRFS workaround in JeOS?
> 
> No, because the DTB is not loaded properly. I suspect the @ and .snapshots
> paths, specific to btrfs, to break it. We should have the default path to
> use latest version of the file.

Is there a bug report for that?
Comment 33 Guillaume GARDET 2021-05-05 13:38:38 UTC
(In reply to Matthias Brugger from comment #32)
> (In reply to Guillaume GARDET from comment #31)
> > > U-Boot got updated. I'll close this bug. Guillaume will you take care to
> > > revert the BTRFS workaround in JeOS?
> > 
> > No, because the DTB is not loaded properly. I suspect the @ and .snapshots
> > paths, specific to btrfs, to break it. We should have the default path to
> > use latest version of the file.
> 
> Is there a bug report for that?

I thought there was one, but I am unable to find it, so I created bug #1185656
Comment 34 OBSbugzilla Bot 2021-10-12 16:41:50 UTC
This is an autogenerated message for OBS integration:
This bug (1183717) was mentioned in
https://build.opensuse.org/request/show/924915 Factory:ARM / JeOS
Comment 35 OBSbugzilla Bot 2021-10-12 18:41:44 UTC
This is an autogenerated message for OBS integration:
This bug (1183717) was mentioned in
https://build.opensuse.org/request/show/924923 Factory:ARM / JeOS