Add Guillaume to CC

WenRou can you have a look?

To me, this looks like something wrong in the initramfs.

Since for aarch64 we're still using 4K page size, x86_64 should be able to mount and verify the fs.

The operation not supported is caused by the qemu emulation.

For aarch64 since we're still using 4K page size, x86 should be able to mount the fs without problem, then you can do all your check there.

If "btrfs check" reports no majar problem, I think it's something more related to the initramfs.

Maybe adding "debug" kernel cmdline option would help debugging it?

Here is the serial output with debug set as kernel option.

https://susepaste.org/34312074

I was able to mount the file system on my Odroid C2 SBC and chroot into it.

I force reinstalled kernel-default 5.15.3-1.4 - FAIL No change in the boot process.

I ran dracut --force to rebuild the initramfs - FAIL No change in the boot process.

I can boot from the 4.14.14-3.3 kernel. So it seems to be some change in the kernel from 5.14 to 5.15.

Sorry for the poor explanation in my previous post.

I was able to reinstall the kernel-default package, but booting the board failed the same as the original bug report.

I was able to rebuild the initramfs, but booting the board failed the same as the original bug report.

>[  175.161700] dracut-initqueue[291]: Warning: dracut[  175.667247] dracut-initqueue[291]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
> [  175.690747] dracut-initqueue[291]: Warning: /lib/dracut/hooksev\x2fdisk\x2fby-uuid\x2f056C-D441.sh: "[ -e "/dev/disk/by-uuid/056C-D441" ]"
> [  175.720596] dracut-initqueue[291]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x-uuid\x2f4e0b4d23-0e9b-4d86-9ce8-fd17dc2332d8.sh: "if ! grep -q After=remote-fs-pre.target /run/temd-cryptsetup@*.service 2>/dev/nun

This looks like the initramfs can not found the root block device.

In fact there is no btrfs related dmesg except the initial load, thus it means it's lower layer, mostly the drivers causing the problem, not btrfs.

Mind to provide the `lsblk` output when using older kernel?
I want to make sure where the rootfs is (usb or sdcard or NVME or whatever).

Also, for the older bootable kernel, please provide the dmesg of a successful boot, as a baseline.

My current assumption is there is some regression related to the root partition you're using (Rock64 dts change maybe?)

unifi:~ # lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
mmcblk1      179:0    0 14.5G  0 disk 
├─mmcblk1p1  179:1    0   16M  0 part /boot/efi
├─mmcblk1p2  179:2    0  500M  0 part [SWAP]
└─mmcblk1p3  179:3    0 13.9G  0 part /var/lib/docker/btrfs
                                      /var
                                      /usr/local
                                      /tmp
                                      /srv
                                      /root
                                      /opt
                                      /home
                                      /boot/grub2/arm64-efi
                                      /.snapshots
                                      /
mmcblk1boot0 179:8    0    4M  1 disk 
mmcblk1boot1 179:16   0    4M  1 disk

Here is a link to the dmesg: https://susepaste.org/16187376

OK, it's more clear that, the newer kernel doesn't even detect mmc, thus no mountable btrfs.

Just to be safer, please provide the dmesg of the failed boot, with "loglevel=7" kernel option for newer kernels, so that we can catch all the dmesg.

Now I guess it's some rockchip related DTS update causing the problem, as they also break my rockpro64 ethernet driver.

Here is the serial output with the 5.15.3 kernel with option loglevel=7

https://susepaste.org/98551531

Now confirmed. With newer kernel there is no MMC detected at all, thus looks like a regression for the board.

And no mmc, no btrfs.

Mark, can you have a look if you see any significant change using lsinitrd on both systems? Maybe we are missing the driver in the initrd.

The difference I see between the two initrd involving mmc are in the 5.14 kernel the drivers are compressed as xz, while in the 5.15 kernel they are compressed as zst.

5.14
drwxr-xr-x   1 root     root            0 Dec  1 19:38 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc
drwxr-xr-x   1 root     root            0 Dec  1 19:38 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/core
-rw-r--r--   1 root     root        25436 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/core/mmc_block.ko.xz
-rw-r--r--   1 root     root        68064 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/core/mmc_core.ko.xz
-rw-r--r--   1 root     root         5764 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/core/pwrseq_emmc.ko.xz
drwxr-xr-x   1 root     root            0 Dec  1 19:38 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/host
-rw-r--r--   1 root     root         6828 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/host/dw_mmc-pltfm.ko.xz
-rw-r--r--   1 root     root         8936 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/host/dw_mmc-rockchip.ko.xz
-rw-r--r--   1 root     root        23056 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/host/dw_mmc.ko.xz
-rw-r--r--   1 root     root        12424 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/host/meson-gx-mmc.ko.xz

5.15
drwxr-xr-x   1 root     root            0 Dec  1 19:38 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc
drwxr-xr-x   1 root     root            0 Dec  1 19:38 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/core
-rw-r--r--   1 root     root        25436 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/core/mmc_block.ko.xz
-rw-r--r--   1 root     root        68064 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/core/mmc_core.ko.xz
-rw-r--r--   1 root     root         5764 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/core/pwrseq_emmc.ko.xz
drwxr-xr-x   1 root     root            0 Dec  1 19:38 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/host
-rw-r--r--   1 root     root         6828 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/host/dw_mmc-pltfm.ko.xz
-rw-r--r--   1 root     root         8936 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/host/dw_mmc-rockchip.ko.xz
-rw-r--r--   1 root     root        23056 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/host/dw_mmc.ko.xz
-rw-r--r--   1 root     root        12424 Nov 15 23:53 usr/lib/modules/5.14.14-3-default/kernel/drivers/mmc/host/meson-gx-mmc.ko.xz

Here is a link to the full 5.14 lsinitrd: https://susepaste.org/77437141

Here is a link to the full 5.15 lsinitrd: https://susepaste.org/83265176

Oops, sorry. I am wrong on that. They are both compressed with xz. - Apparently I'm still sleepy.

it seems no kernel module is missing and no udev rule has changed neither.

Any DTS change?

IIRC in v5.14.x update, there is some rockchip related change in the DTS, causing my rockpro64 losing its eth phy.

Maybe it's a similar case?

I upgraded the kernel to 5.15.7-1.1 with the same no-boot issue. - The root filesystem on the eMMC is not detected.

I installed the proprietary TPL bootloader on the eMMC but got the same no-boot issue with the 5.15.7 kernel.

The latest Jeos 20211214 image flashed to a uSD card boots the 5.15.7-1.1 kernel without issue.

Is there any hope of someone knowledgeable looking at this? or am I stuck on 5.14 kernel or a different operating system?

I am willing to do some testing, but I have no idea what to modify etc in the dtb or kernel config.

Thanks.

Mark

I guess our ARM team is a little busy, thus please allow me to help (I'm just a BTRFS guys with tons of interest in ARM boards).

First thing first, I don't have Rock64 board (it's a little poor in performance, the same reason I have bought a pinephone).
Thus I can't do testing on real hardware.

But on the other hand, I have a strong feeling it's the device tree causing the problem.

Would you please upload the dtb file for your board? Both working and not working one, so that I can use fdt to decode the dtb and see what's changed.

Created attachment 854685 [details]
5.15 dtb

Attached is the 5.15 dtb. Unfortunately I don't have the 5.14 dtb. The oldest btrfs snapshot I have has the 5.15 dtb's only. I had added 5.14.14 to the kernel multiversion line in zypp.conf so it didn't get uninstalled however, installing the 5.15 kernel installed the 5.15 dtb's. I didn't perform a zypper in -f kernel-default-5.14.14-3.3 after I reverted back to the 5.14 kernel, and now the 5.14.14-3.3 kernel isn't available in the repository. The board boots the 5.14 kernel with the 5.15 dtb.

So for the bootable kernel with newer dtb, it can still boot without problem?

Then the things are getting complex.

From the dtc, the involved two mmcs are:

                mmc0 = "/mmc@ff500000";
                mmc1 = "/mmc@ff520000";

And their definitions are:

        mmc@ff500000 {
                compatible = "rockchip,rk3328-dw-mshc\0rockchip,rk3288-dw-mshc";
                reg = <0x00 0xff500000 0x00 0x4000>;
                interrupts = <0x00 0x0c 0x04>;
                clocks = <0x02 0x13d 0x02 0x21 0x02 0x4a 0x02 0x4e>;
                clock-names = "biu\0ciu\0ciu-drive\0ciu-sample";
                fifo-depth = <0x100>;
                max-frequency = <0x8f0d180>;
                status = "okay";
                bus-width = <0x04>;
                cap-mmc-highspeed;
                cap-sd-highspeed;
                disable-wp;
                pinctrl-names = "default";
                pinctrl-0 = <0x47 0x48 0x49 0x4a>;
                vmmc-supply = <0x4b>;
                phandle = <0x8f>;
        };

And

        mmc@ff520000 {
                compatible = "rockchip,rk3328-dw-mshc\0rockchip,rk3288-dw-mshc";
                reg = <0x00 0xff520000 0x00 0x4000>;
                interrupts = <0x00 0x0e 0x04>;
                clocks = <0x02 0x13f 0x02 0x23 0x02 0x4c 0x02 0x50>;
                clock-names = "biu\0ciu\0ciu-drive\0ciu-sample";
                fifo-depth = <0x100>;
                max-frequency = <0x8f0d180>;
                status = "okay";
                bus-width = <0x08>;
                cap-mmc-highspeed;
                mmc-hs200-1_8v;
                non-removable;
                pinctrl-names = "default";
                pinctrl-0 = <0x4c 0x4d 0x4e>;
                vmmc-supply = <0x1c>;
                vqmmc-supply = <0x1d>;
                phandle = <0x91>;
        };


And I didn't see any change after 2014 related to the compatible string nor the dw_mmc-rockchip driver.


For the unbootable kernel + fdt combination, if you can still access the emergency shell, and got a way to transfer file,
could you upload the `/sys/firmware/fdt` of the unbootable kernel/fdt combination?

Furthermore, my current RK3399 board is using the same rk3288-dw-mshc driver for sdmmc.

And I haven't notice any regression since v5.15.

Hi,

I've just tested 

openSUSE-Tumbleweed-ARM-JeOS-rock64.aarch64-2021.12.06-Build1.23.raw.xz

and it boots fine on my Rock64.

Created attachment 854718 [details]
/sys/firmware/fdt

I'm not sure if this file is readable/helpful, minicom and my serial adapter don't always play well. - If there is a better way to get the info let me know.

Mark

Matwey Kornilov was that installation running on an eMMC? uSD card installations work for me.

Mark

(In reply to Mark Petersen from comment #25)
> Matwey Kornilov was that installation running on an eMMC? uSD card
> installations work for me.
> 
> Mark

It is uSD. My rock64 doesn't have eMMC, so I cannot test it.

This is a very old bug. Closing this in a hope this got fixed upstream. Sorry for that.