Bug 1122614 - armv7 efistub enablement missing in GRUB2
armv7 efistub enablement missing in GRUB2
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Bootloader
Current
armv7 openSUSE Factory
: P5 - None : Normal (vote)
: ---
Assigned To: Michael Chang
Jiri Srain
:
Depends on: 1104833
Blocks: 1129950 1145646
  Show dependency treegraph
 
Reported: 2019-01-21 10:43 UTC by Guillaume GARDET
Modified: 2019-12-17 14:23 UTC (History)
14 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---
mchang: needinfo? (afaerber)


Attachments
lsefimmap working on arndale board with u-boot and Grub2.04 (1.07 KB, text/x-log)
2019-07-10 07:52 UTC, Guillaume GARDET
Details
lsefimmap broken on RPi2 board with u-boot and grub 2.04 (3.01 MB, image/jpeg)
2019-07-10 07:53 UTC, Guillaume GARDET
Details
lsefimmap broken on Chromebook with u-boot and grub 2.04 (2.74 MB, image/jpeg)
2019-07-10 09:09 UTC, Guillaume GARDET
Details
lsefimmap working on qemu with UEFI firmware (1.54 KB, text/plain)
2019-07-10 09:15 UTC, Guillaume GARDET
Details
0001-arm-efi-use-conventional-memory-region-for-kernel.patch (2.08 KB, patch)
2019-07-11 05:38 UTC, Joey Lee
Details | Diff
A patch to solve FDT paging issue. (887 bytes, patch)
2019-07-22 04:03 UTC, Chester Lin
Details | Diff
0001-efi-arm-fix-allocation-failure-when-reserving-the-ke.patch (4.61 KB, patch)
2019-08-01 08:19 UTC, Chester Lin
Details | Diff
armv6 data abort after kernel and initrd are loaded. (642 bytes, text/plain)
2019-08-01 11:50 UTC, Guillaume GARDET
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Guillaume GARDET 2019-01-21 10:43:34 UTC
+++ This bug was initially created as a clone of Bug #1104833 +++

While solving initramfs bug (boo#1104833) on armv7, discussions about efistub on armv7 came up. This bug is the new place to discuss it, as original bug is fixed.

Background: efistub has been enabled in kernel 4.20. To use it, we need to enable it in grub2, but once enabled, non-efistub kernel will stop to work, as per comment https://bugzilla.opensuse.org/show_bug.cgi?id=1104833#c44
Comment 1 Andreas Färber 2019-01-21 18:09:36 UTC
(In reply to Guillaume GARDET from comment #0)
> Background: efistub has been enabled in kernel 4.20. To use it, we need to
> enable it in grub2, but once enabled, non-efistub kernel will stop to work,
> as per comment https://bugzilla.opensuse.org/show_bug.cgi?id=1104833#c44

We now have kernel 4.20.0 with CONFIG_EFI_STUB=y built in openSUSE:Factory:ARM.

Caution: Last _published_ Factory armv7hl kernel packages are still at 4.19.12.

@Michael, can you work with Guillaume to a) backport the 32-bit arm efistub patches that were mentioned to our Factory grub2 package (or maybe update it :)) and b) have Arm look into backwards compatibility for non-efistub enabled kernels if still missing? Thanks!
Comment 2 Guillaume GARDET 2019-07-05 13:30:23 UTC
Grub 2.04 has been released: https://lists.gnu.org/archive/html/grub-devel/2019-07/msg00000.html

@Michael, could you update grub accordingly, please? As grub 2.04 has the required patch.

Tumbleweed kernel has CONFIG_EFI_STUB=y since a while, so it is safe to update grub in Tumbleweed.
Comment 3 Michael Chang 2019-07-08 08:29:50 UTC
(In reply to Guillaume GARDET from comment #2)
> Grub 2.04 has been released:
> https://lists.gnu.org/archive/html/grub-devel/2019-07/msg00000.html
> 
> @Michael, could you update grub accordingly, please? As grub 2.04 has the
> required patch.

The 2.04 upgrade is currently tracked in the project.

https://build.opensuse.org/package/show/home:michael-chang:devel/grub2

I have tested it on ThunderX2 (aarch64) and worked without problem so far. Would it be possible for you to test on the armv7 if you have one around the corner? If not it will take some time from me to find the hardware. 

> Tumbleweed kernel has CONFIG_EFI_STUB=y since a while, so it is safe to
> update grub in Tumbleweed.

Thanks for the info, do we still care about backward compatibility (ie booting kernels without efi stub, with what Andreas already mentioned in comment#1) ?
Comment 4 Guillaume GARDET 2019-07-08 08:48:10 UTC
(In reply to Michael Chang from comment #3)
> The 2.04 upgrade is currently tracked in the project.
> 
> https://build.opensuse.org/package/show/home:michael-chang:devel/grub2
> 
> I have tested it on ThunderX2 (aarch64) and worked without problem so far.
> Would it be possible for you to test on the armv7 if you have one around the
> corner? If not it will take some time from me to find the hardware. 

I will test on armv7 and I will post here the results.

> > Tumbleweed kernel has CONFIG_EFI_STUB=y since a while, so it is safe to
> > update grub in Tumbleweed.
> 
> Thanks for the info, do we still care about backward compatibility (ie
> booting kernels without efi stub, with what Andreas already mentioned in
> comment#1) ?

From my point of view, it is not necessary. @andreas, WDYT?
Comment 5 Guillaume GARDET 2019-07-08 13:09:16 UTC
armv7 tests results:
* qemu armv7 is ok
* Raspberry Pi 2, model B is broken, as the Grub menu is not shown on serial anymore, only on HDMI output, and once we want to boot a kernel, we get: 
  EFI stub: Booting Linux Kernel...
  EFI stub: ERROR: Unable to allocate memory for uncompressed kernel.
  EFI stub: ERROR: Failed to relocate kernel

My test on RPi2 B has been done by using latest released Tumbleweed image and then updating grub2* 2.04 packages from Michael's home project.
Comment 6 Guillaume GARDET 2019-07-08 14:24:36 UTC
Also broken on Chromebook ARM, but I cannot give much details on this failure as I have no serial on this device and screen goes back to grub menu.

bootarm.efi is very light now, around 170KB instead of 750KB previously.

If I re-use the file from /boot/grub2/arm-efi/grub.efi from rootfs partition (not updated apparently) and copy it to (EFI part)/EFI/BOOT/bootarm.efi it does boot again.

If I copy /usr/share/grub2/arm-efi/grub.efi, then it does fail too.
Comment 7 Michael Chang 2019-07-09 05:34:02 UTC
(In reply to Guillaume GARDET from comment #5)
> armv7 tests results:
> * qemu armv7 is ok
> * Raspberry Pi 2, model B is broken, as the Grub menu is not shown on serial
> anymore, only on HDMI output, and once we want to boot a kernel, we get:

AFAIK, the serial output is redirected to "console" in efi, so it is unclear to me which driver you were talking here? Is it text based "console" (with serial redirection to it) or the native "serial" console ?
 
>   EFI stub: Booting Linux Kernel...
>   EFI stub: ERROR: Unable to allocate memory for uncompressed kernel.
>   EFI stub: ERROR: Failed to relocate kernel

The kernel message suggested we got some issues in the new efistub enablement, previously the booting was done by non-efistub entry (zImage) and it did not fail.

> My test on RPi2 B has been done by using latest released Tumbleweed image
> and then updating grub2* 2.04 packages from Michael's home project.

I think in general it would be helpful to verify the upstream build in this regard to see whether a regression in upstream. In this case very much likely as we are much aligned with upstream for armv7. I am currently working on obs project providing vanilla grub package, and will let you know once finished.
Comment 8 Michael Chang 2019-07-09 06:12:14 UTC
(In reply to Guillaume GARDET from comment #6)
> Also broken on Chromebook ARM, but I cannot give much details on this
> failure as I have no serial on this device and screen goes back to grub menu.
> 
> bootarm.efi is very light now, around 170KB instead of 750KB previously.

I compared the kernel.img size between packages in Base:System and 2.04, there's no much difference (88KB vs 87KB).

Looks like the comparison is between /boot/grub2/arm-efi/grub.efi and bootarm.efi created by grub2-install. The first is created in OBS with lots of modules builtin, used in boot media, secure boot and so on. The latter is created wrt the runtime system by grub2-install, only with minimum required modules built-in.

Hm. Now that it seems to me the serial console problem could be related. Try if this work ? 

  grub2-install --modules=serial

> If I re-use the file from /boot/grub2/arm-efi/grub.efi from rootfs partition
> (not updated apparently) and copy it to (EFI part)/EFI/BOOT/bootarm.efi it
> does boot again.
> 
> If I copy /usr/share/grub2/arm-efi/grub.efi, then it does fail too.

The failure might be in the same efistub, we can check by new kernel update after we have it fixed.
Comment 9 Michael Chang 2019-07-09 06:16:53 UTC
Looped in kernel team to diagnose the efistub problem.
Comment 10 Guillaume GARDET 2019-07-09 08:54:13 UTC
(In reply to Michael Chang from comment #8)
> Hm. Now that it seems to me the serial console problem could be related. Try
> if this work ? 
> 
>   grub2-install --modules=serial


It returns:
  Installing for arm-uboot platform.
  grub2-install: warning: no hints available for your platform. Expect reduced performance.
  grub2-install: error: cannot open `/usr/share/grub2/arm-uboot/serial.mod': No such file or directory.


And there is no grub.efi in /usr/share/grub2/arm-uboot/ is it expected?
Comment 11 Michael Chang 2019-07-09 09:44:24 UTC
(In reply to Guillaume GARDET from comment #10)
> (In reply to Michael Chang from comment #8)
> > Hm. Now that it seems to me the serial console problem could be related. Try
> > if this work ? 
> > 
> >   grub2-install --modules=serial
> 
> 
> It returns:
>   Installing for arm-uboot platform.
>   grub2-install: warning: no hints available for your platform. Expect
> reduced performance.
>   grub2-install: error: cannot open `/usr/share/grub2/arm-uboot/serial.mod':
> No such file or directory.
> 
> 
> And there is no grub.efi in /usr/share/grub2/arm-uboot/ is it expected?

Sorry I think the correct command is :

  grub2-install --with-platform=arm-efi -no-nvram --removable --modules=serial

The default architecture of grub utility we packaged for armv7 is arm-uboot. 
While running it will detect efi firmware through the presence of /sys/firmware/efi and change the installation to arm-efi architecture accordingly. For the EFI on U-Boot, no efi boot variable on the system so no 
/sys/firmware/efi, therefore we need to manually specify the platform parameters for grub-install (This is also how update-bootloader handles it)
Comment 12 Gary Ching-Pang Lin 2019-07-09 09:56:55 UTC
From the error message, it looks like the kernel failed at reserve_kernel_base(). The kernel might try to allocate the memory pages in a memory region other than EFI_BOOT_SERVICE_* and EFI_CONVENTIONAL_MEMORY. What I don't understand is why reserve_kernel_base() couldn't just find a free region and set dram_base accordingly.

https://github.com/torvalds/linux/blob/v5.1/drivers/firmware/efi/libstub/arm32-stub.c#L64-L188
Comment 13 Guillaume GARDET 2019-07-09 11:21:12 UTC
(In reply to Michael Chang from comment #11)
> Sorry I think the correct command is :
> 
>   grub2-install --with-platform=arm-efi -no-nvram --removable
> --modules=serial

The correct command seems to be:
  grub2-install --target=arm-efi --no-nvram --removable --modules=serial

But it does not help. Even copying /usr/share/grub2/arm-efi/grub.efi does not show the grub menu on serial. So, maybe some drivers are missing/broken?
Comment 14 Guillaume GARDET 2019-07-09 11:47:03 UTC
(In reply to Gary Ching-Pang Lin from comment #12)
> From the error message, it looks like the kernel failed at
> reserve_kernel_base(). The kernel might try to allocate the memory pages in
> a memory region other than EFI_BOOT_SERVICE_* and EFI_CONVENTIONAL_MEMORY.
> What I don't understand is why reserve_kernel_base() couldn't just find a
> free region and set dram_base accordingly.
> 
> https://github.com/torvalds/linux/blob/v5.1/drivers/firmware/efi/libstub/
> arm32-stub.c#L64-L188

Maybe u-boot does not pass the right information? As UEFI based qemu system is fine.
Comment 15 Andreas Färber 2019-07-09 16:57:15 UTC
Michael, can you just drop the non-portable arm-uboot target and default to the arm-efi target like on aarch64?

Guillaume, any chance you or someone else from Arm could look into restoring non-efistub boot support upstream, from Git history? It affects the upgrade paths from Leap to Tumbleweed as well as to future Leap versions that may inherit this Factory grub2, plus any other old or new non-SUSE kernels that chose not to enable efistub. Doesn't need to hold up this submission, but should be mentioned in .changes & taken care of before such Factory package gets branched for 15.2.
Comment 16 Chester Lin 2019-07-10 03:53:08 UTC
(In reply to Guillaume GARDET from comment #14)
> (In reply to Gary Ching-Pang Lin from comment #12)
> > From the error message, it looks like the kernel failed at
> > reserve_kernel_base(). The kernel might try to allocate the memory pages in
> > a memory region other than EFI_BOOT_SERVICE_* and EFI_CONVENTIONAL_MEMORY.
> > What I don't understand is why reserve_kernel_base() couldn't just find a
> > free region and set dram_base accordingly.
> > 
> > https://github.com/torvalds/linux/blob/v5.1/drivers/firmware/efi/libstub/
> > arm32-stub.c#L64-L188
> 
> Maybe u-boot does not pass the right information? As UEFI based qemu system
> is fine.

grub has a 'lsefimmap' command to dump entire EFI memmap, would you mind if you can try it so that we can check if there's any difference between UEFI and U-Boot?
Comment 17 Michael Chang 2019-07-10 06:22:05 UTC
(In reply to Andreas Färber from comment #15)
> Michael, can you just drop the non-portable arm-uboot target and default to
> the arm-efi target like on aarch64?

Sorry, maybe "default" is a bit misleading here. The grub2-install has to support arm-uboot and arm-efi targets. Without any target specified it will try to figure it out via testing the /sys/firmware/efi. (It is therefore more likely a runtime behavior than a platform specific default bounded to configure).

The behavior above is common for grub2-install built from ./configure --with-platfomr=efi and /configure --with-platform=uboot. It did not change if we change the packaged grub2-install from uboot to efi.

Or do you mean to drop the arm-uboot package ?
Comment 18 Michael Chang 2019-07-10 07:19:36 UTC
(In reply to Guillaume GARDET from comment #13)

> But it does not help. Even copying /usr/share/grub2/arm-efi/grub.efi does
> not show the grub menu on serial. So, maybe some drivers are missing/broken?

The git-log did not reveal upstream serial driver for efi received any change since 2.02

> git --no-pager log --oneline 2.02..grub-2.04 -- grub-core/term/serial.c grub-core/term/efi/serial.c

Also the firmware's console driver has no obvious change.

> git --no-pager log --oneline 2.02..grub-2.04 -- grub-core/term/efi/console.c
> edece25a7 efi/console: Fix the "enter" key not working on x86 tablets
> bdd89d239 core: use GRUB_TERM_ definitions when handling term characters

Also checked the Makefile.core.def, moddep.lst did not change for the platform module in use and dependency wrt arm-efi console/serial driver.

Would you please attach grub.cfg for reference ? Maybe can find some clue there. Meanwhile I'll try to get RPI2 to reproduce the problem.
Comment 19 Guillaume GARDET 2019-07-10 07:42:34 UTC
I tested grub 2.04 on Arndale board (armv7 with 2GB of RAM) and it does boot properly! And EFI is exposed to /sys/firmware/efi/.

On RPi2, we use a DT provided and updated by the firmware. Could it be the problem?
Comment 20 Guillaume GARDET 2019-07-10 07:52:33 UTC
Created attachment 809934 [details]
lsefimmap working on arndale board with u-boot and Grub2.04
Comment 21 Guillaume GARDET 2019-07-10 07:53:22 UTC
Created attachment 809935 [details]
lsefimmap broken on RPi2 board with u-boot and grub 2.04
Comment 22 Gary Ching-Pang Lin 2019-07-10 08:19:30 UTC
I suspect that the kernel tried to allocate the memory in the "reserved" 4KB region.

For Arndale, the first region is free and large enough, so the kernel can just allocate a space for the uncompressed kernel.

The kernel probably has to find the first WB and EFI_CONVENTIONAL region instead of just looking the first WB region.
Comment 23 Gary Ching-Pang Lin 2019-07-10 08:33:22 UTC
Just checked the u-boot code, it always marks the first page as WB and RESERVED for its own spin table.

Maybe we can change the logic of get_dram_base() in kernel from

if (md->attribute & EFI_MEMORY_WB) {

to

if (md->attribute & EFI_MEMORY_WB &&
    md->type == EFI_CONVENTIONAL_MEMORY) {

This way, get_dram_base() will always find the first free region.
Comment 24 Joey Lee 2019-07-10 08:46:00 UTC
(In reply to Gary Ching-Pang Lin from comment #23)
> Just checked the u-boot code, it always marks the first page as WB and
> RESERVED for its own spin table.
> 
> Maybe we can change the logic of get_dram_base() in kernel from
> 
> if (md->attribute & EFI_MEMORY_WB) {
> 
> to
> 
> if (md->attribute & EFI_MEMORY_WB &&
>     md->type == EFI_CONVENTIONAL_MEMORY) {
> 
> This way, get_dram_base() will always find the first free region.

(In reply to Gary Ching-Pang Lin from comment #23)
> Just checked the u-boot code, it always marks the first page as WB and
> RESERVED for its own spin table.
> 
> Maybe we can change the logic of get_dram_base() in kernel from
> 
> if (md->attribute & EFI_MEMORY_WB) {
> 
> to
> 
> if (md->attribute & EFI_MEMORY_WB &&
>     md->type == EFI_CONVENTIONAL_MEMORY) {
> 
> This way, get_dram_base() will always find the first free region.

As Gary's suggestion, I am building kernel RPM for testing:

---
 drivers/firmware/efi/libstub/efi-stub-helper.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/drivers/firmware/efi/libstub/efi-stub-helper.c
+++ b/drivers/firmware/efi/libstub/efi-stub-helper.c
@@ -153,7 +153,7 @@ unsigned long get_dram_base(efi_system_t
        map.map_end = map.map + map_size;

        for_each_efi_memory_desc_in_map(&map, md) {
-               if (md->attribute & EFI_MEMORY_WB) {
+               if ((md->type == EFI_CONVENTIONAL_MEMORY) && (md->attribute & EFI_MEMORY_WB)) {
                        if (membase > md->phys_addr)
                                membase = md->phys_addr;
                }
Comment 25 Guillaume GARDET 2019-07-10 09:09:43 UTC
Created attachment 809966 [details]
lsefimmap broken on Chromebook with u-boot and grub 2.04

The Chromebook is very similar to arndale board and has a similar efimmap but the boot is broken. I do not have any messages though, as there is no serial and once kernel/initrd are loaded, grub menu is reloaded and no error message are readable.
Comment 26 Guillaume GARDET 2019-07-10 09:15:14 UTC
Created attachment 809969 [details]
lsefimmap working on qemu with UEFI firmware
Comment 27 Andreas Färber 2019-07-10 10:56:44 UTC
(In reply to Michael Chang from comment #17)
> Or do you mean to drop the arm-uboot package ?

Yes.

1) It hardcoded the physical RAM location and would need per-SoC builds.
2) This feature is not enabled in U-Boot by default, unusable with our packages.
Comment 28 Joey Lee 2019-07-10 13:15:46 UTC
(In reply to Joey Lee from comment #24)
> (In reply to Gary Ching-Pang Lin from comment #23)
[...snip]
> > 
> > if (md->attribute & EFI_MEMORY_WB &&
> >     md->type == EFI_CONVENTIONAL_MEMORY) {
> > 
> > This way, get_dram_base() will always find the first free region.
> 
> As Gary's suggestion, I am building kernel RPM for testing:
> 
> ---
>  drivers/firmware/efi/libstub/efi-stub-helper.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> --- a/drivers/firmware/efi/libstub/efi-stub-helper.c
> +++ b/drivers/firmware/efi/libstub/efi-stub-helper.c
> @@ -153,7 +153,7 @@ unsigned long get_dram_base(efi_system_t
>         map.map_end = map.map + map_size;
> 
>         for_each_efi_memory_desc_in_map(&map, md) {
> -               if (md->attribute & EFI_MEMORY_WB) {
> +               if ((md->type == EFI_CONVENTIONAL_MEMORY) && (md->attribute
> & EFI_MEMORY_WB)) {
>                         if (membase > md->phys_addr)
>                                 membase = md->phys_addr;
>                 }

I have pushed this patch to OBS for building kernel RPMs now:

https://build.opensuse.org/package/show/home:joeyli:branches:openSUSE:Factory:bsc1122614/kernel-default

I didn't environment to test it yet.
Comment 29 Guillaume GARDET 2019-07-10 17:14:40 UTC
I tried the patched kernel with grub 2.04 on Raspberry Pi2 B+ and I get the following message:
  EFI stub: Booting Linux Kernel...
  EFI stub: ERROR: reserve_kernel_base(): allocate boot data pass.
  EFI stub: ERROR: reserve_kernel_base(): allocate boot data success.
  EFI stub: Using DTB from configuration table
  EFI stub: Exiting boot services and installing virtual address map...

and the system hangs.
Comment 30 Joey Lee 2019-07-11 05:38:25 UTC
Created attachment 810063 [details]
0001-arm-efi-use-conventional-memory-region-for-kernel.patch

The debug patch that is applied on comment#28.
Comment 31 Joey Lee 2019-07-11 06:23:14 UTC
(In reply to Guillaume GARDET from comment #29)
> I tried the patched kernel with grub 2.04 on Raspberry Pi2 B+ and I get the
> following message:
>   EFI stub: Booting Linux Kernel...
>   EFI stub: ERROR: reserve_kernel_base(): allocate boot data pass.
>   EFI stub: ERROR: reserve_kernel_base(): allocate boot data success.
>   EFI stub: Using DTB from configuration table
>   EFI stub: Exiting boot services and installing virtual address map...
> 
> and the system hangs.

Thanks for your testing!

The allocation of memory space for kernel is already pass. Then system hands when kernel tries to set FDT, call ExitBootServices then set virtual address map.

We must do further debugging.
Comment 32 Michael Chang 2019-07-11 06:32:39 UTC
(In reply to Andreas Färber from comment #27)
> (In reply to Michael Chang from comment #17)
> > Or do you mean to drop the arm-uboot package ?
> 
> Yes.
> 
> 1) It hardcoded the physical RAM location and would need per-SoC builds.
> 2) This feature is not enabled in U-Boot by default, unusable with our
> packages.

Thanks for providing the information. Let's drop arm-uboot package if it is no longer supported and also reduce some loads on build service. :)

Will you open a bug for dropping the grub2-arm-uboot package ? TIA.
Comment 33 Guillaume GARDET 2019-07-11 07:47:32 UTC
On Chromebook ARM, your patch fixes the boot.

On RPi2 it is broken as already reported at https://bugzilla.opensuse.org/show_bug.cgi?id=1122614#c29
I suspect this line of code:
  dram_base = round_up(dram_base, SZ_128M);
from https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/firmware/efi/libstub/arm32-stub.c?h=v5.2#n207

which may cause troubles with the first 4K reserved memory.
Comment 34 Guillaume GARDET 2019-07-11 07:58:28 UTC
(In reply to Guillaume GARDET from comment #33)
> On Chromebook ARM, your patch fixes the boot.

Retrying kernel 5.1.x without your patch on Chromebook ARM and it just works. The only difference is u-boot version (v2018.11 vs v2019.04).
Comment 35 Joey Lee 2019-07-11 09:03:35 UTC
(In reply to Guillaume GARDET from comment #33)
> On Chromebook ARM, your patch fixes the boot.
> 
> On RPi2 it is broken as already reported at
> https://bugzilla.opensuse.org/show_bug.cgi?id=1122614#c29
> I suspect this line of code:
>   dram_base = round_up(dram_base, SZ_128M);
> from
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/
> drivers/firmware/efi/libstub/arm32-stub.c?h=v5.2#n207
> 
> which may cause troubles with the first 4K reserved memory.

If dram_base falls in the first reserved region, then the allocation in reserve_kernel_base() will be failed as the original symptom. With patched kernel, the allocation is pass. The new broken is in allocate_new_fdt_and_exit_boot(), Chester Lin is looking at it.

Actually, it a bit mess in the allocation logic for kernel on ARM32. For example:
The "dram_base + MAX_UNCOMP_KERNEL_SIZE" space be allocated in reserve_kernel_base() as EFI_BOOT_SERVICES_DATA type. But in later efi_relocate_kernel(), it tries to allocate the same space as EFI_LOADER_DATA again. Which means that the efi_low_alloc() will always be performed. Then the first allocation reserve_kernel_base() and efi_relocate_kernel() are useless.

I don't think that many people use the EFI stub code path on ARM32.
Comment 36 Michael Chang 2019-07-19 07:47:34 UTC
Hi Guillaume,

Could you please check if you seen the message "Please press 't' to show the boot menu on this console" and pressing the "t" could bring the menu or not ?

If not working, could you please try modifying /etc/default/grub with:

 GRUB_TERMINAL="console"

And run "grub2-mkconfig -o /boot/grub2/grub.cfg" to direct using the firmware console at boot, thus will eliminate the hotkey 't' not being functional. And if still not work, problem might be within the console terminal itself.

I didn't reproduce this problem yet.
Comment 37 Guillaume GARDET 2019-07-19 10:00:52 UTC
(In reply to Michael Chang from comment #36)
> Hi Guillaume,
> 
> Could you please check if you seen the message "Please press 't' to show the
> boot menu on this console" and pressing the "t" could bring the menu or not ?

No, the last message on serial is: 
  Welcome to GRUB!

Whereas with previous grub2.02, I also have the following lines after the welcome message:
  Waiting for Ethernet connection... done.
  Please press 't' to show the boot menu on this console

> If not working, could you please try modifying /etc/default/grub with:
> 
>  GRUB_TERMINAL="console"
> 
> And run "grub2-mkconfig -o /boot/grub2/grub.cfg" to direct using the
> firmware console at boot, thus will eliminate the hotkey 't' not being
> functional. And if still not work, problem might be within the console
> terminal itself.

GRUB_TERMINAL="console" option allows to have the menu on serial console.
Comment 38 Michael Chang 2019-07-19 10:20:34 UTC
(In reply to Guillaume GARDET from comment #37)
> (In reply to Michael Chang from comment #36)
> > Hi Guillaume,
> > 
> > Could you please check if you seen the message "Please press 't' to show the
> > boot menu on this console" and pressing the "t" could bring the menu or not ?
> 
> No, the last message on serial is: 
>   Welcome to GRUB!
> 
> Whereas with previous grub2.02, I also have the following lines after the
> welcome message:
>   Waiting for Ethernet connection... done.
>   Please press 't' to show the boot menu on this console

Is /sys/firmware/efi present on your the system ? It is required to generate the relevant message and hiddenentry in grub.cfg ..

[1] https://build.opensuse.org/package/view_file/openSUSE:Factory/grub2/grub2-SUSE-Add-the-t-hotkey.patch?expand=1

Other possibilities could be the timeout setting, would you please attach your grub.cfg here for reference ? 

> 
> > If not working, could you please try modifying /etc/default/grub with:
> > 
> >  GRUB_TERMINAL="console"
> > 
> > And run "grub2-mkconfig -o /boot/grub2/grub.cfg" to direct using the
> > firmware console at boot, thus will eliminate the hotkey 't' not being
> > functional. And if still not work, problem might be within the console
> > terminal itself.
> 
> GRUB_TERMINAL="console" option allows to have the menu on serial console.

OK, at least good news for the console driver in 2.04 works.
Comment 39 Guillaume GARDET 2019-07-19 12:37:42 UTC
(In reply to Michael Chang from comment #38)
> (In reply to Guillaume GARDET from comment #37)
> > (In reply to Michael Chang from comment #36)
> > > Hi Guillaume,
> > > 
> > > Could you please check if you seen the message "Please press 't' to show the
> > > boot menu on this console" and pressing the "t" could bring the menu or not ?
> > 
> > No, the last message on serial is: 
> >   Welcome to GRUB!
> > 
> > Whereas with previous grub2.02, I also have the following lines after the
> > welcome message:
> >   Waiting for Ethernet connection... done.
> >   Please press 't' to show the boot menu on this console
> 
> Is /sys/firmware/efi present on your the system ? It is required to generate
> the relevant message and hiddenentry in grub.cfg ..
> 
> [1]
> https://build.opensuse.org/package/view_file/openSUSE:Factory/grub2/grub2-
> SUSE-Add-the-t-hotkey.patch?expand=1

There is no /sys/firmware/efi/ as, IIUC, it would require grub2.04 to be installed and when I install it, it still has grub2.02. And grub2.04 fails to boot properly on RPi2 atm, as reported in #c5.
If I remove the EFI test condition, it is working fine. \o/

So, it leaves the broken boot of kernel, even with patched kernel for #c28

@jlee, did you or Chester made some progress and/or have something to test on RPi2?
Comment 40 Chester Lin 2019-07-22 03:31:01 UTC
(In reply to Guillaume GARDET from comment #39)
> There is no /sys/firmware/efi/ as, IIUC, it would require grub2.04 to be
> installed and when I install it, it still has grub2.02. And grub2.04 fails
> to boot properly on RPi2 atm, as reported in #c5.
> If I remove the EFI test condition, it is working fine. \o/
> 
> So, it leaves the broken boot of kernel, even with patched kernel for #c28
> 
> @jlee, did you or Chester made some progress and/or have something to test
> on RPi2?


Hi Guillaume,

Yes, I do have some findings of the broken boot as follows:

1. Confirmed that the U-Boot's exit_boot_services still works [ver: uboot-2019-04]. Anyway, I found that this issue occurred in the early stage of kernel booting.

2. The first page [0~0xfff] has been reserved by rpi2 firmware thus EFI memmap always marks it as EFI_RESERVED_TYPE. Besides, this line could cause an unexpected crash too:

arm32-stub.c:
  dram_base = round_up(dram_base, SZ_128M);

When applying the patch of #c28, the raw value of dram_base is changed to 0x1000, which is the first available memblock marked as EFI_CONVENTIONAL_MEMORY.
Then the dram_base rounds up to 0x8000000, so the region from 0x0 to 0x7ffffff is totally ignored by early_init_dt_add_memory_arch() and has not been added
into the memblock list.

---
[    0.000000] OF: fdt: Ignoring memory block 0x0 - 0x1000                                                                                                                  
[    0.000000] OF: fdt: Ignoring memory block 0x1000 - 0x7cf6000                                                                                                            
[    0.000000] OF: fdt: Ignoring memory block 0x7cf6000 - 0x7ef6000                                                                                                         
[    0.000000] OF: fdt: Ignoring memory block 0x7ef6000 - 0x7f0a000                                                                                                         
[    0.000000] OF: fdt: Ignoring memory range 0x7f0a000 - 0x8000000                                                                                                         
[    0.000000] memblock_add: [0x0000000008000000-0x0000000007ffffff] early_init_dt_add_memory_arch+0x174/0x17c                                                              
[    0.000000] memblock_add: [0x0000000008000000-0x0000000009ffffff] early_init_dt_add_memory_arch+0x174/0x17c                                                              
[    0.000000] memblock_add: [0x000000000a000000-0x000000000a78ffff] early_init_dt_add_memory_arch+0x174/0x17c                                                              
[    0.000000] memblock_add: [0x000000000a790000-0x000000001f24ffff] early_init_dt_add_memory_arch+0x174/0x17c
---

However the fdt is still allocated in [0x7cf6000 - 0x7ef6000] by EFI stub, which could cause an unexpected error as below, and I think that's because the first 128MB [0 - 0x8000000] has been ignored so there's no page entries for translating the virtual addresses of FDT.

---
[    0.000000] Unable to handle kernel paging request at virtual address bfcf6000                                                                                           
[    0.000000] pgd = (ptrval)                                                                                                                                               
[    0.000000] [bfcf6000] *pgd=80000008306003, *pmd=00000000                                                                                                                
[    0.000000] Internal error: Oops: 206 [#1] SMP ARM                                                                                                                       
[    0.000000] Modules linked in:                                                                                                                                           
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.2.0-10.g80522d2-lpae #1 openSUSE Tumbleweed (unreleased)                                                           
[    0.000000] Hardware name: BCM2835                                                                                                                                       
[    0.000000] PC is at fdt_check_header+0xc/0x13c                                                                                                                          
[    0.000000] LR is at __unflatten_device_tree+0x50/0x27c                                                                                                                  
[    0.000000] pc : [<c0e38f30>]    lr : [<c0c6b750>]    psr: a00000d3                                                                                                      
[    0.000000] sp : c1801ec0  ip : c1801ed0  fp : c1801ecc                                                                                                                  
[    0.000000] r10: c1805d80  r9 : 00000000  r8 : c1ded98c
[    0.000000] r7 : 00000000  r6 : bfcf6000  r5 : c167e2a4  r4 : c167e2a4
[    0.000000] r3 : c167e2a4  r2 : c1ded98c  r1 : 00000000  r0 : bfcf6000
[    0.000000] Flags: NzCv  IRQs off  FIQs off  Mode SVC_32  ISA ARM  Segment user
[    0.000000] Control: 30c5387d  Table: 08303000  DAC: fffffffd
..........
..........
[    0.000000] Backtrace: 
[    0.000000] [<c0e38f24>] (fdt_check_header) from [<c0c6b750>] (__unflatten_device_tree+0x50/0x27c)
[    0.000000] [<c0c6b700>] (__unflatten_device_tree) from [<c167f53c>] (unflatten_device_tree+0x44/0x54)
[    0.000000]  r10:c1805d80 r9:410fc075 r8:07cf6000 r7:c1801fc0 r6:c19299c0 r5:c16a2a40
[    0.000000]  r4:c167e2a4
[    0.000000] [<c167f4f8>] (unflatten_device_tree) from [<c16056cc>] (setup_arch+0x190/0x658)
[    0.000000]  r4:c16915e8
[    0.000000] [<c160553c>] (setup_arch) from [<c1600c68>] (start_kernel+0x78/0x520)
[    0.000000]  r10:c1805d80 r9:410fc075 r8:07cf6000 r7:c1929700 r6:ffffffff r5:00000000
[    0.000000]  r4:00000000
[    0.000000] [<c1600bf0>] (start_kernel) from [<00000000>] (0x0)
[    0.000000]  r10:30c5387d r9:410fc075 r8:07cf6000 r7:ffffffff r6:30c0387d r5:00000000
[    0.000000]  r4:c1600330
[    0.000000] Code: e89da800 e1a0c00d e92dd800 e24cb004 (e5902000) 
[    0.000000] random: get_random_bytes called from print_oops_end_marker+0x34/0x5c with crng_init=0
[    0.000000] ---[ end trace 0000000000000000 ]---
[    0.000000] Kernel panic - not syncing: Attempted to kill the idle task!
[    0.000000] Rebooting in 90 seconds..
[    0.000000] Reboot failed -- System halted
Comment 41 Chester Lin 2019-07-22 04:03:10 UTC
Created attachment 811106 [details]
A patch to solve FDT paging issue.

I have made a patch to try appending the FDT start address behind the uncompressed kernel base so that the page table will have correct mappings. I'm creating a test package on OBS and will give you a download link once it's completed.
Comment 42 Chester Lin 2019-07-22 04:22:53 UTC
Besides, I wonder if we might have any chance to delete the first page reserved by rpi2 firmware? Instead of rebasing the FDT offset I have mentioned in my patch, I found that deleting this reserved page can also solve this issue although I have not sure if there's any side-effect.
Comment 43 Chester Lin 2019-07-22 04:36:51 UTC
(In reply to Chester Lin from comment #42)
> Besides, I wonder if we might have any chance to delete the first page
> reserved by rpi2 firmware? Instead of rebasing the FDT offset I have
> mentioned in my patch, I found that deleting this reserved page can also
> solve this issue although I have not sure if there's any side-effect.

I am thinking about this because the first 128MB will be totally ignored and will not be used once the kernel base rounds up to 0x8000000.
Comment 44 Guillaume GARDET 2019-07-22 06:59:48 UTC
(In reply to Chester Lin from comment #43)
> (In reply to Chester Lin from comment #42)
> > Besides, I wonder if we might have any chance to delete the first page
> > reserved by rpi2 firmware? Instead of rebasing the FDT offset I have
> > mentioned in my patch, I found that deleting this reserved page can also
> > solve this issue although I have not sure if there's any side-effect.
> 
> I am thinking about this because the first 128MB will be totally ignored and
> will not be used once the kernel base rounds up to 0x8000000.

I asked upstream to move this reserved 4KiB of RAM upper in memory: https://github.com/raspberrypi/firmware/issues/1199
Let's see how it goes.
Comment 45 Guillaume GARDET 2019-07-22 08:19:23 UTC
(In reply to Guillaume GARDET from comment #44)
> (In reply to Chester Lin from comment #43)
> > (In reply to Chester Lin from comment #42)
> > > Besides, I wonder if we might have any chance to delete the first page
> > > reserved by rpi2 firmware? Instead of rebasing the FDT offset I have
> > > mentioned in my patch, I found that deleting this reserved page can also
> > > solve this issue although I have not sure if there's any side-effect.
> > 
> > I am thinking about this because the first 128MB will be totally ignored and
> > will not be used once the kernel base rounds up to 0x8000000.
> 
> I asked upstream to move this reserved 4KiB of RAM upper in memory:
> https://github.com/raspberrypi/firmware/issues/1199
> Let's see how it goes.

As expected, we should not expect much from upstream: https://github.com/raspberrypi/firmware/issues/1199#issuecomment-513687604

And we should find a proper solution in kernel.
Comment 46 Chester Lin 2019-07-22 09:34:01 UTC
(In reply to Chester Lin from comment #41)
> Created attachment 811106 [details]
> A patch to solve FDT paging issue.
> 
> I have made a patch to try appending the FDT start address behind the
> uncompressed kernel base so that the page table will have correct mappings.
> I'm creating a test package on OBS and will give you a download link once
> it's completed.

Hi Guillaume,

The test kernel package based on two patches [conventional_memory check and FDT paging] is ready:

https://build.opensuse.org/package/binaries/home:clin:branches:openSUSE:ARM:bsc1122614/kernel-lpae/ARM

I just built lpae-flavour since it was quite slow to build all flavours. Please don't hesitate to let me know if you need others.
Comment 47 Chester Lin 2019-07-22 09:36:40 UTC
(In reply to Guillaume GARDET from comment #45)
> (In reply to Guillaume GARDET from comment #44)
> > 
> > I asked upstream to move this reserved 4KiB of RAM upper in memory:
> > https://github.com/raspberrypi/firmware/issues/1199
> > Let's see how it goes.
> 
> As expected, we should not expect much from upstream:
> https://github.com/raspberrypi/firmware/issues/1199#issuecomment-513687604
> 
> And we should find a proper solution in kernel.

In fact, I also did another experiment, which loaded the uncompressed kernel at 0x1000 without calling round_up(dram_base, SZ_128M), and the kernel could still boot up although it needs more code changes in arch/arm/mm/mmu.c. In this experiment I changed adjust_lowmem_bounds() a little bit so that the memblock_limit would not stick with 0x1000 due to PMD_SIZE alignment check otherwise the kernel could fail to reserve CMA region.

Is there any reason [e.g, cache performance] that the kernel base must align with 128MB? Is it possibie to round up with less alignment size such as 4K(PAGE_SIZE),1M or 2M(PMD_SIZE)?
Comment 48 Michael Chang 2019-08-01 07:29:08 UTC
Is there any ETA for the efistub kernel patch planned? I would like to know because grub 2.04 release is currently blocked. If not could we agree on releasing 2.04 with good old zImage interface for armv7 "temporarily" until kernel's efistub patch arrives ?
Thanks.
Comment 49 Chester Lin 2019-08-01 08:10:20 UTC
(In reply to Michael Chang from comment #48)
> Is there any ETA for the efistub kernel patch planned? I would like to know
> because grub 2.04 release is currently blocked. If not could we agree on
> releasing 2.04 with good old zImage interface for armv7 "temporarily" until
> kernel's efistub patch arrives ?
> Thanks.

Actually I have another idea to avoid dram_base change [It's still 0] and but it could sacrifice the first 2MB in rpi2. All code changes are under arm32 so arm64 will not be affected. I have tested it on my rpi2 but need to know if it also works on other platforms such as Chromebook or Arndale before I submit to the upstream for code review.

Hi Guillaume,

Could you please help me to verify the test kernel? Here is the repo link:

https://download.opensuse.org/repositories/home:/clin:/branches:/openSUSE:/ARM:/bsc1122614v2/ARM/

Please install "kernel-lpae-5.2.1-2.1.gbf5c01b" [e.g, rpi2] or "kernel-default-5.2.1-2.1.gbf5c01b" based on platforms' original flavours. Besides, please compare the difference of the "MemTotal" field in /proc/meminfo. The differene should be about 2MB 

If any failure occurs, please add "memblock=debug efi=debug" in the kernel command line so that I can have more memmap information.
Comment 50 Chester Lin 2019-08-01 08:19:57 UTC
Created attachment 812405 [details]
0001-efi-arm-fix-allocation-failure-when-reserving-the-ke.patch

The patch I described in comment#49 is as attached.
Comment 51 Guillaume GARDET 2019-08-01 11:49:42 UTC
(In reply to Chester Lin from comment #49)
> Could you please help me to verify the test kernel? Here is the repo link:
> 
> https://download.opensuse.org/repositories/home:/clin:/branches:/openSUSE:/
> ARM:/bsc1122614v2/ARM/
> 

I tested the following 32-bit arm systems:
* Chromebook (snow): kernel-lpae: OK
* Raspberry Pi 2: kernel-lpae:    OK (MemTotal: 975808 kB vs 977984 kB)
* Raspberry Pi 2: kernel-default: OK (MemTotal: 979532 kB vs 981708 kB)
* Raspberry Pi 1 (armv6):         KO (data abort)

Except the armv6 problem, the only remaining problem I see ATM, is that some systems (e.g. Chromebook) do not update u-boot as it updates only binaries on the /boot partition but it uses the version copied at an offset of the SD card.
We should at least warn people to update at least to u-boot 2019.04 and manually copy the new binaries to the required offset. Or should we handle that in some %post script in u-boot?
Comment 52 Guillaume GARDET 2019-08-01 11:50:26 UTC
Created attachment 812450 [details]
armv6 data abort after kernel and initrd are loaded.
Comment 53 Guillaume GARDET 2019-08-01 13:07:35 UTC
(In reply to Guillaume GARDET from comment #52)
> Created attachment 812450 [details]
> armv6 data abort after kernel and initrd are loaded.

Forgot to mention that the unpatched kernel has the same 'data abort' problem than the patched kernel. So, the problem is not from the patch, but it may be from grub 2.04 itself.
Comment 54 Chester Lin 2019-08-02 06:01:23 UTC
I also tried this patch + grub-2.04 + openSUSE-Tumbleweed-ARM-JeOS-efi.armv7l-2019.07.25 on QEMU 4.0.0 without seeing an issue.

My QEMU arguments: 
  qemu-system-arm -m 768 -machine virt -cpu cortex-a15 -bios QEMU_EFI.fd ...

The QEMU_EFI.fd is from:

http://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/latest/QEMU-ARM/DEBUG_GCC5/

I have submitted it to the upstream for code review, let's see if there's any comment.

https://patchwork.kernel.org/patch/11072713/
Comment 55 Joey Lee 2019-08-02 06:58:00 UTC
Hi Guillaume, 

(In reply to Guillaume GARDET from comment #53)
> (In reply to Guillaume GARDET from comment #52)
> > Created attachment 812450 [details]
> > armv6 data abort after kernel and initrd are loaded.
> 
> Forgot to mention that the unpatched kernel has the same 'data abort'
> problem than the patched kernel. So, the problem is not from the patch, but
> it may be from grub 2.04 itself.

Looks that this is another issue. Could you please help to file another bug against Raspberry Pi 1 problem?
Comment 56 Chester Lin 2019-08-05 02:16:35 UTC
(In reply to Chester Lin from comment #54)
> I also tried this patch + grub-2.04 +
> openSUSE-Tumbleweed-ARM-JeOS-efi.armv7l-2019.07.25 on QEMU 4.0.0 without
> seeing an issue.
> 
> My QEMU arguments: 
>   qemu-system-arm -m 768 -machine virt -cpu cortex-a15 -bios QEMU_EFI.fd ...
> 
> The QEMU_EFI.fd is from:
> 
> http://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-
> upstream/latest/QEMU-ARM/DEBUG_GCC5/
> 
> I have submitted it to the upstream for code review, let's see if there's
> any comment.
> 
> https://patchwork.kernel.org/patch/11072713/

The maintainer Ard suggests that this issue should be fixed in bootloader or grub.

https://patchwork.kernel.org/patch/11072713/#22797553
Comment 57 Michael Chang 2019-08-06 10:10:09 UTC
(In reply to Guillaume GARDET from comment #53)
> (In reply to Guillaume GARDET from comment #52)
> > Created attachment 812450 [details]
> > armv6 data abort after kernel and initrd are loaded.
> 
> Forgot to mention that the unpatched kernel has the same 'data abort'
> problem than the patched kernel. So, the problem is not from the patch, but
> it may be from grub 2.04 itself.

I think we'd better clarify that the program counter contains the address in the secondary UEFI image after grub and that it should be the linux kernel. 

> UEFI image [0x1cb22000:0x1cb3dfff] '/VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/SD(0)/SD(0)/HD(1,MBR,0x823858ef,0x800,0x8000)/efi\boot\bootarm.efi'
> UEFI image [0x13972000:0x13f58fff] pc=0x7fb0
> Resetting CPU ...
Comment 58 Michael Chang 2019-08-06 10:31:32 UTC
(In reply to Chester Lin from comment #56)
> (In reply to Chester Lin from comment #54)

> The maintainer Ard suggests that this issue should be fixed in bootloader or
> grub.
> 
> https://patchwork.kernel.org/patch/11072713/#22797553

I had some conversation with Chester and it looks to be u-boot currently making the reservation below 128MB, the grub doesn't have such reservation to be corrected.
Comment 59 Andreas Färber 2019-08-07 12:02:13 UTC
(In reply to Michael Chang from comment #58)
> (In reply to Chester Lin from comment #56)
> > (In reply to Chester Lin from comment #54)
> 
> > The maintainer Ard suggests that this issue should be fixed in bootloader or
> > grub.
> > 
> > https://patchwork.kernel.org/patch/11072713/#22797553
> 
> I had some conversation with Chester and it looks to be u-boot currently
> making the reservation below 128MB, the grub doesn't have such reservation
> to be corrected.

Yes, we know U-Boot is making the reservation. But it makes it for a reason.

For RPi3 and RPi4 we are evaluating TF-A, which would avoid the spin-table in favor of a PSCI hypercall (U-Boot could then be extended to check the DT for whether or not to make the reservation), but for RPi2 and earlier I am not aware of anyone investigating TF-A and I'm not familiar of whether in absence of EL3 in ARMv7 there might be some other solution there.
Comment 60 Guillaume GARDET 2019-08-14 14:35:10 UTC
(In reply to Andreas Färber from comment #59)
> (In reply to Michael Chang from comment #58)
> > (In reply to Chester Lin from comment #56)
> > > (In reply to Chester Lin from comment #54)
> > 
> > > The maintainer Ard suggests that this issue should be fixed in bootloader or
> > > grub.
> > > 
> > > https://patchwork.kernel.org/patch/11072713/#22797553
> > 
> > I had some conversation with Chester and it looks to be u-boot currently
> > making the reservation below 128MB, the grub doesn't have such reservation
> > to be corrected.
> 
> Yes, we know U-Boot is making the reservation. But it makes it for a reason.

I think this is the RPi firmware:
https://github.com/raspberrypi/firmware/issues/1199
Comment 61 Guillaume GARDET 2019-08-14 14:40:05 UTC
(In reply to Joey Lee from comment #55)
> Hi Guillaume, 
> 
> (In reply to Guillaume GARDET from comment #53)
> > (In reply to Guillaume GARDET from comment #52)
> > > Created attachment 812450 [details]
> > > armv6 data abort after kernel and initrd are loaded.
> > 
> > Forgot to mention that the unpatched kernel has the same 'data abort'
> > problem than the patched kernel. So, the problem is not from the patch, but
> > it may be from grub 2.04 itself.
> 
> Looks that this is another issue. Could you please help to file another bug
> against Raspberry Pi 1 problem?

Done here: https://bugzilla.suse.com/show_bug.cgi?id=1145646
Comment 62 Chester Lin 2019-08-19 08:31:44 UTC
Ard and Mike have provided two patches to solve arm32-stub and memblock_limit issues respectively: https://patchwork.kernel.org/patch/11072713/

Hi Guillaume,

I have built a test kernel based on these two patches, could you please help to verify if these patches work on your machines? I have tried on my rpi2 model B and it works well.

https://download.opensuse.org/repositories/home:/clin:/branches:/openSUSE:/ARM:/arm32stub-Ard-patch/ARM/
Comment 63 Guillaume GARDET 2019-08-29 15:45:41 UTC
@chester, I tested the following 32-bit arm systems with the latest patchset from https://download.opensuse.org/repositories/home:/clin:/branches:/openSUSE:/ARM:/arm32stub-Ard-patch/ARM/ with grub 2.04:
* Chromebook (snow): kernel-lpae:    OK
* Chromebook (snow): kernel-default: OK
* Raspberry Pi 2: kernel-lpae:       OK
* Raspberry Pi 2: kernel-default:    KO, hangs after "EFI stub: Exiting boot services and installing virtual address map..."
* Raspberry Pi 1 (armv6):            KO (data abort, see bug#1145646)

Any idea why it would fail with kernel-default on RPi2? Maybe an unrelated bug?
Comment 64 Chester Lin 2019-08-30 06:42:52 UTC
(In reply to Guillaume GARDET from comment #63)
> @chester, I tested the following 32-bit arm systems with the latest patchset
> from
> https://download.opensuse.org/repositories/home:/clin:/branches:/openSUSE:/
> ARM:/arm32stub-Ard-patch/ARM/ with grub 2.04:
> * Chromebook (snow): kernel-lpae:    OK
> * Chromebook (snow): kernel-default: OK
> * Raspberry Pi 2: kernel-lpae:       OK
> * Raspberry Pi 2: kernel-default:    KO, hangs after "EFI stub: Exiting boot
> services and installing virtual address map..."
> * Raspberry Pi 1 (armv6):            KO (data abort, see bug#1145646)
> 
> Any idea why it would fail with kernel-default on RPi2? Maybe an unrelated
> bug?

Hi Guillaume,

Please try this repo:

https://build.opensuse.org/project/show/home:clin:branches:openSUSE:ARM:armv7-efistub-fix

It contains the latest patch provided by Ard [arm32stub], Mike and Me [arm-mmu, it's still waiting for Russell King's review].

I have tried 5.2.9-1.g71d4424-default and it works fine on my rpi2. But I noticed that it might take a longer time [about 15 seconds] to boot kernel-default compared to kernel-lpae.
Comment 65 Guillaume GARDET 2019-08-30 09:22:49 UTC
(In reply to Chester Lin from comment #64)
> Please try this repo:
> 
> https://build.opensuse.org/project/show/home:clin:branches:openSUSE:ARM:
> armv7-efistub-fix

Here are my results:
* Chromebook (snow): kernel-lpae:    OK
* Chromebook (snow): kernel-default: OK
* Raspberry Pi 2: kernel-lpae:       OK
* Raspberry Pi 2: kernel-default:    OK
* Raspberry Pi 1 (armv6):            KO (data abort, see bug#1145646)

> I have tried 5.2.9-1.g71d4424-default and it works fine on my rpi2. But I
> noticed that it might take a longer time [about 15 seconds] to boot
> kernel-default compared to kernel-lpae.

I noticed no time difference between:
  "EFI stub: Exiting boot services and installing virtual address map..."
and 
  "Welcome to openSUSE Tumbleweed dracut-044-3.1 (Initramfs)!"
messages.
Comment 66 Guillaume GARDET 2019-10-14 07:08:45 UTC
Are all the patches upstream by now?
Comment 67 Chester Lin 2019-10-14 07:34:01 UTC
(In reply to Guillaume GARDET from comment #66)
> Are all the patches upstream by now?

The following two have been merged into torvalds/linux.git:
-----
commit 00d2ec1e6bd8 "ARM: 8903/1: ensure that usable memory in bank 0 starts from a PMD-aligned address"

commit 1d31999cf04c "ARM: 8904/1: skip nomap memblocks while finding the lowmem/highmem boundary"

-----
However, Ard still holds his patch in his tree and I am not sure when it will be merged.

https://git.kernel.org/pub/scm/linux/kernel/git/efi/efi.git/commit/drivers/firmware/efi/libstub/arm32-stub.c?h=next&id=0eb7bad595e52666b642a02862ad996a0f9bfcc0
Comment 68 Guillaume GARDET 2019-10-15 06:17:58 UTC
I pinged Ard about the last remaining patch. 
He sent it for upstream review: http://lists.infradead.org/pipermail/linux-arm-kernel/2019-October/686533.html

It should be in kernel 5.4.
Comment 69 Michael Chang 2019-10-15 06:45:47 UTC
(In reply to Guillaume GARDET from comment #68)
> I pinged Ard about the last remaining patch. 
> He sent it for upstream review:
> http://lists.infradead.org/pipermail/linux-arm-kernel/2019-October/686533.
> html
> 
> It should be in kernel 5.4.

Thank you. And please let me know when it has been done to release grub 2.04 for tumbleweed.
Comment 70 Guillaume GARDET 2019-10-30 07:34:07 UTC
EFI git pull v2 for 5.4: https://www.spinics.net/lists/linux-efi/msg16935.html
Comment 71 Guillaume GARDET 2019-11-12 15:59:33 UTC
Merged in v5.4-rc6
Comment 72 Guillaume GARDET 2019-12-17 14:23:58 UTC
Kernel 5.4 has been merged in Factory today: https://build.opensuse.org/request/show/757388