Bug 1186784

Summary: VirtualBox modules not allowed to load in Leap 15.3, not properly signed?
Product: [openSUSE] openSUSE Distribution Reporter: S. B. <sb56637>
Component: Virtualization:OtherAssignee: Larry Finger <Larry.Finger>
Status: RESOLVED INVALID QA Contact: Larry Finger <Larry.Finger>
Severity: Normal    
Priority: P5 - None CC: flsq6xoyf2, Larry.Finger, novell, sb56637
Version: Leap 15.3   
Target Milestone: ---   
Hardware: x86-64   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description S. B. 2021-06-03 03:02:08 UTC
Hi, I build custom Leap 15.3 live ISOs with Kiwi for installation to disk with the Calamares installer. For some reason the live ISOs are able to load the vboxguest module automatically, but after installation to disk the system fails to load with the error:

> modprobe: ERROR: module 'vboxguest' is unsupported
> modprobe: ERROR: Use --allow-unsupported or set allow_unsupported_modules 1 in
> modprobe: ERROR: /etc/modprobe.d/10-unsupported-modules.conf
> modprobe: ERROR: could not insert 'vboxguest': Operation not permitted

So I did as it suggested, and that makes it work. But the question is why the official VirtualBox modules from the Leap 15.3 OSS repo aren't supported. From this reference...
https://doc.opensuse.org/release-notes/x86_64/openSUSE/Leap/15.3/#driver-hardware-secure-boot-sign-driver
...it looks like it should be supported:
> Kernel Module Packages (KMPs) from the official openSUSE repositories are not affected, because the modules they contain are signed with the openSUSE key.
So I wonder if maybe the VirtualBox drivers aren't being properly signed?

Thanks a lot for the help.
Comment 1 Larry Finger 2021-06-03 19:49:10 UTC
The VirtualBox modules installed via zypper or PackageKit from the official repos should be signed. How were your modules obtained?
Comment 2 S. B. 2021-06-03 20:19:19 UTC
Hi Larry, thanks for the reply. Yes, they are installed from the official Leap 15.3 repos, that's why I wondered if something was wrong with their signature.
Comment 3 Larry Finger 2021-06-03 21:48:45 UTC
No, they should have been OK.
Comment 4 S. B. 2021-06-03 22:08:59 UTC
Does it have something to do with the fact that the vendor for `kernel-default` is "SUSE LLC" whereas the vendor for the `virtualbox-kmp-default` package is "openSUSE"? Just throwing ideas out there.
Comment 5 Larry Finger 2021-06-04 01:01:36 UTC
That is possible. We had a similar case where a vagrant container had a problem loading vboxsf.

Could you please list the set of steps necessary to reproduce your problem?
Comment 6 S. B. 2021-06-04 16:13:53 UTC
Well, I honestly can't figure out what's going on. I just installed Leap 15.3 from the official netinstall ISO on a VirtualBox machine with EFI and 3D acceleration enabled, and it automatically installed and successfully loaded `vboxguest`. 

Whereas with my ISOs that were built with Kiwi (https://sourceforge.net/projects/geckolinux/files/Static/153.210603/) based on the openSUSE Kiwi templates just with a different package selection and the addition of some Packman packages, the live system also loads `vboxguest` automatically, but the installed system throws the unsupported error that I mentioned earlier.
Comment 7 Link Porterfield 2021-06-09 19:47:55 UTC
My OpenSUSE installation started exhibiting similar behavior following the upgrade from 15.2 to 15.3.

I now see this in dmesg: Lockdown: modprobe: unsigned module loading is restricted; see man kernel_lockdown.7

Signed modules were required by my OS before I upgraded and virtualbox worked.

Information for package virtualbox:
-----------------------------------
Repository     : Main Repository
Name           : virtualbox
Version        : 6.1.20-lp153.1.8
Arch           : x86_64
Vendor         : openSUSE

Information for package virtualbox-kmp-default:
-----------------------------------------------
Repository     : Main Repository
Name           : virtualbox-kmp-default
Version        : 6.1.20_k5.3.18_57-lp153.1.2
Arch           : x86_64
Vendor         : openSUSE

Information for package virtualbox-qt:
--------------------------------------
Repository     : Main Repository
Name           : virtualbox-qt
Version        : 6.1.20-lp153.1.8
Arch           : x86_64
Vendor         : openSUSE
Comment 8 Larry Finger 2021-06-09 20:09:37 UTC
(In reply to Link Porterfield from comment #7)
> My OpenSUSE installation started exhibiting similar behavior following the
> upgrade from 15.2 to 15.3.
> 
> I now see this in dmesg: Lockdown: modprobe: unsigned module loading is
> restricted; see man kernel_lockdown.7

Your problem is somewhat different from that of S. B.

It appears that you got your kmp package from openSUSE. If so, I do not understand your problem as modules from that source are signed.

Please use YaST to "update unconditionally" the virtualbox-kmp-default package to see if that helps.
Comment 9 Link Porterfield 2021-06-10 15:19:25 UTC
Using YaST to "update unconditionally" made no change. Since you indicated this is something else (although if it is indeed an incorrect or missing signature, it seems like it might be the same issue), should I open a new bug?
Comment 10 Larry Finger 2021-06-10 16:44:57 UTC
(In reply to Link Porterfield from comment #9)
> Using YaST to "update unconditionally" made no change. Since you indicated
> this is something else (although if it is indeed an incorrect or missing
> signature, it seems like it might be the same issue), should I open a new
> bug?

I'm not sure if you need a separate bug.

I did some research and found how to verify that a module is signed. Using 'modinfo vboxdrv', I get the following on Leap 15.3:

finger@localhost:~> modinfo vboxdrv
filename:       /lib/modules/5.3.18-57-default/extra/vboxdrv.ko
version:        6.1.20_SUSE r143896 (0x00300000)
license:        GPL
description:    Oracle VM VirtualBox Support Driver
author:         Oracle Corporation
suserelease:    SLE15-SP3
srcversion:     4BA14177643D42355F49C17
depends:        
retpoline:      Y
name:           vboxdrv
vermagic:       5.3.18-57-default SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         openSUSE Secure Boot CA
sig_key:        FA:BE:D8:BF:40:9A:5E:64
sig_hashalgo:   sha256
signature:      43:6D:C0:F5:6C:2E:31:1E:6F:35:B4:C1:8C:F1:49:CF:DF:CF:80:90:
                BB:B1:90:40:B5:24:60:84:A0:74:88:DF:72:53:E0:24:AB:DD:36:42:
                35:EC:90:67:B2:68:B3:CC:27:99:9B:9C:D6:A5:C3:2E:B5:82:93:C2:
                D0:DD:67:0B:4E:2A:FF:7D:17:9F:E3:DE:14:4E:75:55:30:10:02:74:
                C9:8F:C9:7C:4C:7F:72:46:58:36:3E:11:5E:A7:D0:53:4A:00:57:93:
                DC:16:51:72:4E:7E:AE:58:45:6A:37:76:04:C3:12:CE:9C:12:FF:B2:
                02:EB:90:81:84:9E:AE:0C:60:82:14:4E:48:DB:CA:60:FF:43:7F:29:
                5C:30:ED:26:87:FC:79:A2:74:3B:01:5F:DF:AD:50:AA:3F:EF:F2:FB:
                DD:E8:1B:58:4F:A0:4E:4C:18:7A:58:03:E4:A4:A9:A8:F4:92:60:3E:
                DF:75:83:E5:29:FE:9C:61:CF:B1:4C:9D:2A:D9:99:24:4C:FC:3F:6E:
                CD:69:37:46:85:21:17:D7:42:E7:17:23:7B:31:54:A0:97:D4:16:ED:
                91:82:2F:E6:52:97:0C:38:65:84:9D:C1:22:CA:ED:AD:1F:9E:99:45:
                64:C9:BD:D6:49:20:B1:54:CC:E8:27:20:23:EE:EC:8A
parm:           force_async_tsc:force the asynchronous TSC mode (int)

That output shows that openSUSE Secure Boot CA is the signer and that the algorithm is sha256.

Does your system show similar signing info?
Comment 11 Link Porterfield 2021-06-10 20:47:15 UTC
I have identical output for my vboxdrv kernel module.
Comment 12 Link Porterfield 2021-06-10 21:42:43 UTC
Compare that with the output from the kvm module (which is loading) and you will see that the signer is 'SUSE Linux Enterprise Secure Boot CA'

modinfo kvm

filename:       /lib/modules/5.3.18-57-default/kernel/arch/x86/kvm/kvm.ko.xz
license:        GPL
author:         Qumranet
suserelease:    SLE15-SP3
srcversion:     3D8FBA9060D4537359A06FC
depends:        irqbypass
supported:      yes
retpoline:      Y
intree:         Y
name:           kvm
vermagic:       5.3.18-57-default SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         SUSE Linux Enterprise Secure Boot CA
sig_key:        ED:87:85:B7:8F:FC:12:7E
sig_hashalgo:   sha256
signature:      47:25:D6:37:5B:E9:35:A2:E5:1F:CA:02:7B:B0:21:9F:C4:4A:F0:3E:
                16:42:22:3E:FC:4C:87:AC:A0:14:77:EB:1A:07:B0:4D:97:E8:80:3E:
                2E:6C:13:CF:9C:4C:0E:D8:AA:0D:11:A5:7B:EB:3C:21:1D:C2:B3:32:
                AC:4A:9E:F4:B3:E8:D1:DF:04:AA:82:04:82:42:A1:DE:FB:E7:65:E9:
                9E:AD:01:B2:0D:33:5E:DF:8A:F6:7A:D0:4B:97:7A:35:09:D5:4C:FA:
                80:C9:9D:B2:87:1E:4E:5F:84:74:B0:69:40:51:E2:E9:27:02:C7:DF:
                55:63:39:E0:D3:EE:FD:E1:AF:84:59:17:22:A5:AF:6E:F7:99:EC:FE:
                CA:65:45:D3:80:3D:88:18:63:90:A2:F6:6A:A9:4D:8C:90:F5:07:A6:
                88:B3:5B:E6:43:AB:AA:6A:A8:BE:50:0C:8A:EB:B2:FF:40:39:F1:34:
                C7:AF:9A:E4:68:EA:BC:40:83:A4:84:CB:33:68:E7:F9:6A:DF:99:61:
                28:FA:D1:5C:03:0F:B7:60:D2:DB:7E:5C:7C:9E:67:CD:12:21:7B:AF:
                04:9A:DD:A9:ED:75:F8:DB:1C:1F:84:B8:40:C5:40:2D:0E:19:98:B8:
                A0:41:AA:93:39:4B:B0:24:D7:59:0A:0D:6B:45:4B:34
parm:           nx_huge_pages:bool
parm:           nx_huge_pages_recovery_ratio:uint
parm:           ignore_msrs:bool
parm:           report_ignored_msrs:bool
parm:           min_timer_period_us:uint
parm:           kvmclock_periodic_sync:bool
parm:           tsc_tolerance_ppm:uint
parm:           lapic_timer_advance_ns:int
parm:           vector_hashing:bool
parm:           enable_vmware_backdoor:bool
parm:           force_emulation_prefix:bool
parm:           pi_inject_timer:bint
parm:           halt_poll_ns:uint
parm:           halt_poll_ns_grow:uint
parm:           halt_poll_ns_grow_start:uint
parm:           halt_poll_ns_shrink:uint
Comment 13 Larry Finger 2021-06-10 22:59:18 UTC
Please open a separate bug for the kernel. In that explain that modules signed by SUSE Linux Enterprise Secure Boot CA will load, but that those signed by openSUSE Secure Boot CA will not. Please let me know the ID.
Comment 14 Link Porterfield 2021-06-10 23:44:07 UTC
I'm not sure that I need to open a new bug, as I managed to work through it with some help from #opensuse on IRC.

'mokutil -l' showed only the SUSE Linux Enterprise Secure Boot CA certificate enrolled.

Reinstalling the openSUSE-signkey-cert package from YaST with upgrade unconditionally made the openSUSE Secure Boot Signkey appear in the output of 'mokutil -N' and shim presented an enrollment prompt on reboot.

Unfortunately it also asked for a password to enroll the certificate, and I had no idea what it wanted for that. Rather than force reinstall with YaST again, I used 'mokutil -i /etc/uefi/certs/BDD31A9E-kmp.crt' to stage the certificate again, and it prompted me to set a password, which I then was able to use in shim to enroll that certificate.

Both certificates are now enrolled and appear in the output of 'mokutil -l'.

Back to the original bug, it might be worth checking the affected system using 'mokutil -l' to see if it is missing the penSUSE Secure Boot Signkey provided by the openSUSE-signkey-cert package.
Comment 15 Larry Finger 2021-06-20 18:56:22 UTC
Has this problem been fixed by the addition of the extra key?
Comment 16 S. B. 2021-06-20 21:53:52 UTC
Hi Larry, thanks for following up. I honestly don't understand what's the deal here. I ended up releasing my images on 2021-06-08 with ` allow_unsupported_modules 1 ` in
/etc/modprobe.d/10-unsupported-modules.conf . Today I installed it in VirtualBox and commented out the change in 10-unsupported-modules.conf , but this time it still loaded the vboxguest module automatically. Something must have gotten fixed between the time I filed this bug and the date that I generated the ISOs. Sorry for the inconclusive test results.
Comment 17 Larry Finger 2021-06-21 16:39:16 UTC
As far as I know, there has never any problem with Leap 15.3 installed in VirtualBox. I turned secure boot off in my laptop BIOS as I generate my own kernels, and a lot of out-of-kernel modules that are not signed.

I think that the difference is that a Leap 15.3 distro has the "openSUSE Secure Boot CA" key that was probably missing from you Kiwi/Calamares setup.

I am going to close this bug report.
Comment 18 John 2022-02-26 03:06:41 UTC
Could this bug get reopened and fixed please? It is still a problem today after many VirtalBox versions released for Leap 15.3. And other people run into this issue as well as  you can see in the forum:


- VirtualBox:Linux kernel driver not loaded
https://forums.opensuse.org/showthread.php/566268-VirtualBox-Linux-kernel-driver-not-loaded

- Virtualbox kernel driver no loading - secureboot enabled - how to sign modules?
https://forums.opensuse.org/showthread.php/555343-Virtualbox-kernel-driver-no-loading-secureboot-enabled-how-to-sign-modules


I see now a related release note for Leap 15.3:

https://doc.opensuse.org/release-notes/x86_64/openSUSE/Leap/15.3/#driver-hardware-secure-boot-opensuse-kmps

What I don't understand is why it still doesn't get fixed after many months, in Leap 15.2 VirtualBox worked without issue and there was no need to add a second certificate for modules signed with a different key. Either VirtualBox should be signed with the same key or both required certificates should be in the default install which is not the case (installed today from https://download.opensuse.org/distribution/leap/15.3/iso/openSUSE-Leap-15.3-DVD-x86_64-Current.iso)

I don't know who does sign the modules but could this bug get fixed or reassigned please?

Also does anyone know how to disable CONFIG_MODULE_SIG=y as a workaround until this is fixed?

Thanks
Comment 19 Larry Finger 2022-02-26 19:35:26 UTC
The modules are signed by "SUSE Linux Enterprise Secure Boot CA".

You probably do not have that key in your EFI database. You can check that with the 'modutil -l | grep Issuer' command. Please post that output.

There are two steps to installing a new key. Step 1 is to import a signing certificate into the EFI database, and step 2 is to complete the enrollment on rebooting. Step 1 is done by openSUSE, but you need to do the second. If you fail the completion, the key is not available.
Comment 20 John 2022-02-28 09:07:18 UTC
(In reply to Larry Finger from comment #19)
> The modules are signed by "SUSE Linux Enterprise Secure Boot CA".

no, they are not. that's exactly the problem:

# modinfo vboxdrv | grep signer:
signer:         openSUSE Secure Boot CA


while the only accepted issuer in the default opensuse 15.3 installation is "SUSE Linux Enterprise Secure Boot CA"

# mokutil --list-enrolled | grep Issuer
        Issuer: CN=SUSE Linux Enterprise Secure Boot CA, C=DE, L=Nuremberg, O=SUSE Linux Products GmbH, OU=Build Team/emailAddress=build@suse.de


there are 2 solutions to this problem (excluding having end users add manually missing certificates)

- have next versions of virtualbox kernel modules signed with the "SUSE Linux Enterprise Secure Boot CA" key instead of the "openSUSE Secure Boot CA" key

- have both certificates for issuers  "SUSE Linux Enterprise Secure Boot CA" AND "openSUSE Secure Boot CA" included in the default opensuse install


so that virtualbox can be installed and executed without having to mess manually with certificates (which in my opinion is not good practice), like in previous releases (opensuse leap 15.2 worked fine)


now the questions is, who is responsible for signing the modules and could this bug get assigned to that person?
Comment 21 Larry Finger 2022-02-28 18:57:17 UTC
I do not know how I got the wrong key from modinfo. Today, I got the same as you.

Did you try the "fix" of comment #14, i.e. forcing the reinstallation of the openSUSE-signkey-cert package.

I filed a separate bug to see if I can get the key changed at build time as boo#1196563.
Comment 22 Larry Finger 2022-03-01 16:07:10 UTC
The reason for using a different key for the VB modules than for the kernel is to make it more difficult for SLE users to use VB. They must install a separate package, which increases the likelihood that they know what they are doing.

Does your system have the openSUSE-signkey-cert package installed?
Comment 23 Larry Finger 2022-03-02 20:14:49 UTC
I know more about this situation. First of all, the use of a different signing key for the VirtualBox modules than was used for the kernel is not a mistake or a bug. It is deliberate and arises from the SLE heritage of Leap 15.3 and presumably beyond.

To simulate your case. I added multiple versions to my secure boot system. I started with Leap 15.2, then added Tumbleweed. Next I installed Leap 15.3. When I finished, I used mokutil to find out what keys were installed. I had both, thus VB worked. To reproduce your situation, I deleted the "openSUSE Secure Boot CA" key with mokutil and rebooted. I got the MOK screen and completed the delete. At this point, I only had the "SUSE Linux Enterprise Secure Boot CA" key, and the VB modules could not load.

I then used YaST to remove the openSUSE-signkey-cert package, then reinstalled it. A forced reinstall might work. I then rebooted, got the MOK blue screen and installed the "new" key. After the reboot, both keys showed in the mokutil list, and VB worked. I do not know if you consider this "manual messing with certificates," but this is the only solution so far that works.
Comment 24 John 2022-03-02 21:36:47 UTC
thanks Larry!

(In reply to Larry Finger from comment #21)
> Did you try the "fix" of comment #14, i.e. forcing the reinstallation of the
> openSUSE-signkey-cert package.

no, i just used mokutil to form an enrolling request:

mokutil -i /etc/uefi/certs/BDD31A9E-kmp.crt

after restarting i enrolled it with the same password entered in mokutil (that does NOT have to be the root password as stated in a forum post)
 
> I filed a separate bug to see if I can get the key changed at build time as
> boo#1196563.

thanks, i'm reading now the original bug 

https://bugzilla.opensuse.org/show_bug.cgi?id=1195118

what i can say for sure is that doing a fresh Leap 15.3 install from the DVD iso (i didn't upgrade from 15.2) and installing virtualbox after that (and after upgrading packages) will lead to the problem. the package openSUSE-signkey-cert does get installed, but at no point is a password requested or an enrollment completed. in other words, it's not that i skipped or ignored it during installation, there was no request to set a password nor a blue EFI/BIOS screen to enroll the certificate, that appears only when doing it manually with mokutil -i or reinstalling the openSUSE-signkey-cert package manually as you wrote. 

still reading that bug, but even if it worked the expected way -if i  understand correctly- users would still have to enroll a second certificate manually (set and enter the password after rebooting). i wouldn't call that "user friendly" and hope a real fix is found where no manual interaction is needed at all, like it was in the past.