Bug 1171055 - [Build 20200502] kdump_and_crash: Failed to start Switch Root
[Build 20200502] kdump_and_crash: Failed to start Switch Root
Status: RESOLVED DUPLICATE of bug 1172670
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Basesystem
Current
Other Other
: P5 - None : Normal (vote)
: ---
Assigned To: Petr Tesařík
E-mail List
https://openqa.opensuse.org/tests/125...
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2020-05-04 10:15 UTC by Dominique Leuenberger
Modified: 2020-09-01 06:32 UTC (History)
5 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
0001-create-initrd.target.wants-directory-for-kdump-depen.patch (816 bytes, patch)
2020-05-18 15:11 UTC, Thomas Blume
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Dominique Leuenberger 2020-05-04 10:15:42 UTC
## Observation

openQA test in scenario opensuse-Tumbleweed-JeOS-for-kvm-and-xen-x86_64-jeos-extra@64bit_virtio-2G fails in
[kdump_and_crash](https://openqa.opensuse.org/tests/1254001/modules/kdump_and_crash/steps/45)

## Test suite description
Same as jeos, plus some more tests.


## Reproducible

Fails since (at least) Build [20200502](https://openqa.opensuse.org/tests/1253486)


## Expected result

Last good: [20200501](https://openqa.opensuse.org/tests/1252309) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=JeOS-for-kvm-and-xen&machine=64bit_virtio-2G&test=jeos-extra&version=Tumbleweed)
Comment 1 Dominique Leuenberger 2020-05-04 10:17:58 UTC
Assigning to the drcut maintainers first; dracut is amongst the most likely changes in snapshot 0502
Comment 2 Thomas Blume 2020-05-04 13:01:01 UTC
(In reply to Dominique Leuenberger from comment #1)
> Assigning to the drcut maintainers first; dracut is amongst the most likely
> changes in snapshot 0502

Hm, the initrd creation from https://openqa.opensuse.org/tests/1254001/file/serial_terminal.txt shows 2 errors:

-->
dracut: *** Including module: systemd ***
Failed to add dependency on unit, unit systemd-ask-password-plymouth.service does not exist.
[...]
dracut-install: Failed to find module 'crc32c'
dracut: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.3z3lRx/initramfs -H -N i2o_scsi --kerneldir /lib/modules/5.6.8-1-default/ -m crc32c
--<

But that isn'c from the kdump initrd.
I don't see any console log from the kexec boot either.
Is that available?

Otherwise I will try booting the asset via kvm and reproducing the error unless you have a better idea how to get the logs. ;)
Comment 3 Dominique Leuenberger 2020-05-04 13:07:23 UTC
The easiest will likely really be to grab the disk image (as produced by the parent test) and reproduce it locally; the hdd image should already have the correct repositories against openQA repo registered
Comment 4 Thomas Blume 2020-05-05 07:46:56 UTC
I could reproduce the issue, the rdsosreport log shows:

-->
[    3.433455] localhost systemctl[355]: Failed to switch root: Specified switch root path '/sysroot' does not seem to be an OS tree. os-release file is missing.
[    3.434389] localhost systemd[1]: initrd-switch-root.service: Main process exited, code=exited, status=1/FAILURE
[    3.434509] localhost systemd[1]: initrd-switch-root.service: Failed with result 'exit-code'.
[    3.434744] localhost systemd[1]: Failed to start Switch Root.
[    3.437077] localhost systemd[1]: initrd-switch-root.service: Triggering OnFailure= dependencies.
--<

This is because /kdump is mounted to /sysroot:

92 1 0:2 /kdump /sysroot ro shared:1 - rootfs none rw

where dracut normally expects the operating system.
But the /sysroot mount only shows:

-->
:/root# ls -l /sysroot/
total 0
lrwxrwxrwx 1 root root 10 May  5 09:11 boot -> mnt0/boot/
drwxr-xr-x 2 root root  0 May  5 09:11 mnt0
drwxr-xr-x 3 root root  0 May  5 09:11 mnt1
drwxrwxrwt 2 root root  0 May  5 09:11 tmp
--<

I'm not really sure yet how this is supposed to work, but the above is clearly not what should be present below /sysroot to continue the boot.
Comment 5 Thomas Blume 2020-05-07 07:34:49 UTC
During my tests, I saw that some files seem to be missing from the kdump package:

-->
c747:~ # zypper in kdump
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following NEW package is going to be installed:
  kdump

1 new package to install.
Overall download size: 253.0 KiB. Already cached: 0 B. After the operation, additional 640.4 KiB will be used.
Continue? [y/n/v/...? shows all options] (y): 
Retrieving package kdump-0.9.0-13.1.x86_64                                                                                                               (1/1), 253.0 KiB (640.4 KiB unpacked)
Retrieving: kdump-0.9.0-13.1.x86_64.rpm ...................................................................................................................................[done (10.1 KiB/s)]

Checking for file conflicts: ...........................................................................................................................................................[done]
(1/1) Installing: kdump-0.9.0-13.1.x86_64 ..............................................................................................................................................[done]
Additional rpm output:
Updating /etc/sysconfig/kdump ...


c747:~ # rpm -ql kdump | grep mkdumprd
/usr/sbin/mkdumprd
/usr/share/man/man8/mkdumprd.8.gz
c747:~ # 
c747:~ # ls -l /usr/share/man/man8/mkdumprd.8.gz
ls: cannot access '/usr/share/man/man8/mkdumprd.8.gz': No such file or directory
c747:~ # 
--<

Petr, could you take a look?
Comment 6 Daniel Molkentin 2020-05-18 08:05:07 UTC
Ping?
Comment 7 Thomas Blume 2020-05-18 14:31:54 UTC
That looks suspicious:

-->
# rpm -q kdump
kdump-0.9.0-13.1.x86_64
# mkdumprd -f
Regenerating kdump initrd ...
dracut: Executing: /usr/bin/dracut --force --hostonly --omit "plymouth resume usrmount" "--compress=xz -0 --check=crc32" --add kdump /boot/initrd-5.6.8-1-default-kdump 5.6.8-1-default
[...]
dracut: *** Including module: kdump ***
ln: failed to create symbolic link '/var/tmp/dracut.3o3Ldx/initramfs//usr/lib/systemd/system/initrd.target.wants/kdump-save.service': No such file or directory
[...]
#
#
# rpm -ql kdump | grep kdump-save
/usr/lib/dracut/modules.d/99kdump/kdump-save.service.in
                                                     ^^
#
# ls -l /usr/lib/dracut/modules.d/99kdump/kdump-save.service*
-rw-r--r-- 1 root root 826 Apr  4 12:40 /usr/lib/dracut/modules.d/99kdump/kdump-save.service.in

The package really seems to be broken.
Comment 8 Thomas Blume 2020-05-18 15:10:43 UTC
(In reply to Thomas Blume from comment #7)
> That looks suspicious:
> 
> -->
> # rpm -q kdump
> kdump-0.9.0-13.1.x86_64
> # mkdumprd -f
> Regenerating kdump initrd ...
> dracut: Executing: /usr/bin/dracut --force --hostonly --omit "plymouth
> resume usrmount" "--compress=xz -0 --check=crc32" --add kdump
> /boot/initrd-5.6.8-1-default-kdump 5.6.8-1-default
> [...]
> dracut: *** Including module: kdump ***
> ln: failed to create symbolic link
> '/var/tmp/dracut.3o3Ldx/initramfs//usr/lib/systemd/system/initrd.target.
> wants/kdump-save.service': No such file or directory

Argh, it is actually caused by a change in dracut!
dracut-0.50 ships a rootfs-generator that creates the initrd.target.wants directory on the fly:

-->
/usr/lib/dracut/modules.d/98dracut-systemd/rootfs-generator.sh:        [ -d "$GENERATOR_DIR"/initrd.target.wants ] || mkdir -p "$GENERATOR_DIR"/initrd.target.wants
--<

So initrd.target.wants is not present at initrd creation time and the symlink creation for kdump-save.service fails.
Comment 9 Thomas Blume 2020-05-18 15:11:47 UTC
Created attachment 837940 [details]
0001-create-initrd.target.wants-directory-for-kdump-depen.patch

The attached patch for the kdump package fixed it in my tests.
Comment 10 Daniel Molkentin 2020-05-18 15:33:27 UTC
Thanks a lot Thomas. One nitpick: It seems that calling "systemctl -q --root "$initdir" add-wants ..." is preferred these days and should handle the directory creation implicitly. At least that's what upstream did change to more and more in the past, and it seems reasonable not to duplicate code paths.
Comment 11 Thomas Blume 2020-05-22 15:57:00 UTC
(In reply to Daniel Molkentin from comment #10)
> Thanks a lot Thomas. One nitpick: It seems that calling "systemctl -q --root
> "$initdir" add-wants ..." is preferred these days and should handle the
> directory creation implicitly. At least that's what upstream did change to
> more and more in the past, and it seems reasonable not to duplicate code
> paths.

Ok, thanks Daniel.
Dominique, I have created testpackages at:

https://build.opensuse.org/package/show/home:tsaupe:branches:openSUSE:Factory:kdump-bsc1171055/kdump

feel free to give it a try.
Comment 14 Fabian Vogt 2020-07-27 07:59:19 UTC
This is still completely broken in Tumbleweed, I assume the fix was not submitted?

Reassigning to kdump maintainer.
Comment 15 Fabian Vogt 2020-09-01 06:32:37 UTC
While the other bug is a duplicate of this one, let's mark it the other way around as the other one has a PR pending.

*** This bug has been marked as a duplicate of bug 1172670 ***