Bug 904987 - Dracut not working as expected with full-disk encryption
Dracut not working as expected with full-disk encryption
Status: RESOLVED INVALID
: 1024240 (view as bug list)
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Bootloader
Current
i686 All
: P2 - High : Critical with 5 votes (vote)
: Current
Assigned To: Daniel Molkentin
Jiri Srain
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2014-11-11 23:11 UTC by Federico Vecchiarelli
Modified: 2017-09-05 14:27 UTC (History)
16 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---
daniel: needinfo?


Attachments
Screenshot - emergency console after upgrade (1.38 MB, image/jpeg)
2014-11-11 23:11 UTC, Federico Vecchiarelli
Details
Screenshot - List of disks by-uuid (1.46 MB, image/jpeg)
2014-11-11 23:16 UTC, Federico Vecchiarelli
Details
dracut-shell (4.44 MB, image/jpeg)
2017-01-08 23:12 UTC, Alexander Naumov
Details
Unit systemd-cryptsetup@luks.service not found (2.59 MB, image/jpeg)
2017-02-01 14:51 UTC, Chris Targett
Details
lsinitrd outputs (3.71 MB, image/jpeg)
2017-02-01 15:01 UTC, Chris Targett
Details
Output of systemd-cryptsetup-generator (784.73 KB, image/jpeg)
2017-02-08 17:00 UTC, Joost W
Details
output of ls -l /dev/disk/by-id (3.01 MB, image/jpeg)
2017-02-08 18:07 UTC, Joost W
Details
/dev listing with changed devi/by-id/ names and old /etc/crypttab (4.03 KB, text/plain)
2017-07-25 15:55 UTC, Chris Scheible
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Federico Vecchiarelli 2014-11-11 23:11:38 UTC
Created attachment 613252 [details]
Screenshot -  emergency console after upgrade

Hi there,

My laptop has full disk encryption with luks. It was working without problems until openSUSE 13.2.

It seems dracut checks for encrypted partitions and volumes on crypttab however I did not have the partition listed in crypttab because all this time the luks partition information was added by grub (if not mistaken). Well, dracut seems can't recognize an unlocked luks partition/volume on its own nor from the previous used settings.

Dracut was then trying to look for the volumes for home and root without trying to unlock first for the encrypted partition.

There are two workarounds for this. One, use the kernel boot parameter rd.luks.uuid the other, add the disk to crypttab and then re-run dracut -M --force

This however it is not obvious for those doing an upgrade. After doing an upgrade from 13.1 to 13.2, the system is no longer bootable unless you take care of the above first.

Thank you.

Regards,

Fede
Comment 1 Federico Vecchiarelli 2014-11-11 23:16:35 UTC
Created attachment 613254 [details]
Screenshot - List of disks by-uuid

In this screenshot you can see dracut looking for the root volume, however, because it has not unlocked the encrypted luks (on the screenshot shown as 3c8dd99d-28c7-4258-a19f-c75e0a067076), the boot process can't continue.
Comment 2 Thomas Renninger 2014-11-12 16:40:19 UTC
There was a similar report already?
As the 13.2 medium is out already I guess all that can be done is writing an article on the opensuse wiki page...

Federico: Would you mind doing so, please.
Please take care to have key words like 13.1 to 13.2 update/upgrade crypt/whatever included, so that others can find it quickly.

Then just paste in your workaround as described in this bug.
A reference to this bug may also be nice if we still find some more that can be done.

This would help others with a similar setup a lot.
Afaik Julian tried an update and it worked, but the setup may have been slightly different?
Comment 3 Federico Vecchiarelli 2014-11-13 04:06:27 UTC
I found a mention of this problem in this thread:

https://www.mail-archive.com/initramfs@vger.kernel.org/msg03622.html

The actual thread mentions a few things but you will find the same particular issue happened. Here is an excerpt:

================
> This doesn't work with full disk encryption. Unit file does not get
> generated and I see this message:
> 
> [    7.142993] testhost dracut-initqueue[202]: Failed to start 
> systemd-cryptsetup@luks\x2d342d2414\x2d159c\x2d48d7\x2da0b7\x2d5b59fa6e27a5.service:
>  Unit 
> systemd-cryptsetup@luks\x2d342d2414\x2d159c\x2d48d7\x2da0b7\x2d5b59fa6e27a5.service
>  failed to load: No such file or directory.

Note that I don't have an entry for this luks device in /etc/crypttab. I
always thought that this is not necessary since all needed options
are in cmdline.
=================

If the best is to write a wiki article about it, I surely can give that a try.

Regards,

Federico
Comment 4 Vojtech Zeisek 2014-11-27 09:52:48 UTC
I didn't experience that problem I have in laptop mSATA with boot and encrypted LVM containing root and swap, and large HDD contains encrypted home. root and home use ext4. So that in 13.1 I was asked for two passwords during the boot. I on-line upgraded to 13.2 and there was no problem in the boot. The only think is I'm asked only for one password during the boot. I have no idea how his is possible. But it works. All partitions are correctly mounted.
Comment 5 Federico Vecchiarelli 2014-11-27 10:25:19 UTC
In my case I did not have the entry for my encrypted volumes in crypttab. That is what I think caused the issue.

Do you have an entry for yours?
Comment 6 Vojtech Zeisek 2014-11-27 11:28:04 UTC
Yes, I do

$ cat /etc/crypttab 
cr_home         /dev/disk/by-id/ata-ST1000LM014-1EJ164_W3802Q1Y-part1 none       none
cr_ata-KINGSTON_SMS200S360G_50026B72420708CE-part3 /dev/disk/by-id/ata-KINGSTON_SMS200S360G_50026B72420708CE-part3 none       none

$ cat /etc/fstab
UUID=63dab9f4-dbe1-44be-a93d-b7a6e141b44d       /       ext4    noatime,defaults,noacl,user_xattr 1 1 
/dev/mapper/cr_home     /home   ext4    acl,user_xattr,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0,nofail 0 2 
/dev/disk/by-id/ata-KINGSTON_SMS200S360G_50026B72420708CE-part2 /boot   ext4    acl,user_xattr 1 2 
UUID=cb0a2ff8-b112-4eb4-af0b-549c8ca05697       swap    swap    defaults 0 0 
/dev/disk/by-id/ata-KINGSTON_SMS200S360G_50026B72420708CE-part1 /boot/efi       vfat    umask=0002,utf8=true 0 0

I haven't touched those files.
Comment 7 Federico Vecchiarelli 2014-11-27 14:37:54 UTC
OK, so the issue remains. Without an entry on crypttab, dracut will not provide support for the encrypted volumes (even if they are currently mounted and grub add sorry for them).

In the past it was not really necessary to have the entry in crypttab because grub would take care of them. Now, without that entry, an upgrade to 13.2 will render the system unbootable.

Regards
Comment 8 Vojtech Zeisek 2014-11-27 15:06:43 UTC
I didn't create cryptab. I encrypted the system during installation of 13.1, so the installer had to create it, I guess.
Comment 9 Alexander Naumov 2017-01-08 23:12:40 UTC
Created attachment 709050 [details]
dracut-shell
Comment 10 Alexander Naumov 2017-01-08 23:24:31 UTC
Hey,

I can reproduce it with kernel 4.9.0-2.gfd5379c-pae (i686).

After entering passphrase system froze for 2 mins. After that we get kernel-message every second:
dracut-initqueue[131]: Warning dracut-initqueue timeout - starting timeout stripts 

After 1 min we get dracut-shell...

I can't mount USB stick (for copying /run/initramfs/rdsosreport.txt), kernel doesn't see new devices.
Kernel 4.8.13-1-default works well.
Comment 11 Alexander Naumov 2017-01-09 09:02:50 UTC
Update kernel to 4.10 (from /Kernel:/HEAD/standard) fixes this problem.
Comment 12 Daniel Molkentin 2017-01-13 13:44:56 UTC
@Alexander Naumov Please describe your upgrade path. When did this start to fail exactly? Was it related to a distro upgrade?
Comment 13 Chris Targett 2017-02-01 14:51:56 UTC
Created attachment 712432 [details]
Unit systemd-cryptsetup@luks.service not found

I am getting

> Failed to start systemd-cryptsetup@luks.service: Unit systemd-cryptsetup@luks.service not found.

This is the result of a default/suggested LUKS+LVM install using the 30-Jan Tumbleweed snapshot ISO.

I also believe it occurred after when doing a `zypper up` ~2 days ago.
Comment 14 Chris Targett 2017-02-01 15:01:58 UTC
Created attachment 712434 [details]
lsinitrd outputs
Comment 15 Daniel Molkentin 2017-02-06 17:46:24 UTC
I just setup tumbleweed from 04022017 with lvm-based cryptoroot. I cannot reproduce this. Do you have a crypttab in your initrd? Specifically, please provide the output of

lsinitrd -f etc/crypttab.
Comment 16 Joost W 2017-02-08 10:38:40 UTC
Hi everybody, sorry, found this only after placing another bug, please have a look here as my issue seems similar:

https://bugzilla.opensuse.org/show_bug.cgi?id=1024240

Bye, 

Joost
Comment 17 Daniel Molkentin 2017-02-08 14:44:24 UTC
*** Bug 1024240 has been marked as a duplicate of this bug. ***
Comment 18 Daniel Molkentin 2017-02-08 14:46:37 UTC
Please provide the output of the following:

cat /etc/crypttab
sudo lsinitrd -f etc/crypttab
/usr/lib/systemd/system-generators/systemd-cryptsetup-generator && ls /tmp/*.service
Comment 19 Joost W 2017-02-08 16:53:15 UTC
(In reply to Daniel Molkentin from comment #18)
> Please provide the output of the following:
> 
> cat /etc/crypttab
> sudo lsinitrd -f etc/crypttab
> /usr/lib/systemd/system-generators/systemd-cryptsetup-generator && ls
> /tmp/*.service

Hi, thanks a lot for taking this up.

crypttab in initrd is a 0 byte file (empty).


Chrooting to manually mounted system partition with:

cryptsetup luksOpen /dev/nvme0n1p3 system
--- PW entry
lvm_scan
lvm vgchange -ay
mkdir /mnt
mount /dev/mapper/system-root /mnt
mount --rbind /dev /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys /mnt/sys
chroot /mnt


system /etc/crypttab:
cr_nvme-200080d030005f1a7-part3 /dev/disk/by-id/nvme-200080d030005f1a7-part3 none none

Resulting file after generator run: please see attachment (as I can not boot anymore, it's typing or taking a picture for me).
Comment 20 Joost W 2017-02-08 17:00:15 UTC
Created attachment 713413 [details]
Output of systemd-cryptsetup-generator
Comment 21 Joost W 2017-02-08 17:53:58 UTC
One additional thought: is the unit file dependent on the /dev/disk/by-id devnames (I get this impression from looking at the file).

I ran ls -l /dev/disk/by-id in my initrd, where I get a completely different id for the luks encrypted partition than what is written in the crypttab and in the unit file (please see attachment). Might that be a lead?
Comment 22 Joost W 2017-02-08 18:07:16 UTC
Created attachment 713424 [details]
output of ls -l /dev/disk/by-id
Comment 23 Joost W 2017-02-08 21:59:12 UTC
Ok, some progress. I managed to boot into multiuser.target (not into graphical.target) with 

- chrooting from dracut rescue shell into manually mounted system partition (see above)
- changing system /etc/crypttab, using one line/entry with the /dev/disk/by-uuid/ name of the encrypted partition, options "none none"
- mount /boot & dracut -f in the chroot

Now, the crypttab gets copied into the initrd and the password prompt appears on next boot.

Though my boot is still not completely restored, as a boot into graphical.target hangs indefinitely after "Switching root" at the end of the systemd boot sequence, without X / SDDM coming up and without dropping to a shell.
Work around for now is to boot into multi-user.target (add "3" to the kernel boot command in GRUB), and then log in, afterwards run

systemctl isolate graphical.target.

Then X comes up. I have to 

systemctl restart NetworkManager

to get networking, then things seem to work ok.
Comment 24 Steffen Golle 2017-02-19 20:26:48 UTC
I ran into the same problem. openSUSE Tumbleweed with Kernel 4.9.10 is not starting anymore. At boot, the plymouth splash screen doesn't show the password prompt. The system message from dracut-initqueue is: Failed to connect to the bus. No such file or directory.

Can you tell me, how to enter the dracut shell from grub?
Comment 25 Joost W 2017-02-20 19:51:29 UTC
(In reply to Steffen Golle from comment #24)
> I ran into the same problem. openSUSE Tumbleweed with Kernel 4.9.10 is not
> starting anymore. At boot, the plymouth splash screen doesn't show the
> password prompt. The system message from dracut-initqueue is: Failed to
> connect to the bus. No such file or directory.
> 
> Can you tell me, how to enter the dracut shell from grub?
The dracut shell should come up automatically if booting fails, though it has a delay of 1-2minutes; try to boot and come back 5 minutes later.
At least your error message is different from the one I had, perhaps you have more luck in fixing it. I had to give up and do a complete reinstall.
Comment 26 Chris Targett 2017-03-06 20:50:37 UTC
I have just attempted to install Tumbleweed on a new hardware.

 - Intel NUC
 - Corsair Force NVME SSD
 - OpenSUSE-Tumbleweed-DVD-x86_64-Snapshot20170304-Media.iso (4.10.1-1-default)

*all* installs have been booted through UEFI.

* First install attempt using the LUKS+LVM option resulted in 3 partitions. efi(vfat), boot(ext4), and system(lvm).

  On boot no password entry was requested, and was not present on any alternate TTYs. System dropped to a dracut shell.

* Second install attempt, the installer initially wanted to install to the USB stick that was installing it. When changing the target disk and selecting the LUKS+LVM option, only two partitions were created. efi(vfat) and system(lvm). 

  On boot Grub requested the LVM password, and the subsequent graphical boot also requested the password. System booted flawlessly.


* Third install attempt matched the third. No password prompt on any TTY.

  In dracut shell:
   * /etc/crypttab was empty

  In chrooted mount (from dracut shell):
   * /etc/crypttab contains `cr_-Force-part3 /dev/disk/by-id/-Force-part3 none none` There is no /dev/disk/by-id matching that name.
   * `lsinitrd -f etc/crypttab` returns no output. But `lsinitrd | grep crypttab` lists a 0 byte file.
   * `/usr/lib/systemd/system-generators/systemd-cryptsetup-generator` creates `cryptsetup.target.requires`, `dev-disk-by-id\x2did-\x2dForce\x2dpart3.device.wants`, `dev-mapper-cr_\x2dForce\x2dpart3.device.d`, `dev-mapper-cr_\x2dForce\x2dpart3.device.requires`, `systemd-cryptsetup@cr_\x2dForce\x2dpart3.service`. 


Running `dracut -f` did not change the size of crypttab in `lsinitrd`. 
Fixing the crypttab to point to somewhere that does exist, and then running `dracut -f` fixed the size of the crypttab in the initrd. On boot the password was requested, but did not progress past the loading throbber.
Comment 27 Fabian Vogt 2017-03-06 21:06:03 UTC
(In reply to Chris Targett from comment #26)
> I have just attempted to install Tumbleweed on a new hardware.
> 
>  - Intel NUC
>  - Corsair Force NVME SSD
>  - OpenSUSE-Tumbleweed-DVD-x86_64-Snapshot20170304-Media.iso
> (4.10.1-1-default)
> 
> *all* installs have been booted through UEFI.
> 
[...]
> * Third install attempt matched the third. No password prompt on any TTY.
> 
>   In dracut shell:
>    * /etc/crypttab was empty
> 
>   In chrooted mount (from dracut shell):
>    * /etc/crypttab contains `cr_-Force-part3 /dev/disk/by-id/-Force-part3
> none none` There is no /dev/disk/by-id matching that name.

Please open a new bug report against YaST2, as it puts a wrong udev id in there and attach the file generated by "save_y2logs" and the output of "hwinfo --disk".
Comment 28 Markus Greger 2017-07-07 22:57:50 UTC
Had same problem (boot only after manually entering cryptsetup luksOpen <device> <name> and then exiting) because of empty /etc/crypttab.

Updated /etc/crypttab and am able to boot properly (at least with own kernels).
Comment 29 Sebastian Rettenberger 2017-07-24 11:19:25 UTC
Is there any update to this issue?

I stumbled across when applying updates to Leap 42.2. The default installation works well but after installing all updates, the system does not show the password prompt.

The system did not fall into the dracut shell but I can access it with the rescue system from the installation medium.

- /etc/crypttab is correct
- etc/crypttab in the initrd was empty
- Running dracut -f fixed the crypttab in the initrd but still no password prompt after reboot.

Is there anything else I need to run on the rescue system?
Comment 30 Dariusz Ostolski 2017-07-24 11:27:50 UTC
Hello,

I have the same issue but on Leap 42.2. My setup is LVM+LUKS, after doing zypper up to the latest version system is unbootable. I've tried to reproduce this issue on VM but it was not possible, so I guess it is somehow related to my configuration which is mSATA drive.

Link to the duplicated issue:
https://bugzilla.opensuse.org/show_bug.cgi?id=1049359
Comment 31 Dan Elder 2017-07-24 14:23:10 UTC
I'm running into the exact same problem as everyone else on 42.2 with latest updates applied using an NVMe.  I was able to restore a previous kernel/initrd from offline backup (4.4.70-18.9-default) and get back into a usable system but the latest updates never prompt for the password.  In case it helps at all:

delder:~ # cat /etc/crypttab
cr_nvme-200080d0200127f94-part4 /dev/disk/by-id/nvme-200080d0200127f94-part4 none       none
cr_ata-Samsung_SSD_850_EVO_2TB_S2HCNWAG702239M-part1 /dev/disk/by-id/ata-Samsung_SSD_850_EVO_2TB_S2HCNWAG702239M-part1 none       none
cr_swap         /Slow/swap           none       noauto

delder:~ # lsinitrd -f etc/crypttab
cr_nvme-200080d0200127f94-part4 /dev/disk/by-id/nvme-200080d0200127f94-part4 none       none

delder:~ # /usr/lib/systemd/system-generators/systemd-cryptsetup-generator && ls /tmp/*.service
Failed to create unit file /tmp/systemd-cryptsetup@cr_nvme\x2d200080d0200127f94\x2dpart4.service: File exists
/tmp/systemd-cryptsetup@cr_ata\x2dSamsung_SSD_850_EVO_2TB_S2HCNWAG702239M\x2dpart1.service  /tmp/systemd-cryptsetup@cr_swap.service
/tmp/systemd-cryptsetup@cr_nvme\x2d200080d0200127f94\x2dpart4.service
Comment 32 Chris Scheible 2017-07-25 15:47:04 UTC
Hi,

I hit the same issue after applying the latest patches to OpenSuse 42.2 two weeks ago on my system (HP Spetre 13 w/ NVME SSD). The system is configured with LUKS encrypted system LVM, only /boot is unencrypted. The encrypted LVM setup was done by the OpenSuse installer when I provisioned the system.

The symptom for me was exactly the same as in the other bug reports: no password prompt for unlocking and mounting the rootfs logical volume during boot.

The root cause that I found on the system was that the OpenSuse patch update automagically renamed the partitions in /dev/disk/by-id/ . However, the update neither rebuilt the /etc/crypttab with the new partition names nor rebuilt the initrd.

To get the system working again, I used the following procedure:

* Boot into emergency shell from grub menu (add '1' to end of kernel boot line), the shell with the boot output just errors out after not finding the rootfs and tells me to rebuild the initrd... 
* Selected emergency shell with ALT+F7 (shouldn't the emergency shell prompt open on the same shell as the boot messages?)
* dracut emergency shell /etc/crypttab contained old /dev/disk/by-id/ partition ...

* Booted rescue system from USB stick and setup chroot environment. Did the following in the chroot environment:

ls -l /dev/disk/by-id/
(find partition with the encrypted physical volume for the system LVM)
cryptsetup luksOpen /dev/nvme0n1p9 
lvscan
cp /etc/crypttab /etc/crypttab.old
vi /etc/crypttab
(replaced the wrong /dev/disk/by-id/... path with the absolute partition path /dev/nvme0n1p9)
mkinitrd
(rebuilt initrd, copies the new /etc/crypptab to initrd filesystem)
exit chroot
reboot

==> system now asks for password to unlock encrypted partition and boots as expected after password entry.
Comment 33 Chris Scheible 2017-07-25 15:55:42 UTC
Created attachment 733768 [details]
/dev listing with changed devi/by-id/ names and old /etc/crypttab
Comment 34 Daniel Molkentin 2017-07-26 12:58:34 UTC
This looks a lot like bug 1048679 (udev regression).
Comment 35 Daniel Molkentin 2017-09-05 14:27:33 UTC
I agree with Fabian, this bug is not describing a single issue. I am therefore closing it. Please reopen new issues where applicable.