Bug 1170255 - Dracut 049 / 050 use kernel device names instead of udev names when calling ifup
Dracut 049 / 050 use kernel device names instead of udev names when calling ifup
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Basesystem
Current
Other Other
: P5 - None : Normal (vote)
: ---
Assigned To: dracut maintainers
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2020-04-22 18:19 UTC by Alberto Planas Dominguez
Modified: 2020-04-29 14:45 UTC (History)
2 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
dracut log file (1.87 MB, text/plain)
2020-04-22 18:19 UTC, Alberto Planas Dominguez
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Alberto Planas Dominguez 2020-04-22 18:19:05 UTC
Created attachment 836490 [details]
dracut log file

When booting a OEM Kiwi via PXE boot, when dracut try to set the network configuration, a timeout appears:

Warning: dracut-initqueue timeout - starting timeout scripts

After some debug (see attached) looks like that the pre-udev hook call net-genrules, that enqueue calls to "ifup eth0" and "ifup eth1", instead of the expected "ifup ens3" and "ifup ens4".

Note that a manual execution of "ifup ensX" during the recovery shell succeed.
Comment 1 Thomas Blume 2020-04-23 08:44:18 UTC
(In reply to Alberto Planas Dominguez from comment #0)
> Created attachment 836490 [details]
> dracut log file
> 
> When booting a OEM Kiwi via PXE boot, when dracut try to set the network
> configuration, a timeout appears:
> 
> Warning: dracut-initqueue timeout - starting timeout scripts
> 
> After some debug (see attached) looks like that the pre-udev hook call
> net-genrules, that enqueue calls to "ifup eth0" and "ifup eth1", instead of
> the expected "ifup ens3" and "ifup ens4".

It seems that the boot parameters are missing ifname= settings, hence the kernel names are used.

/usr/lib/dracut/modules.d/40network/ifname-genrules.sh shows:

-->
# if there are no ifname parameters, just use NAME=KERNEL
--<

Could you attach the mkinitrd.log for the kiwi initrd?
Comment 2 Alberto Planas Dominguez 2020-04-23 09:06:13 UTC
(In reply to Thomas Blume from comment #1)

> It seems that the boot parameters are missing ifname= settings, hence the
> kernel names are used.

It is Tumbleweed, so ifname should not be required as I want to use the predictive names.

This same command line (except the debuging parameters) was used for some years, and was OK on TW and SLE. One taking the predictive names and setting up the network, and other using the kernel ones. So there is a regression somewhere.

Also note that in the logs, at the beginning, `ip a s` list the interfaces with ensX, that are indeed the ones that I want to use. Dracut should call `ifup ensX`, and no `ifup ethX`

> /usr/lib/dracut/modules.d/40network/ifname-genrules.sh shows:
> 
> -->
> # if there are no ifname parameters, just use NAME=KERNEL
> --<

Interesting, this file was also not updated since long time ago. Also I can be wrong, but I would expect that no rule is needed, as is udev the one that decides how to name the interface. We are in the stage where the first PXE initrd is taking over, and is configuring the network before downloading the second kernel and the full image.

> Could you attach the mkinitrd.log for the kiwi initrd?

Where I can find it? I can point to:

https://build.opensuse.org/build/systemsmanagement:yomi/images/x86_64/openSUSE-Tumbleweed-Yomi:pxeboot/_log
Comment 3 Alberto Planas Dominguez 2020-04-23 09:08:33 UTC
(In reply to Alberto Planas Dominguez from comment #2)
> (In reply to Thomas Blume from comment #1)

> > Could you attach the mkinitrd.log for the kiwi initrd?
> 
> Where I can find it? I can point to:
> 
> https://build.opensuse.org/build/systemsmanagement:yomi/images/x86_64/
> openSUSE-Tumbleweed-Yomi:pxeboot/_log

Also the same bug is happening on ARM too.
Comment 4 Alberto Planas Dominguez 2020-04-23 10:58:41 UTC
(In reply to Thomas Blume from comment #1)
cted "ifup ens3" and "ifup ens4".
> 
> It seems that the boot parameters are missing ifname= settings, hence the
> kernel names are used.

Disabling udev names on tumbleweed (net.ifnames=0) works, but we should be able to use consistent naming on TW too.

Also tested an explicit net.ifnames=1, and as expected shows the same bug (device names are ensX, that is correct, but ifup is called under ethX names)
Comment 5 Thomas Blume 2020-04-23 13:19:35 UTC
(In reply to Alberto Planas Dominguez from comment #4)
> (In reply to Thomas Blume from comment #1)
> cted "ifup ens3" and "ifup ens4".
> > 
> > It seems that the boot parameters are missing ifname= settings, hence the
> > kernel names are used.
> 
> Disabling udev names on tumbleweed (net.ifnames=0) works, but we should be
> able to use consistent naming on TW too.
> 
> Also tested an explicit net.ifnames=1, and as expected shows the same bug
> (device name

I could reproduce the issue on a current tumbleweed when using the kiwi parameters to build the initrd:

/usr/bin/dracut --verbose --no-hostonly --no-hostonly-cmdline --omit " multipath " /boot/initrd_kiwi_install.xz

The debug log shows that the udev device processing is triggered before the interface gets renamed:

-->
[   18.926278] localhost kernel: virtio_net virtio0 ens3: renamed from eth0
[...]
--<

versus:

-->
[   18.829288] localhost systemd-udevd[479]: eth0: Processing device (SEQNUM=2047, ACTION=add)
[   18.829303] localhost systemd-udevd[479]: eth0: /usr/lib/udev/rules.d/75-net-description.rules:6 Importing properties from results of builtin command 'net_id'
[   18.829320] localhost systemd-udevd[479]: Using default interface naming scheme 'v245'.
[   18.829335] localhost systemd-udevd[479]: eth0: /usr/lib/udev/rules.d/75-net-description.rules:12 Importing properties from results of builtin command 'hwdb --subsystem=pci'
[   18.829350] localhost systemd-udevd[479]: eth0: /usr/lib/udev/rules.d/75-net-description.rules:12 Failed to run builtin 'hwdb --subsystem=pci': Invalid argument
[   18.829364] localhost systemd-udevd[479]: eth0: /usr/lib/udev/rules.d/80-net-setup-link.rules:5 Importing properties from results of builtin command 'path_id'
[   18.829379] localhost systemd-udevd[479]: eth0: /usr/lib/udev/rules.d/80-net-setup-link.rules:9 Importing properties from results of builtin command 'net_setup_link'
[   18.829395] localhost systemd-udevd[479]: eth0: Config file /usr/lib/systemd/network/99-default.link is applied
[   18.829410] localhost systemd-udevd[479]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[   18.829425] localhost systemd-udevd[479]: eth0: Failed to query name_assign_type: Invalid argument
[   18.829440] localhost systemd-udevd[479]: eth0: Policy *slot* yields "ens3".
[   18.829456] localhost systemd-udevd[479]: eth0: Device has addr_assign_type=0
[   18.829471] localhost systemd-udevd[479]: eth0: MAC on the device already matches policy *persistent*
[   18.829485] localhost systemd-udevd[479]: eth0: /usr/lib/udev/rules.d/80-net-setup-link.rules:11 NAME 'ens3'
[   18.829500] localhost systemd-udevd[479]: eth0: /etc/udev/rules.d/91-default-net.rules:1 RUN '/sbin/initqueue --name ifup-$env{INTERFACE} --unique --onetime /sbin/ifup $env{INTERFACE}'
[   18.829515] localhost systemd-udevd[479]: eth0: /usr/lib/udev/rules.d/99-systemd.rules:62 RUN '/usr/lib/systemd/systemd-sysctl --prefix=/net/ipv4/conf/$name --prefix=/net/ipv4/neigh/$name --prefix=/net/ipv6/conf/$name --prefix=/net/ipv6/neigh/$name'
[   18.829532] localhost systemd-udevd[479]: eth0: Network interface 2 is renamed from 'eth0' to 'ens3'
[   18.829547] localhost systemd-udevd[479]: eth0: sd-device: Created db file '/run/udev/data/n2' for '/devices/pci0000:00/0000:00:03.0/virtio0/net/ens3'
[   18.829562] localhost systemd-udevd[479]: eth0: Running command "/sbin/initqueue --name ifup-eth0 --unique --onetime /sbin/ifup eth0"
[   18.829577] localhost systemd-udevd[479]: eth0: Starting '/sbin/initqueue --name ifup-eth0 --unique --onetime /sbin/ifup eth0'
--<

So, we have a race condition here.
Investigating why....
Comment 6 Alberto Planas Dominguez 2020-04-23 13:25:56 UTC
I found that too, but I was a bit confusing from where this message comes from. 

For example, in the attached log we can see the "ip a s" is already showing the udev names (ens3), later or we see the calls to ifup to eth0, and a bit later the virtio rename from eth0 to ens3.

systemd has been recently updated, can be related?
Comment 7 Thomas Blume 2020-04-23 14:01:12 UTC
(In reply to Alberto Planas Dominguez from comment #6)
> I found that too, but I was a bit confusing from where this message comes
> from. 
> 
> For example, in the attached log we can see the "ip a s" is already showing
> the udev names (ens3), later or we see the calls to ifup to eth0, and a bit
> later the virtio rename from eth0 to ens3.

Yeah, it is confusing, systemd-udevd seems to recognize the device quite in time, but ens3 only gets plugged about 1 second later:

-->
[   19.766868] localhost systemd-udevd[479]: ens3: sd-device: Created db file '/run/udev/data/n2' for '/devices/pci0000:00/0000:00:03.0/virtio0/net/ens3'
[   19.766883] localhost systemd-udevd[479]: ens3: Device (SEQNUM=2144, ACTION=move) processed
[   19.766900] localhost systemd[1]: sys-subsystem-net-devices-ens3.device: Changed dead -> plugged
[   19.766918] localhost systemd[1]: sys-devices-pci0000:00-0000:00:03.0-virtio0-net-ens3.device: Changed dead -> plugged
--<

> systemd has been recently updated, can be related?

Yes, the policy, mentioned in the logs from comment#5, could play a role, but I need to look this up.
Comment 8 Thomas Blume 2020-04-29 08:24:27 UTC
Issue is fixed in dracut-0.50 from the devel repo (Base:System/dracut):


-->
Apr 29 10:18:36 localhost systemd-udevd[503]: eth0: /usr/lib/udev/rules.d/80-net-setup-link.rules:11 NAME 'ens3'
Apr 29 10:18:36 localhost systemd-udevd[503]: eth0: /etc/udev/rules.d/91-default-net.rules:1 RUN '/sbin/initqueue --name ifup-$name --unique --onetime /sbin/ifup $name'
Apr 29 10:18:36 localhost systemd-udevd[503]: eth0: /usr/lib/udev/rules.d/99-systemd.rules:62 RUN '/usr/lib/systemd/systemd-sysctl --prefix=/net/ipv4/conf/$name --prefix=/net/ipv4/neigh/$na>
Apr 29 10:18:36 localhost systemd-udevd[503]: eth0: Network interface 2 is renamed from 'eth0' to 'ens3'
Apr 29 10:18:36 localhost systemd-udevd[503]: eth0: sd-device: Created db file '/run/udev/data/n2' for '/devices/pci0000:00/0000:00:03.0/virtio0/net/ens3'
Apr 29 10:18:36 localhost systemd-udevd[503]: eth0: Running command "/sbin/initqueue --name ifup-ens3 --unique --onetime /sbin/ifup ens3"
Apr 29 10:18:36 localhost systemd-udevd[503]: eth0: Starting '/sbin/initqueue --name ifup-ens3 --unique --onetime /sbin/ifup ens3'
--<

The upstream commit therefore is:

-->
commit a8ba1c4e25051c3c482bb7a6a754c9d785463917
Author: Harald Hoyer <harald@redhat.com>
Date:   Wed Feb 26 16:53:47 2020 +0100

    network-legacy/net-genrules.sh: use $name instead of $env{INTERFACE}
    
    The original behavior of $env{INTERFACE} was undocumented and changed in
    the recent udev versions, breaking the ability to bring up networking
    reliably. Switching to $name directive should fix this issue.
    
    Related links:
     - systemd/systemd#12700 (udev PR)
     - systemd/systemd#12291 (related udev issue)
     - systemd/systemd#14935 (this issue, udev side)
     - #732 (this issue, dracut side)
    
    Fixes: #732
--<

Alberto, can you please confirm that it works for you with dracut-0.50 from Base:System?
Comment 9 Alberto Planas Dominguez 2020-04-29 14:33:21 UTC
> Alberto, can you please confirm that it works for you with dracut-0.50 from
> Base:System?

Almost yes!!

The main problem is out. There is still a network problem on ARM (but funny enough not in x86_64) when there are two interfaces. Only one gets activated, and not the second one. I can manually ifup the second interface via the emergency shell.

I propose to close this bug and open another one for this case.

Thanks!
Comment 10 Alberto Planas Dominguez 2020-04-29 14:45:27 UTC
FYI: bsc#1170842