Bug 1171224

Indeed, the old script recognizes 'flavor' source (and source-rt, source-vanilla), and the -devel packages as part of the corresponding source flavor (s/devel/source/). Looks like zypper does not do this yet.

FTR: there is logic to not strip dashes from source flavors so that source-rt and devel-rt are correctly assigned to separate flavor from default source but there is logic to strip -base from kernel-default-base and similar to get these under the same flavor as kernel-default.

There are several related problems.

1. When new kernel is pure package rebuild zypper *replaces* binary RPMs kernel-flavor and kernel-flavor-devel (effectively doing rpm -U instead of rpm -i), but it installs two versions of kernel-devel in parallel, This is allowed because they contain identical files (part of kernel source tree):

2020-04-18 09:18:38|install|kernel-macros|5.6.2-1.3|noarch||openSUSE-20170729-0|d154e4dde32f2d20313f642f5f5dfa92c999cf7d2db1d04fb6d72642ef960219|
2020-04-18 09:23:56|install|kernel-devel|5.6.2-1.3|noarch||openSUSE-20170729-0|fe7c9c277dd3cba8086b9db8c609b3adc2014d184a44a2f5d051ec9088cda08b|
2020-04-18 09:24:02|remove |kernel-default-devel|5.6.2-1.2|x86_64||
2020-04-18 09:26:36|remove |kernel-default|5.6.2-1.2|x86_64||
2020-04-18 09:27:48|install|kernel-default-devel|5.6.2-1.3|x86_64||openSUSE-20170729-0|a687a89d7bbdf83aee2b6a478a128a578a5aeba080be0447aefd11be1380bba3|
# 2020-04-18 09:29:19 kernel-default-5.6.2-1.3.x86_64.rpm installed ok

This now results in two packages present on the system - kernel-devel-5.6.2-1.2 and kernel-devel 5.6.2-1.3, but only one kernel-default-5.6.2-1.3.

2. zypper purge-kernels only removes kernel-devel for which it also removes exact version of kernel-flavor. This makes zypper skip 5.6.2-1.2 in this case.

2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):298 Found installed kernel (689)kernel-default-5.6.2-1.3.x86_64(@System)
2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):298 Found installed kernel (690)kernel-default-5.6.4-1.2.x86_64(@System)
2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels] PurgeKernels.cc(removePackageAndCheck):109 Removing package: I_Tu_u(689)kernel-default-5.6.2-1.3.x86_64(@System)
2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels] PurgeKernels.cc(removePackageAndCheck):109 Removing package: I_Tu_(693)kernel-default-devel-5.6.2-1.3.x86_64(@System)
2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels] PurgeKernels.cc(cleanDevelAndSrcPackages):232 Skipping source package (699)kernel-devel-5.6.2-1.2.noarch(@System) no corresponding kernel with the same version was installed.
2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels] PurgeKernels.cc(removePackageAndCheck):109 Removing package: I_Tu_(700)kernel-devel-5.6.2-1.3.noarch(@System)


Both are questionable. In particular, old purge-kernels script removed all kernel-devel (and kernel-source) for which no corresponding binary kernel packages were present on system. So this is certainly behavior change. Fixing 1 would mitigate the problem mostly, but still will not clean up system in case packages were removed outside of "zypper purge-kernels".

(In reply to Andrei Borzenkov from comment #2)
> There are several related problems.
> 
> 1. When new kernel is pure package rebuild zypper *replaces* binary RPMs
> kernel-flavor and kernel-flavor-devel (effectively doing rpm -U instead of
> rpm -i), but it installs two versions of kernel-devel in parallel, This is
> allowed because they contain identical files (part of kernel source tree):

AFAIK zypper does not update the running kernel, it installs the new one side by side. Thats why we need the purge-kernel feature to clean up at the next boot. 
So rpm -U is not used. 

> 
> 2020-04-18
> 09:18:38|install|kernel-macros|5.6.2-1.3|noarch||openSUSE-20170729-
> 0|d154e4dde32f2d20313f642f5f5dfa92c999cf7d2db1d04fb6d72642ef960219|
> 2020-04-18
> 09:23:56|install|kernel-devel|5.6.2-1.3|noarch||openSUSE-20170729-
> 0|fe7c9c277dd3cba8086b9db8c609b3adc2014d184a44a2f5d051ec9088cda08b|
> 2020-04-18 09:24:02|remove |kernel-default-devel|5.6.2-1.2|x86_64||
> 2020-04-18 09:26:36|remove |kernel-default|5.6.2-1.2|x86_64||
> 2020-04-18
> 09:27:48|install|kernel-default-devel|5.6.2-1.3|x86_64||openSUSE-20170729-
> 0|a687a89d7bbdf83aee2b6a478a128a578a5aeba080be0447aefd11be1380bba3|
> # 2020-04-18 09:29:19 kernel-default-5.6.2-1.3.x86_64.rpm installed ok
> 
> This now results in two packages present on the system -
> kernel-devel-5.6.2-1.2 and kernel-devel 5.6.2-1.3, but only one
> kernel-default-5.6.2-1.3.

zypper purge-kernels does not update/install packages this is part of the normal update process and was not changed for this feature. 

> 
> 2. zypper purge-kernels only removes kernel-devel for which it also removes
> exact version of kernel-flavor. This makes zypper skip 5.6.2-1.2 in this
> case.
> 
> 2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels]
> PurgeKernels.cc(markObsoleteKernels):298 Found installed kernel
> (689)kernel-default-5.6.2-1.3.x86_64(@System)
> 2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels]
> PurgeKernels.cc(markObsoleteKernels):298 Found installed kernel
> (690)kernel-default-5.6.4-1.2.x86_64(@System)
> 2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels]
> PurgeKernels.cc(removePackageAndCheck):109 Removing package:
> I_Tu_u(689)kernel-default-5.6.2-1.3.x86_64(@System)
> 2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels]
> PurgeKernels.cc(removePackageAndCheck):109 Removing package:
> I_Tu_(693)kernel-default-devel-5.6.2-1.3.x86_64(@System)
> 2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels]
> PurgeKernels.cc(cleanDevelAndSrcPackages):232 Skipping source package
> (699)kernel-devel-5.6.2-1.2.noarch(@System) no corresponding kernel with the
> same version was installed.
> 2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels]
> PurgeKernels.cc(removePackageAndCheck):109 Removing package:
> I_Tu_(700)kernel-devel-5.6.2-1.3.noarch(@System)
> 
> 
> Both are questionable. In particular, old purge-kernels script removed all
> kernel-devel (and kernel-source) for which no corresponding binary kernel
> packages were present on system. So this is certainly behavior change.
> Fixing 1 would mitigate the problem mostly, but still will not clean up
> system in case packages were removed outside of "zypper purge-kernels".

Currently the code removes only devel packages it had a kernel installed for, simply for the reason that we do not want to clean out devel packages that a user has installed to work with them.

(In reply to Benjamin Zeller from comment #3)
> 
> AFAIK zypper does not update the running kernel, it installs the new one
> side by side.

You are aware of what "package rebuild" is? It cannot be installed side-by-side because it contains the same files that overlap each other.

> Thats why we need the purge-kernel feature to clean up at the next boot. 

Zypper removes previous kernel binary package that differs only in second digit release number.

> So rpm -U is not used. 
> 

I said "effectively". rpm -U installs new package version and removes old package version. zypper removes old version and installs new version. Big difference indeed.

> > 2020-04-18 09:24:02|remove |kernel-default-devel|5.6.2-1.2|x86_64||
> > 2020-04-18 09:26:36|remove |kernel-default|5.6.2-1.2|x86_64||
> > 2020-04-18 09:27:48|install|kernel-default-devel|5.6.2-1.3|x86_64||openSUSE-20170729-0|a687a89d7bbdf83aee2b6a478a128a578a5aeba080be0447aefd11be1380bba3|
> > # 2020-04-18 09:29:19 kernel-default-5.6.2-1.3.x86_64.rpm installed ok
> > 
> > This now results in two packages present on the system -
> > kernel-devel-5.6.2-1.2 and kernel-devel 5.6.2-1.3, but only one
> > kernel-default-5.6.2-1.3.
> 
> zypper purge-kernels does not update/install packages this is part of the
> normal update process and was not changed for this feature. 
> 

Once more - rpm "upgrade" does install followed by remove. After "upgrade" new version is present and old version not. That is *exactly* what happens after zypper in kernel-default or zyper in kerne-default-devel for package rebuilds.

> > Both are questionable. In particular, old purge-kernels script removed all
> > kernel-devel (and kernel-source) for which no corresponding binary kernel
> > packages were present on system. So this is certainly behavior change.
> > Fixing 1 would mitigate the problem mostly, but still will not clean up
> > system in case packages were removed outside of "zypper purge-kernels".
> 
> Currently the code removes only devel packages it had a kernel installed
> for, simply for the reason that we do not want to clean out devel packages
> that a user has installed to work with them.

How do you know user is not going to continue to work with devel package even though user removed corresponding kernel binary? By this logic no devel should ever be removed at all.

But at least fix installation of kernel-devel and kernel-source package rebuilds to not leave previous versions behind. You seem to miss the simple fact that two rebuilds of kernel-devel and kernel-source are complete duplicate and contain exactly the same files so having them both makes no sense.

(In reply to Andrei Borzenkov from comment #4)
> (In reply to Benjamin Zeller from comment #3)
> > 
> > AFAIK zypper does not update the running kernel, it installs the new one
> > side by side.
> 
> You are aware of what "package rebuild" is? It cannot be installed
> side-by-side because it contains the same files that overlap each other.

But you know that this is in the first place the packages business.

The package itself indicates whether it want's to be installed by 'rpm -U' or by 'rpm -i' (aka 'multiversion' mode). So overlapping content in different releases of the packages have to be managed by the package, not by the installer.

The remove and install action shown in your history snippet are driven by the packages dependencies. There are no special rules for kernel or any other package here.

Only the purge-kernel command (unfortunately) needs to have some knowledge about the structure of the kernel packages. One reason for this IMO comes from the fact, that it's debatable whether packages like kernel-devel or -source should actually be 'multiversion' packages. Same issue with kmps.

As they need to follow a specific kernel version, they could also be created as a set of individual packages carrying the kernel version they are intended for in their name ('kernel-devel-for_5.6.2'). This way they could be installed, updated, obsoleted and removes lika any other package and without interfering with other kernel-* packages for other kernel version. But that's the packagers issue.


Anyway, we'll fix purge-kernel to cleanup orphaned kernel-* packages.

(In reply to Michael Andres from comment #5)
> 
> The remove and install action shown in your history snippet are driven by
> the packages dependencies. There are no special rules for kernel or any
> other package here.
> 

You are right. kernel binary packages include Obsoletes for previous package rebuilds which forces zypper to remove them. These Obsoletes are not present in kernel-devel or kernel-source.

https://bugzilla.opensuse.org/show_bug.cgi?id=1172073

https://github.com/openSUSE/libzypp/pull/222

Will be fixed with libzypp >= 17.23.5

Created attachment 838681 [details]
Typescript from "zypper purge-kernels"

That update to libzypp has arrived in Leap 15.2.  So I decided to give it a test.

After the update, which also included a newer kernel, I removed "/boot/do_purge_kernels" before reboot.  That way, I could run "zypper purge-kernels" manually and see what happens.

I have attached a typescript of that.  The typescript also include the output of "zypper se -si kernel-d" before the purge-kernels.

And "zypper purge-kernels" wants to remove all kernel devel packages, including the ones for the newly installed kernel.  That surely cannot be right.

I'm reopening this bug for checking on that.

I guess it doesn't allow me to reopen on adding attachment.  Second try at reopen.

Let me joint this and bug#1173106. It's not exactly the same issue, but both are related to the current purge kernel impementation. As we're going to do a complete review (and probably rewrite), it makes sense to collect all issues at one place.

The resulting fix will then be distributed to TW and CODE.15 products.

*** This bug has been marked as a duplicate of bug 1173106 ***

SUSE-RU-2020:1869-1: An update that has 13 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1130873,1154803,1164543,1165476,1165573,1166610,1167122,1168990,1169947,1170801,1171224,1172135,1172925
CVE References: 
Sources used:
SUSE Linux Enterprise Server for SAP 15 (src):    libsigc++2-2.10.0-3.5.1, libsolv-0.7.14-3.30.2, libyui-ncurses-pkg-2.48.5.2-3.7.12, libyui-ncurses-pkg-doc-2.48.5.2-3.7.12, libyui-qt-pkg-2.45.15.2-3.7.13, libyui-qt-pkg-doc-2.45.15.2-3.7.12, libzypp-17.23.8-3.43.1, yast2-pkg-bindings-4.0.13-3.9.12, zypper-1.14.37-3.34.6
SUSE Linux Enterprise Server 15-LTSS (src):    libsigc++2-2.10.0-3.5.1, libsolv-0.7.14-3.30.2, libyui-ncurses-pkg-2.48.5.2-3.7.12, libyui-ncurses-pkg-doc-2.48.5.2-3.7.12, libyui-qt-pkg-2.45.15.2-3.7.13, libyui-qt-pkg-doc-2.45.15.2-3.7.12, libzypp-17.23.8-3.43.1, yast2-pkg-bindings-4.0.13-3.9.12, zypper-1.14.37-3.34.6
SUSE Linux Enterprise Module for Desktop Applications 15-SP1 (src):    libsigc++2-2.10.0-3.5.1
SUSE Linux Enterprise Module for Basesystem 15-SP1 (src):    libsigc++2-2.10.0-3.5.1
SUSE Linux Enterprise Installer 15 (src):    libsigc++2-2.10.0-3.5.1, libsolv-0.7.14-3.30.2, libyui-ncurses-pkg-2.48.5.2-3.7.12, libyui-qt-pkg-2.45.15.2-3.7.13, libzypp-17.23.8-3.43.1, yast2-pkg-bindings-4.0.13-3.9.12, zypper-1.14.37-3.34.6
SUSE Linux Enterprise High Performance Computing 15-LTSS (src):    libsigc++2-2.10.0-3.5.1, libsolv-0.7.14-3.30.2, libyui-ncurses-pkg-2.48.5.2-3.7.12, libyui-ncurses-pkg-doc-2.48.5.2-3.7.12, libyui-qt-pkg-2.45.15.2-3.7.13, libyui-qt-pkg-doc-2.45.15.2-3.7.12, libzypp-17.23.8-3.43.1, yast2-pkg-bindings-4.0.13-3.9.12, zypper-1.14.37-3.34.6
SUSE Linux Enterprise High Performance Computing 15-ESPOS (src):    libsigc++2-2.10.0-3.5.1, libsolv-0.7.14-3.30.2, libyui-ncurses-pkg-2.48.5.2-3.7.12, libyui-ncurses-pkg-doc-2.48.5.2-3.7.12, libyui-qt-pkg-2.45.15.2-3.7.13, libyui-qt-pkg-doc-2.45.15.2-3.7.12, libzypp-17.23.8-3.43.1, yast2-pkg-bindings-4.0.13-3.9.12, zypper-1.14.37-3.34.6

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

SUSE-RU-2020:2040-1: An update that has 5 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1170801,1171224,1172135,1173106,1174011
CVE References: 
Sources used:
SUSE Linux Enterprise Module for Packagehub Subpackages 15-SP1 (src):    libsolv-0.7.14-3.22.2
SUSE Linux Enterprise Module for Development Tools 15-SP1 (src):    libsolv-0.7.14-3.22.2
SUSE Linux Enterprise Module for Basesystem 15-SP1 (src):    libsolv-0.7.14-3.22.2, libzypp-17.24.1-3.28.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.

openSUSE-RU-2020:1109-1: An update that has 5 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1170801,1171224,1172135,1173106,1174011
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.1 (src):    libsolv-0.7.14-lp151.2.19.1, libzypp-17.24.1-lp151.2.19.1