Bug 1171224 - Older kernel-devel packages are not properly purged
Older kernel-devel packages are not properly purged
Status: RESOLVED DUPLICATE of bug 1173106
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: libzypp
Current
x86-64 Other
: P5 - None : Normal (vote)
: ---
Assigned To: E-mail List
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2020-05-05 18:43 UTC by Neil Rickert
Modified: 2020-07-28 16:13 UTC (History)
4 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
Typescript from "zypper purge-kernels" (4.15 KB, text/plain)
2020-06-10 19:34 UTC, Neil Rickert
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Neil Rickert 2020-05-05 18:43:30 UTC
User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Build Identifier: 

Note: I'm setting component to "base-system", but that's possibly wrong.  And I'll note that I am also seeing this on Leap 15.2.

Here's some partial output:
 ---
# zypper se -si kernel
Loading repository data...
Reading installed packages...

S  | Name                        | Type    | Version      | Arch   | Repository
---+-----------------------------+---------+--------------+--------+------------------------
i+ | kernel-default              | package | 5.6.4-1.2    | x86_64 | (System Packages)
i+ | kernel-default              | package | 5.5.11-1.2   | x86_64 | (System Packages)
i+ | kernel-default              | package | 5.4.14-2.1   | x86_64 | (System Packages)
i+ | kernel-default              | package | 5.6.6-1.1    | x86_64 | openSUSE-Tumbleweed-Oss
i  | kernel-default-devel        | package | 5.6.4-1.2    | x86_64 | (System Packages)
i  | kernel-default-devel        | package | 5.5.11-1.2   | x86_64 | (System Packages)
i  | kernel-default-devel        | package | 5.4.14-2.1   | x86_64 | (System Packages)
i  | kernel-default-devel        | package | 5.6.6-1.1    | x86_64 | openSUSE-Tumbleweed-Oss
i+ | kernel-devel                | package | 5.6.4-1.2    | noarch | (System Packages)
i+ | kernel-devel                | package | 5.6.4-1.1    | noarch | (System Packages)
i+ | kernel-devel                | package | 5.6.0-1.1    | noarch | (System Packages)
i+ | kernel-devel                | package | 5.5.11-1.2   | noarch | (System Packages)
i+ | kernel-devel                | package | 5.5.7-1.1    | noarch | (System Packages)
i+ | kernel-devel                | package | 5.4.14-2.1   | noarch | (System Packages)
i+ | kernel-devel                | package | 5.6.6-1.1    | noarch | openSUSE-Tumbleweed-Oss
...
  ---

The retained "kernel-default" and "kernel-default-devel" are correct according to my settings in "zypp.conf" (I'm currently retaining "oldest" and "oldest+1" on that system).

The problem is that there are addition "kernel-devel" versions that should not be there.

Let's look at the first of those, kernel-devel 5.6.4-1.1.

The corresponding kernel-default 5.6.4-1.1 was removed by "zypper dup" at the same time as it installed kernel-devel 5.6.4-1.2.  And, as best I recall, it was the same for kernel-default-devel.  But that didn't happen with kernel-default.

It looks to me as if all of the extraneous kernel-default packages are of this form -- packages where there was a later recompile.  I checked this extensively for Leap 15.2, but only casually for Tumbleweed (using "/var/log/zypp/history").

Either this is a bug in distribution release procedures, or it is a bug in "zypper purge-kernels".

Reproducible: Always
Comment 1 Michal Suchanek 2020-05-06 04:41:07 UTC
Indeed, the old script recognizes 'flavor' source (and source-rt, source-vanilla), and the -devel packages as part of the corresponding source flavor (s/devel/source/). Looks like zypper does not do this yet.

FTR: there is logic to not strip dashes from source flavors so that source-rt and devel-rt are correctly assigned to separate flavor from default source but there is logic to strip -base from kernel-default-base and similar to get these under the same flavor as kernel-default.
Comment 2 Andrei Borzenkov 2020-05-06 05:46:05 UTC
There are several related problems.

1. When new kernel is pure package rebuild zypper *replaces* binary RPMs kernel-flavor and kernel-flavor-devel (effectively doing rpm -U instead of rpm -i), but it installs two versions of kernel-devel in parallel, This is allowed because they contain identical files (part of kernel source tree):

2020-04-18 09:18:38|install|kernel-macros|5.6.2-1.3|noarch||openSUSE-20170729-0|d154e4dde32f2d20313f642f5f5dfa92c999cf7d2db1d04fb6d72642ef960219|
2020-04-18 09:23:56|install|kernel-devel|5.6.2-1.3|noarch||openSUSE-20170729-0|fe7c9c277dd3cba8086b9db8c609b3adc2014d184a44a2f5d051ec9088cda08b|
2020-04-18 09:24:02|remove |kernel-default-devel|5.6.2-1.2|x86_64||
2020-04-18 09:26:36|remove |kernel-default|5.6.2-1.2|x86_64||
2020-04-18 09:27:48|install|kernel-default-devel|5.6.2-1.3|x86_64||openSUSE-20170729-0|a687a89d7bbdf83aee2b6a478a128a578a5aeba080be0447aefd11be1380bba3|
# 2020-04-18 09:29:19 kernel-default-5.6.2-1.3.x86_64.rpm installed ok

This now results in two packages present on the system - kernel-devel-5.6.2-1.2 and kernel-devel 5.6.2-1.3, but only one kernel-default-5.6.2-1.3.

2. zypper purge-kernels only removes kernel-devel for which it also removes exact version of kernel-flavor. This makes zypper skip 5.6.2-1.2 in this case.

2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):298 Found installed kernel (689)kernel-default-5.6.2-1.3.x86_64(@System)
2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):298 Found installed kernel (690)kernel-default-5.6.4-1.2.x86_64(@System)
2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels] PurgeKernels.cc(removePackageAndCheck):109 Removing package: I_Tu_u(689)kernel-default-5.6.2-1.3.x86_64(@System)
2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels] PurgeKernels.cc(removePackageAndCheck):109 Removing package: I_Tu_(693)kernel-default-devel-5.6.2-1.3.x86_64(@System)
2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels] PurgeKernels.cc(cleanDevelAndSrcPackages):232 Skipping source package (699)kernel-devel-5.6.2-1.2.noarch(@System) no corresponding kernel with the same version was installed.
2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels] PurgeKernels.cc(removePackageAndCheck):109 Removing package: I_Tu_(700)kernel-devel-5.6.2-1.3.noarch(@System)


Both are questionable. In particular, old purge-kernels script removed all kernel-devel (and kernel-source) for which no corresponding binary kernel packages were present on system. So this is certainly behavior change. Fixing 1 would mitigate the problem mostly, but still will not clean up system in case packages were removed outside of "zypper purge-kernels".
Comment 3 Benjamin Zeller 2020-05-06 07:00:23 UTC
(In reply to Andrei Borzenkov from comment #2)
> There are several related problems.
> 
> 1. When new kernel is pure package rebuild zypper *replaces* binary RPMs
> kernel-flavor and kernel-flavor-devel (effectively doing rpm -U instead of
> rpm -i), but it installs two versions of kernel-devel in parallel, This is
> allowed because they contain identical files (part of kernel source tree):

AFAIK zypper does not update the running kernel, it installs the new one side by side. Thats why we need the purge-kernel feature to clean up at the next boot. 
So rpm -U is not used. 

> 
> 2020-04-18
> 09:18:38|install|kernel-macros|5.6.2-1.3|noarch||openSUSE-20170729-
> 0|d154e4dde32f2d20313f642f5f5dfa92c999cf7d2db1d04fb6d72642ef960219|
> 2020-04-18
> 09:23:56|install|kernel-devel|5.6.2-1.3|noarch||openSUSE-20170729-
> 0|fe7c9c277dd3cba8086b9db8c609b3adc2014d184a44a2f5d051ec9088cda08b|
> 2020-04-18 09:24:02|remove |kernel-default-devel|5.6.2-1.2|x86_64||
> 2020-04-18 09:26:36|remove |kernel-default|5.6.2-1.2|x86_64||
> 2020-04-18
> 09:27:48|install|kernel-default-devel|5.6.2-1.3|x86_64||openSUSE-20170729-
> 0|a687a89d7bbdf83aee2b6a478a128a578a5aeba080be0447aefd11be1380bba3|
> # 2020-04-18 09:29:19 kernel-default-5.6.2-1.3.x86_64.rpm installed ok
> 
> This now results in two packages present on the system -
> kernel-devel-5.6.2-1.2 and kernel-devel 5.6.2-1.3, but only one
> kernel-default-5.6.2-1.3.

zypper purge-kernels does not update/install packages this is part of the normal update process and was not changed for this feature. 

> 
> 2. zypper purge-kernels only removes kernel-devel for which it also removes
> exact version of kernel-flavor. This makes zypper skip 5.6.2-1.2 in this
> case.
> 
> 2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels]
> PurgeKernels.cc(markObsoleteKernels):298 Found installed kernel
> (689)kernel-default-5.6.2-1.3.x86_64(@System)
> 2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels]
> PurgeKernels.cc(markObsoleteKernels):298 Found installed kernel
> (690)kernel-default-5.6.4-1.2.x86_64(@System)
> 2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels]
> PurgeKernels.cc(removePackageAndCheck):109 Removing package:
> I_Tu_u(689)kernel-default-5.6.2-1.3.x86_64(@System)
> 2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels]
> PurgeKernels.cc(removePackageAndCheck):109 Removing package:
> I_Tu_(693)kernel-default-devel-5.6.2-1.3.x86_64(@System)
> 2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels]
> PurgeKernels.cc(cleanDevelAndSrcPackages):232 Skipping source package
> (699)kernel-devel-5.6.2-1.2.noarch(@System) no corresponding kernel with the
> same version was installed.
> 2020-05-05 17:34:19 <1> tw.0.2.15(935) [PurgeKernels]
> PurgeKernels.cc(removePackageAndCheck):109 Removing package:
> I_Tu_(700)kernel-devel-5.6.2-1.3.noarch(@System)
> 
> 
> Both are questionable. In particular, old purge-kernels script removed all
> kernel-devel (and kernel-source) for which no corresponding binary kernel
> packages were present on system. So this is certainly behavior change.
> Fixing 1 would mitigate the problem mostly, but still will not clean up
> system in case packages were removed outside of "zypper purge-kernels".

Currently the code removes only devel packages it had a kernel installed for, simply for the reason that we do not want to clean out devel packages that a user has installed to work with them.
Comment 4 Andrei Borzenkov 2020-05-06 07:32:43 UTC
(In reply to Benjamin Zeller from comment #3)
> 
> AFAIK zypper does not update the running kernel, it installs the new one
> side by side.

You are aware of what "package rebuild" is? It cannot be installed side-by-side because it contains the same files that overlap each other.

> Thats why we need the purge-kernel feature to clean up at the next boot. 

Zypper removes previous kernel binary package that differs only in second digit release number.

> So rpm -U is not used. 
> 

I said "effectively". rpm -U installs new package version and removes old package version. zypper removes old version and installs new version. Big difference indeed.

> > 2020-04-18 09:24:02|remove |kernel-default-devel|5.6.2-1.2|x86_64||
> > 2020-04-18 09:26:36|remove |kernel-default|5.6.2-1.2|x86_64||
> > 2020-04-18 09:27:48|install|kernel-default-devel|5.6.2-1.3|x86_64||openSUSE-20170729-0|a687a89d7bbdf83aee2b6a478a128a578a5aeba080be0447aefd11be1380bba3|
> > # 2020-04-18 09:29:19 kernel-default-5.6.2-1.3.x86_64.rpm installed ok
> > 
> > This now results in two packages present on the system -
> > kernel-devel-5.6.2-1.2 and kernel-devel 5.6.2-1.3, but only one
> > kernel-default-5.6.2-1.3.
> 
> zypper purge-kernels does not update/install packages this is part of the
> normal update process and was not changed for this feature. 
> 

Once more - rpm "upgrade" does install followed by remove. After "upgrade" new version is present and old version not. That is *exactly* what happens after zypper in kernel-default or zyper in kerne-default-devel for package rebuilds.

> > Both are questionable. In particular, old purge-kernels script removed all
> > kernel-devel (and kernel-source) for which no corresponding binary kernel
> > packages were present on system. So this is certainly behavior change.
> > Fixing 1 would mitigate the problem mostly, but still will not clean up
> > system in case packages were removed outside of "zypper purge-kernels".
> 
> Currently the code removes only devel packages it had a kernel installed
> for, simply for the reason that we do not want to clean out devel packages
> that a user has installed to work with them.

How do you know user is not going to continue to work with devel package even though user removed corresponding kernel binary? By this logic no devel should ever be removed at all.

But at least fix installation of kernel-devel and kernel-source package rebuilds to not leave previous versions behind. You seem to miss the simple fact that two rebuilds of kernel-devel and kernel-source are complete duplicate and contain exactly the same files so having them both makes no sense.
Comment 5 Michael Andres 2020-05-11 12:34:27 UTC
(In reply to Andrei Borzenkov from comment #4)
> (In reply to Benjamin Zeller from comment #3)
> > 
> > AFAIK zypper does not update the running kernel, it installs the new one
> > side by side.
> 
> You are aware of what "package rebuild" is? It cannot be installed
> side-by-side because it contains the same files that overlap each other.

But you know that this is in the first place the packages business.

The package itself indicates whether it want's to be installed by 'rpm -U' or by 'rpm -i' (aka 'multiversion' mode). So overlapping content in different releases of the packages have to be managed by the package, not by the installer.

The remove and install action shown in your history snippet are driven by the packages dependencies. There are no special rules for kernel or any other package here.

Only the purge-kernel command (unfortunately) needs to have some knowledge about the structure of the kernel packages. One reason for this IMO comes from the fact, that it's debatable whether packages like kernel-devel or -source should actually be 'multiversion' packages. Same issue with kmps.

As they need to follow a specific kernel version, they could also be created as a set of individual packages carrying the kernel version they are intended for in their name ('kernel-devel-for_5.6.2'). This way they could be installed, updated, obsoleted and removes lika any other package and without interfering with other kernel-* packages for other kernel version. But that's the packagers issue.


Anyway, we'll fix purge-kernel to cleanup orphaned kernel-* packages.
Comment 6 Andrei Borzenkov 2020-05-26 05:23:34 UTC
(In reply to Michael Andres from comment #5)
> 
> The remove and install action shown in your history snippet are driven by
> the packages dependencies. There are no special rules for kernel or any
> other package here.
> 

You are right. kernel binary packages include Obsoletes for previous package rebuilds which forces zypper to remove them. These Obsoletes are not present in kernel-devel or kernel-source.

https://bugzilla.opensuse.org/show_bug.cgi?id=1172073
Comment 7 Benjamin Zeller 2020-05-26 13:10:21 UTC
https://github.com/openSUSE/libzypp/pull/222
Comment 8 Benjamin Zeller 2020-05-27 09:17:31 UTC
Will be fixed with libzypp >= 17.23.5
Comment 9 Neil Rickert 2020-06-10 19:34:50 UTC
Created attachment 838681 [details]
Typescript from "zypper purge-kernels"

That update to libzypp has arrived in Leap 15.2.  So I decided to give it a test.

After the update, which also included a newer kernel, I removed "/boot/do_purge_kernels" before reboot.  That way, I could run "zypper purge-kernels" manually and see what happens.

I have attached a typescript of that.  The typescript also include the output of "zypper se -si kernel-d" before the purge-kernels.

And "zypper purge-kernels" wants to remove all kernel devel packages, including the ones for the newly installed kernel.  That surely cannot be right.

I'm reopening this bug for checking on that.
Comment 10 Neil Rickert 2020-06-10 19:36:53 UTC
I guess it doesn't allow me to reopen on adding attachment.  Second try at reopen.
Comment 13 Michael Andres 2020-06-26 17:05:42 UTC
Let me joint this and bug#1173106. It's not exactly the same issue, but both are related to the current purge kernel impementation. As we're going to do a complete review (and probably rewrite), it makes sense to collect all issues at one place.

The resulting fix will then be distributed to TW and CODE.15 products.

*** This bug has been marked as a duplicate of bug 1173106 ***
Comment 14 Swamp Workflow Management 2020-07-07 16:16:40 UTC
SUSE-RU-2020:1869-1: An update that has 13 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1130873,1154803,1164543,1165476,1165573,1166610,1167122,1168990,1169947,1170801,1171224,1172135,1172925
CVE References: 
Sources used:
SUSE Linux Enterprise Server for SAP 15 (src):    libsigc++2-2.10.0-3.5.1, libsolv-0.7.14-3.30.2, libyui-ncurses-pkg-2.48.5.2-3.7.12, libyui-ncurses-pkg-doc-2.48.5.2-3.7.12, libyui-qt-pkg-2.45.15.2-3.7.13, libyui-qt-pkg-doc-2.45.15.2-3.7.12, libzypp-17.23.8-3.43.1, yast2-pkg-bindings-4.0.13-3.9.12, zypper-1.14.37-3.34.6
SUSE Linux Enterprise Server 15-LTSS (src):    libsigc++2-2.10.0-3.5.1, libsolv-0.7.14-3.30.2, libyui-ncurses-pkg-2.48.5.2-3.7.12, libyui-ncurses-pkg-doc-2.48.5.2-3.7.12, libyui-qt-pkg-2.45.15.2-3.7.13, libyui-qt-pkg-doc-2.45.15.2-3.7.12, libzypp-17.23.8-3.43.1, yast2-pkg-bindings-4.0.13-3.9.12, zypper-1.14.37-3.34.6
SUSE Linux Enterprise Module for Desktop Applications 15-SP1 (src):    libsigc++2-2.10.0-3.5.1
SUSE Linux Enterprise Module for Basesystem 15-SP1 (src):    libsigc++2-2.10.0-3.5.1
SUSE Linux Enterprise Installer 15 (src):    libsigc++2-2.10.0-3.5.1, libsolv-0.7.14-3.30.2, libyui-ncurses-pkg-2.48.5.2-3.7.12, libyui-qt-pkg-2.45.15.2-3.7.13, libzypp-17.23.8-3.43.1, yast2-pkg-bindings-4.0.13-3.9.12, zypper-1.14.37-3.34.6
SUSE Linux Enterprise High Performance Computing 15-LTSS (src):    libsigc++2-2.10.0-3.5.1, libsolv-0.7.14-3.30.2, libyui-ncurses-pkg-2.48.5.2-3.7.12, libyui-ncurses-pkg-doc-2.48.5.2-3.7.12, libyui-qt-pkg-2.45.15.2-3.7.13, libyui-qt-pkg-doc-2.45.15.2-3.7.12, libzypp-17.23.8-3.43.1, yast2-pkg-bindings-4.0.13-3.9.12, zypper-1.14.37-3.34.6
SUSE Linux Enterprise High Performance Computing 15-ESPOS (src):    libsigc++2-2.10.0-3.5.1, libsolv-0.7.14-3.30.2, libyui-ncurses-pkg-2.48.5.2-3.7.12, libyui-ncurses-pkg-doc-2.48.5.2-3.7.12, libyui-qt-pkg-2.45.15.2-3.7.13, libyui-qt-pkg-doc-2.45.15.2-3.7.12, libzypp-17.23.8-3.43.1, yast2-pkg-bindings-4.0.13-3.9.12, zypper-1.14.37-3.34.6

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 17 Swamp Workflow Management 2020-07-24 16:25:29 UTC
SUSE-RU-2020:2040-1: An update that has 5 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1170801,1171224,1172135,1173106,1174011
CVE References: 
Sources used:
SUSE Linux Enterprise Module for Packagehub Subpackages 15-SP1 (src):    libsolv-0.7.14-3.22.2
SUSE Linux Enterprise Module for Development Tools 15-SP1 (src):    libsolv-0.7.14-3.22.2
SUSE Linux Enterprise Module for Basesystem 15-SP1 (src):    libsolv-0.7.14-3.22.2, libzypp-17.24.1-3.28.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 18 Swamp Workflow Management 2020-07-28 16:13:05 UTC
openSUSE-RU-2020:1109-1: An update that has 5 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1170801,1171224,1172135,1173106,1174011
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.1 (src):    libsolv-0.7.14-lp151.2.19.1, libzypp-17.24.1-lp151.2.19.1