Bug 1193852 - armv7 kernel crashes on lzo1x_1_do_compress+0xe8/0x600
armv7 kernel crashes on lzo1x_1_do_compress+0xe8/0x600
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Distribution
Classification: openSUSE
Component: Kernel
Leap 15.4
armv7 Other
: P2 - High : Major (vote)
: ---
Assigned To: David Sterba
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2021-12-17 08:29 UTC by Guillaume GARDET
Modified: 2022-08-01 13:27 UTC (History)
9 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
full log (983.17 KB, text/plain)
2021-12-17 08:29 UTC, Guillaume GARDET
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Guillaume GARDET 2021-12-17 08:29:30 UTC
Created attachment 854664 [details]
full log

armv7 kernel crashes in OBS while building kiwi images, based on BTRFS, at least on armbuild22 and obs-arm-{5,6} with: 

[  814s] [  777.766838][    T7] Unable to handle kernel NULL pointer dereference at virtual address 00000005
[  814s] [  777.766847][    T7] pgd = 6c052b6c
[  814s] [  777.766856][    T7] [00000005] *pgd=80000040304003, *pmd=00000000
[  814s] [  777.771184][T14168] Supported: No, Unreleased kernel
[  814s] [  777.771201][T14168] CPU: 1 PID: 14168 Comm: kworker/u8:4 Tainted: G                 N 5.14.21-2-lpae #1 SLE15-SP4 (unreleased) 1b3defe9159c7f0c570387bff6a0df8b599a4396
[  814s] [  777.771214][T14168] Hardware name: Generic DT based system
[  814s] [  777.771225][T14168] Workqueue: btrfs-delalloc btrfs_work_helper [btrfs]
[  814s] [  777.772140][  T101] Unable to handle kernel NULL pointer dereference at virtual address 00000005
[  814s] [  777.773696][T14168] 
[  814s] [  777.775396][  T101] pgd = 6c052b6c
[  814s] [  777.776756][T14168] PC is at lzo1x_1_do_compress+0xe8/0x600
[  814s] [  777.777904][  T101] [00000005] *pgd=80000040304003
Comment 1 Takashi Iwai 2022-01-03 14:08:15 UTC
Sounds like something to do with btrfs compression.  Adding Goldwyn for distributing to filesystem team.
Comment 2 Dirk Mueller 2022-01-04 13:42:47 UTC
looks like none of the btrfs lzo fixes (including memory leak fixes) have been added to 5.14. likely candidates to fix this are 

commit ccaa66c8dd277ac02f96914168bb7177f7ea8117
Author: David Sterba <dsterba@suse.com>
Date:   Wed Oct 27 10:44:21 2021 +0200

    Revert "btrfs: compression: drop kmap/kunmap from lzo"
    
    This reverts commit 8c945d32e60427cbc0859cf7045bbe6196bb03d8.
    
    The kmaps in compression code are still needed and cause crashes on
    32bit machines (ARM, x86). Reproducible eg. by running fstest btrfs/004
    with enabled LZO or ZSTD compression.


and the followon commits from Linus Torvals and Qu Wenruo
Comment 3 David Sterba 2022-01-10 15:53:08 UTC
Yes the crash on 32bit arches is related to the kmap patches, there are a few plus the fixups. I'll put together list of commits.
Comment 4 Guillaume GARDET 2022-01-26 09:24:15 UTC
Any update here?
Comment 5 David Sterba 2022-01-27 16:23:05 UTC
Patches are in branch users/dsterba/SLE15-SP4/bsc1193852. I don't have 32bit arch to test it, builds on x86_64.
Comment 6 David Sterba 2022-01-31 14:31:17 UTC
I can't find the 32bit setup I had to test the patches, can you please test it on the arm setup?
Comment 7 Guillaume GARDET 2022-01-31 15:29:33 UTC
The crash happens in OBS while building an image with kiwi. See: https://build.opensuse.org/package/live_build_log/openSUSE:Leap:15.4:ARM:Images/JeOS:JeOS-efi-pxe/images/armv7l

To reproduce, you can run a local build on an Arm system:
 osc co openSUSE:Leap:15.4:ARM:Images/JeOS
 cd openSUSE:Leap:15.4:ARM:Images/JeOS
 osc build images armv7l JeOS-efi-pxe.kiwi --vm-type=kvm
Comment 8 David Sterba 2022-02-03 12:48:14 UTC
This has a few unknowns, where does the patched kernel appear in that? Also I don't have any decent ARM machines to run it and trying to find one in Orthos is usually fruitless. So I'll verify that on the 32bit build on intel.
Comment 9 Dirk Mueller 2022-02-03 16:58:02 UTC
I'm testing your kernel fixes in openSUSE:Step:15-SP4 and will let you know.
Comment 10 Dirk Mueller 2022-02-10 22:29:49 UTC
it appears to be working fine. Great work! could you please submit it?
Comment 11 David Sterba 2022-02-15 19:24:49 UTC
Thanks for testing, branch sent for merge.
Comment 12 Guillaume GARDET 2022-03-02 09:02:26 UTC
Any ETA to have a kernel update and drop the overlay?
Comment 13 Dirk Mueller 2022-03-03 08:20:31 UTC
the overlay has already been dropped. I can see the changes in the git tree and in the devel project but not in the official kernel. no idea why?!

-------------------------------------------------------------------
Thu Jan 27 17:05:04 CET 2022 - dsterba@suse.com

- btrfs: fix lzo_decompress_bio() kmap leakage (bsc#1193852).
- Revert "btrfs: compression: drop kmap/kunmap from lzo"
  (bsc#1193852).
- Revert "btrfs: compression: drop kmap/kunmap from zlib"
  (bsc#1193852).
- Revert "btrfs: compression: drop kmap/kunmap from zstd"
  (bsc#1193852).
- Revert "btrfs: compression: drop kmap/kunmap from generic
  helpers" (bsc#1193852).
- commit 43ad82f
Comment 14 Guillaume GARDET 2022-03-14 14:22:30 UTC
(In reply to Dirk Mueller from comment #13)
> the overlay has already been dropped. I can see the changes in the git tree
> and in the devel project but not in the official kernel. no idea why?!

Indeed, the JeOS images are failing again.

Any ETA to get the fixed kernel?
Comment 15 Fabian Vogt 2022-03-14 14:44:18 UTC
(In reply to Guillaume GARDET from comment #14)
> (In reply to Dirk Mueller from comment #13)
> > the overlay has already been dropped. I can see the changes in the git tree
> > and in the devel project but not in the official kernel. no idea why?!
> 
> Indeed, the JeOS images are failing again.

I guess this is SLE15-SP4 vs. SLE15-SP4-GA. Currently the -GA kernel is used, which does not have the fixes merged.

> Any ETA to get the fixed kernel?

Adding needinfo.
Comment 16 David Sterba 2022-03-24 15:31:04 UTC
I don't know what else is needed from me, the patches are in SLE15-SP4 git branch and I can't affect what kernel is installed in what product.
Comment 17 Guillaume GARDET 2022-03-29 07:35:18 UTC
It seems openSUSE:Step:15-SP4/kernel-source now has the required commits.
Comment 19 Takashi Iwai 2022-04-05 09:12:52 UTC
Those patches should be merged to SLE15-SP4-GA kernel.

David, I suppose those are safe to be cherry-picked to GA kernel (for 64bit systems), right?
Comment 20 Takashi Iwai 2022-04-06 09:06:54 UTC
David?
Comment 21 Dirk Mueller 2022-04-07 07:22:46 UTC
I'm not david, but my understanding is that those patches are noops for 64 bit platforms as the kmap/unmap are empty functions there.
Comment 22 David Sterba 2022-04-07 13:56:15 UTC
(In reply to Takashi Iwai from comment #19)
> Those patches should be merged to SLE15-SP4-GA kernel.
> 
> David, I suppose those are safe to be cherry-picked to GA kernel (for 64bit
> systems), right?

The patches affect only 32bit architectures and are safe for 64bit arches. I don't understand the question about "SLE15-SP4-GA kernel", the patches have been merged since 17.2., that should be enough time to get them to the released kernels or at least to the update channels. If you still see bugs caused by missing the patches in OBS, then I'm not the person to ask but rather ask those responsible for deploying updated packages.
Comment 23 Takashi Iwai 2022-04-07 14:24:09 UTC
(In reply to David Sterba from comment #22)
> (In reply to Takashi Iwai from comment #19)
> > Those patches should be merged to SLE15-SP4-GA kernel.
> > 
> > David, I suppose those are safe to be cherry-picked to GA kernel (for 64bit
> > systems), right?
> 
> The patches affect only 32bit architectures and are safe for 64bit arches.

OK, thanks for confirmation.

>  I
> don't understand the question about "SLE15-SP4-GA kernel", the patches have
> been merged since 17.2., that should be enough time to get them to the
> released kernels or at least to the update channels.

SLE15-SP4 git branch is *NOT* for SLE15-SP4 GM release any longer.  Instead, it's for the first maintenance update kernel of SLE15-SP4.  That is, it's no branch for the SLE15-SP4 installer kernel, and if we need to fix it, the fix has to be merged into SLE15-SP4-GA branch instead.  This kernel branch split (happened months ago) was informed on various places many times.

BTW, Dirk has already sent a PR to SLE15-SP4-GA branch with cherry-picks of your commits in SLE15-SP4, so you don't need to do anything for now :)

I'll wait for ACK from SLE15-SP4 release managers, then they'll be fixed in GM.  (Unfortunately it already slipped for this week's submission, but maybe in the next week.)
Comment 24 Takashi Iwai 2022-04-11 07:02:16 UTC
The patches are merged now to SLE15-SP4-GA branch.  The next snapshot will include them.
Comment 29 Swamp Workflow Management 2022-07-21 22:26:01 UTC
SUSE-SU-2022:2520-1: An update that solves 49 vulnerabilities, contains 26 features and has 207 fixes is now available.

Category: security (important)
Bug References: 1055117,1061840,1065729,1071995,1089644,1103269,1118212,1121726,1137728,1156395,1157038,1157923,1175667,1179439,1179639,1180814,1183682,1183872,1184318,1184924,1187716,1188885,1189998,1190137,1190208,1190336,1190497,1190768,1190786,1190812,1191271,1191663,1192483,1193064,1193277,1193289,1193431,1193556,1193629,1193640,1193787,1193823,1193852,1194086,1194111,1194191,1194409,1194501,1194523,1194526,1194583,1194585,1194586,1194625,1194765,1194826,1194869,1195099,1195287,1195478,1195482,1195504,1195651,1195668,1195669,1195775,1195823,1195826,1195913,1195915,1195926,1195944,1195957,1195987,1196079,1196114,1196130,1196213,1196306,1196367,1196400,1196426,1196478,1196514,1196570,1196723,1196779,1196830,1196836,1196866,1196868,1196869,1196901,1196930,1196942,1196960,1197016,1197157,1197227,1197243,1197292,1197302,1197303,1197304,1197362,1197386,1197501,1197601,1197661,1197675,1197761,1197817,1197819,1197820,1197888,1197889,1197894,1197915,1197917,1197918,1197920,1197921,1197922,1197926,1198009,1198010,1198012,1198013,1198014,1198015,1198016,1198017,1198018,1198019,1198020,1198021,1198022,1198023,1198024,1198027,1198030,1198034,1198058,1198217,1198379,1198400,1198402,1198410,1198412,1198413,1198438,1198484,1198577,1198585,1198660,1198802,1198803,1198806,1198811,1198826,1198829,1198835,1198968,1198971,1199011,1199024,1199035,1199046,1199052,1199063,1199163,1199173,1199260,1199314,1199390,1199426,1199433,1199439,1199482,1199487,1199505,1199507,1199605,1199611,1199626,1199631,1199650,1199657,1199674,1199736,1199793,1199839,1199875,1199909,1200015,1200019,1200045,1200046,1200144,1200205,1200211,1200259,1200263,1200284,1200315,1200343,1200420,1200442,1200475,1200502,1200567,1200569,1200571,1200599,1200600,1200608,1200611,1200619,1200692,1200762,1200763,1200806,1200807,1200808,1200809,1200810,1200812,1200813,1200815,1200816,1200820,1200821,1200822,1200824,1200825,1200827,1200828,1200829,1200830,1200845,1200882,1200925,1201050,1201080,1201160,1201171,1201177,1201193,1201196,1201218,1201222,1201228,1201251,1201381,1201471,1201524
CVE References: CVE-2021-26341,CVE-2021-33061,CVE-2021-4204,CVE-2021-44879,CVE-2021-45402,CVE-2022-0264,CVE-2022-0494,CVE-2022-0617,CVE-2022-1012,CVE-2022-1016,CVE-2022-1184,CVE-2022-1198,CVE-2022-1205,CVE-2022-1462,CVE-2022-1508,CVE-2022-1651,CVE-2022-1652,CVE-2022-1671,CVE-2022-1679,CVE-2022-1729,CVE-2022-1734,CVE-2022-1789,CVE-2022-1852,CVE-2022-1966,CVE-2022-1972,CVE-2022-1974,CVE-2022-1998,CVE-2022-20132,CVE-2022-20154,CVE-2022-21123,CVE-2022-21125,CVE-2022-21127,CVE-2022-21166,CVE-2022-21180,CVE-2022-21499,CVE-2022-2318,CVE-2022-23222,CVE-2022-26365,CVE-2022-26490,CVE-2022-29582,CVE-2022-29900,CVE-2022-29901,CVE-2022-30594,CVE-2022-33740,CVE-2022-33741,CVE-2022-33742,CVE-2022-33743,CVE-2022-33981,CVE-2022-34918
JIRA References: SLE-13513,SLE-13521,SLE-15442,SLE-17855,SLE-18194,SLE-18234,SLE-18375,SLE-18377,SLE-18378,SLE-18382,SLE-18385,SLE-18901,SLE-18938,SLE-18978,SLE-19001,SLE-19026,SLE-19242,SLE-19249,SLE-19253,SLE-19924,SLE-21315,SLE-23643,SLE-24072,SLE-24093,SLE-24350,SLE-24549
Sources used:
openSUSE Leap 15.4 (src):    dtb-aarch64-5.14.21-150400.24.11.1, kernel-64kb-5.14.21-150400.24.11.1, kernel-debug-5.14.21-150400.24.11.1, kernel-default-5.14.21-150400.24.11.1, kernel-default-base-5.14.21-150400.24.11.1.150400.24.3.6, kernel-docs-5.14.21-150400.24.11.1, kernel-kvmsmall-5.14.21-150400.24.11.1, kernel-obs-build-5.14.21-150400.24.11.1, kernel-obs-qa-5.14.21-150400.24.11.1, kernel-source-5.14.21-150400.24.11.1, kernel-syms-5.14.21-150400.24.11.1, kernel-zfcpdump-5.14.21-150400.24.11.1
SUSE Linux Enterprise Workstation Extension 15-SP4 (src):    kernel-default-5.14.21-150400.24.11.1
SUSE Linux Enterprise Module for Live Patching 15-SP4 (src):    kernel-default-5.14.21-150400.24.11.1, kernel-livepatch-SLE15-SP4_Update_1-1-150400.9.5.3
SUSE Linux Enterprise Module for Legacy Software 15-SP4 (src):    kernel-default-5.14.21-150400.24.11.1
SUSE Linux Enterprise Module for Development Tools 15-SP4 (src):    kernel-docs-5.14.21-150400.24.11.1, kernel-obs-build-5.14.21-150400.24.11.1, kernel-source-5.14.21-150400.24.11.1, kernel-syms-5.14.21-150400.24.11.1
SUSE Linux Enterprise Module for Basesystem 15-SP4 (src):    kernel-64kb-5.14.21-150400.24.11.1, kernel-default-5.14.21-150400.24.11.1, kernel-default-base-5.14.21-150400.24.11.1.150400.24.3.6, kernel-source-5.14.21-150400.24.11.1, kernel-zfcpdump-5.14.21-150400.24.11.1
SUSE Linux Enterprise High Availability 15-SP4 (src):    kernel-default-5.14.21-150400.24.11.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 30 Swamp Workflow Management 2022-08-01 13:27:34 UTC
SUSE-SU-2022:2615-1: An update that solves 48 vulnerabilities, contains 26 features and has 202 fixes is now available.

Category: security (important)
Bug References: 1055117,1061840,1065729,1071995,1089644,1103269,1118212,1121726,1137728,1156395,1157038,1157923,1175667,1179439,1179639,1180814,1183682,1183872,1184318,1184924,1187716,1188885,1189998,1190137,1190208,1190336,1190497,1190768,1190786,1190812,1191271,1191663,1192483,1193064,1193277,1193289,1193431,1193556,1193629,1193640,1193787,1193823,1193852,1194086,1194111,1194191,1194409,1194501,1194523,1194526,1194583,1194585,1194586,1194625,1194765,1194826,1194869,1195099,1195287,1195478,1195482,1195504,1195651,1195668,1195669,1195775,1195823,1195826,1195913,1195915,1195926,1195944,1195957,1195987,1196079,1196114,1196130,1196213,1196306,1196367,1196400,1196426,1196478,1196514,1196570,1196723,1196779,1196830,1196836,1196866,1196868,1196869,1196901,1196930,1196942,1196960,1197016,1197157,1197227,1197243,1197292,1197302,1197303,1197304,1197362,1197386,1197501,1197601,1197661,1197675,1197761,1197817,1197819,1197820,1197888,1197889,1197894,1197915,1197917,1197918,1197920,1197921,1197922,1197926,1198009,1198010,1198012,1198013,1198014,1198015,1198016,1198017,1198018,1198019,1198020,1198021,1198022,1198023,1198024,1198027,1198030,1198034,1198058,1198217,1198379,1198400,1198402,1198412,1198413,1198438,1198484,1198577,1198585,1198660,1198802,1198803,1198806,1198811,1198826,1198835,1198968,1198971,1199011,1199024,1199035,1199046,1199052,1199063,1199163,1199173,1199260,1199314,1199390,1199426,1199433,1199439,1199482,1199487,1199505,1199507,1199605,1199611,1199626,1199631,1199650,1199657,1199674,1199736,1199793,1199839,1199875,1199909,1200015,1200019,1200045,1200046,1200144,1200205,1200211,1200259,1200263,1200284,1200315,1200343,1200420,1200442,1200475,1200502,1200567,1200569,1200571,1200572,1200599,1200600,1200608,1200611,1200619,1200692,1200762,1200763,1200806,1200807,1200808,1200809,1200810,1200812,1200815,1200816,1200820,1200822,1200824,1200825,1200827,1200828,1200829,1200830,1200845,1200882,1200925,1201050,1201160,1201171,1201177,1201193,1201196,1201218,1201222,1201228,1201251,150300
CVE References: CVE-2021-26341,CVE-2021-33061,CVE-2021-4204,CVE-2021-44879,CVE-2021-45402,CVE-2022-0264,CVE-2022-0494,CVE-2022-0617,CVE-2022-1012,CVE-2022-1016,CVE-2022-1184,CVE-2022-1198,CVE-2022-1205,CVE-2022-1508,CVE-2022-1651,CVE-2022-1652,CVE-2022-1671,CVE-2022-1679,CVE-2022-1729,CVE-2022-1734,CVE-2022-1789,CVE-2022-1852,CVE-2022-1966,CVE-2022-1972,CVE-2022-1974,CVE-2022-1998,CVE-2022-20132,CVE-2022-20154,CVE-2022-21123,CVE-2022-21125,CVE-2022-21127,CVE-2022-21166,CVE-2022-21180,CVE-2022-21499,CVE-2022-2318,CVE-2022-23222,CVE-2022-26365,CVE-2022-26490,CVE-2022-29582,CVE-2022-29900,CVE-2022-29901,CVE-2022-30594,CVE-2022-33740,CVE-2022-33741,CVE-2022-33742,CVE-2022-33743,CVE-2022-33981,CVE-2022-34918
JIRA References: SLE-13513,SLE-13521,SLE-15442,SLE-17855,SLE-18194,SLE-18234,SLE-18375,SLE-18377,SLE-18378,SLE-18382,SLE-18385,SLE-18901,SLE-18938,SLE-18978,SLE-19001,SLE-19026,SLE-19242,SLE-19249,SLE-19253,SLE-19924,SLE-21315,SLE-23643,SLE-24072,SLE-24093,SLE-24350,SLE-24549
Sources used:
openSUSE Leap 15.4 (src):    kernel-azure-5.14.21-150400.14.7.1, kernel-source-azure-5.14.21-150400.14.7.1, kernel-syms-azure-5.14.21-150400.14.7.1
SUSE Linux Enterprise Module for Public Cloud 15-SP4 (src):    kernel-azure-5.14.21-150400.14.7.1, kernel-source-azure-5.14.21-150400.14.7.1, kernel-syms-azure-5.14.21-150400.14.7.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.