Bug 1105271 - Kernel 4.4.143-65-default does not boot on Dell Precision T5810 (Xeon E5-1620 v3)
Kernel 4.4.143-65-default does not boot on Dell Precision T5810 (Xeon E5-1620...
Status: RESOLVED FIXED
: 1105352 1105385 (view as bug list)
Classification: openSUSE
Product: openSUSE Distribution
Classification: openSUSE
Component: Kernel
Leap 42.3
x86-64 Other
: P5 - None : Major (vote)
: ---
Assigned To: Borislav Petkov
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-08-18 11:18 UTC by Sebastian Parschauer
Modified: 2018-09-25 19:22 UTC (History)
8 users (show)

See Also:
Found By: L3
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
Photo of screen when hanging after early ucode update (123.36 KB, image/jpeg)
2018-08-18 11:18 UTC, Sebastian Parschauer
Details
Contents of /proc/cpuinfo (9.04 KB, text/plain)
2018-08-18 17:54 UTC, Sebastian Parschauer
Details
/proc/cpuinfo from latest Leap 15 kernel (9.29 KB, text/plain)
2018-08-18 18:30 UTC, Sebastian Parschauer
Details
Photo of screen with BIOS A27 (134.35 KB, image/jpeg)
2018-08-21 07:01 UTC, Sebastian Parschauer
Details
[PATCH] process SMT static key even in CPU_STARTING hotplug event (1.91 KB, patch)
2018-08-22 12:07 UTC, Jiri Kosina
Details | Diff
[PATCH] Update static key in CPU notifier, not the sched one (1.92 KB, patch)
2018-08-22 15:45 UTC, Jiri Kosina
Details | Diff
Serial port log from patch of comment 45 (166.45 KB, text/plain)
2018-08-22 17:24 UTC, Sebastian Parschauer
Details
Serial port log 4.4.143-65-default (13.79 KB, text/plain)
2018-08-22 17:39 UTC, Sebastian Parschauer
Details
Serial port log 4.12.14-lp150.12.16-default (46.19 KB, text/plain)
2018-08-22 18:42 UTC, Sebastian Parschauer
Details
Serial port logs 4.4.151 kernel ab97704c13 with and w/o bisected patch (15.51 KB, application/gzip)
2018-08-22 19:34 UTC, Sebastian Parschauer
Details
Log with maxcpus=1 (47.40 KB, text/plain)
2018-08-23 05:50 UTC, Sebastian Parschauer
Details
Log with tsc=reliable (46.63 KB, text/plain)
2018-08-23 05:55 UTC, Sebastian Parschauer
Details
Log with the two lines commented out (48.23 KB, text/plain)
2018-08-23 06:43 UTC, Sebastian Parschauer
Details
test patch (655 bytes, patch)
2018-08-23 08:50 UTC, Borislav Petkov
Details | Diff
Log for debug patch c69 (19.00 KB, text/plain)
2018-08-24 06:16 UTC, Sebastian Parschauer
Details
test patch 2 (1.06 KB, patch)
2018-08-24 08:12 UTC, Borislav Petkov
Details | Diff
test patch 3 (4.60 KB, patch)
2018-08-24 16:51 UTC, Borislav Petkov
Details | Diff
Log for debug patch c79 (39.03 KB, text/plain)
2018-08-25 09:36 UTC, Sebastian Parschauer
Details
Log jltest (160.04 KB, text/plain)
2018-08-29 06:52 UTC, Sebastian Parschauer
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Sebastian Parschauer 2018-08-18 11:18:45 UTC
Created attachment 780104 [details]
Photo of screen when hanging after early ucode update

The new Leap 42.3 kernel 4.4.143-65-default does not boot on my Dell Precision Tower 5810 SUSE R&D workstation with Intel Xeon E5-1620 v3 CPU.

Please note: *That CPU has an unstable TSC.*

Boot hangs directly after early microcode update. Had to boot the previous kernel 4.4.140-62-default.
Comment 1 Sebastian Parschauer 2018-08-18 11:36:48 UTC
/proc/cmdline when booting previous kernel:

BOOT_IMAGE=/boot/vmlinuz-4.4.140-62-default root=UUID=176a3ccd-fd1a-4fa5-b151-573078dc205e resume=/dev/nvme0n1p3 showopts plymouth.enable=0 video=vesa:off vga=normal usbcore.autosuspend=-1

I just don't like Plymouth and USB autosuspend and let nothing write to a framebuffer device due to nvidia driver warnings. NVIDIA Quadro K620 graphics with proprietary driver.

No tainted modules (besides nvidia proprietary driver from official repo), base system stuff only from main repos. Booting from Samsung NVMe (SM951 NVMe SAMSUNG 256GB). Doing legacy boot with pretty recent BIOS A26:

BIOS Information
        Vendor: Dell Inc.
        Version: A26
        Release Date: 04/19/2018

Hmm, okay A27 is available with importance "urgent".
> https://www.dell.com/support/home/us/en/04/drivers/driversdetails?driverId=FPNN0

Will update BIOS to A27 and retest.
Comment 2 Sebastian Parschauer 2018-08-18 12:07:10 UTC
Same issue with latest BIOS A27. Fully patched system.

BIOS Information
        Vendor: Dell Inc.
        Version: A27
        Release Date: 06/25/2018

@Boris or Jiri: Can you please help me here? TIA
Comment 3 Borislav Petkov 2018-08-18 12:38:57 UTC
Does it boot with the latest KOTD?

Does it boot if you supply "dis_ucode_ldr" on the kernel command line?
Comment 4 Sebastian Parschauer 2018-08-18 13:25:37 UTC
"dis_ucode_ldr" does not help. With BIOS A27 the ucode also does not have to be updated. Looks related to timers.

Testing the latest 4.4.y KOTD (3775de0, 2018-Aug-17 21:00:08) next.
Comment 6 Sebastian Parschauer 2018-08-18 14:50:31 UTC
Will bisect with KOTDs which commit introduced the bug.
Comment 8 Sebastian Parschauer 2018-08-18 17:54:22 UTC
Created attachment 780109 [details]
Contents of /proc/cpuinfo

Collected with the last good KOTD kernel-default-4.4.147-1.1.g8302b03. It's strange for me that the cores have different MHz values.
Comment 9 Sebastian Parschauer 2018-08-18 18:30:14 UTC
Created attachment 780110 [details]
/proc/cpuinfo from latest Leap 15 kernel

Different (correct looking) MHz values, "l1tf" added to bugs.

Added flags:
cpuid, cpuid_fault, flush_l1d, intel_ppin, l1tf, pti

Removed flags:
eagerfpu, kaiser
Comment 10 Borislav Petkov 2018-08-19 05:47:13 UTC
(In reply to Sebastian Parschauer from comment #8)
> It's strange for me that the cores have different MHz values.

That's because it is sampling the effective, *current* core frequency from APERF/MPERF, for something like 10ms, I think. VS the static P0 frequency which you're probably expecting but that would be a lie so the actual freq sample is kinda closer to the current load on the box.
Comment 11 Sebastian Parschauer 2018-08-20 11:15:59 UTC
Hmm, with the Leap 15 kernel also the TSC is working:

> DMI: Dell Inc. Precision Tower 5810/0K240Y, BIOS A27 06/25/2018
> tsc: Fast TSC calibration using PIT
...
> clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484882848 ns
> hpet clockevent registered
> tsc: Fast TSC calibration using PIT
> tsc: Detected 3492.005 MHz processor
> [Firmware Bug]: TSC ADJUST: CPU0: -3570856054 force to 0
> Calibrating delay loop (skipped), value calculated using timer frequency.. 6984.01 BogoMIPS (lpj=13968020)
...
> DMAR-IR: Enabled IRQ remapping in xapic mode
> x2apic: IRQ remapping doesn't support X2APIC mode
> ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
> TSC deadline timer enabled
> smpboot: CPU0: Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz (family: 0x6, model: 0x3f, stepping: 0x2)
> Performance Events: PEBS fmt2+, Haswell events, 16-deep LBR, full-width counters, Intel PMU driver.
> ... version:                3
> ... bit width:              48
> ... generic registers:      4
> ... value mask:             0000ffffffffffff
> ... max period:             00007fffffffffff
> ... fixed-purpose events:   3
> ... event mask:             000000070000000f
> smp: Bringing up secondary CPUs ...
> x86: Booting SMP configuration:
> .... node  #0, CPUs:      #1
> [Firmware Bug]: TSC ADJUST differs within socket(s), fixing all errors
>  #2
> NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
>  #3 #4 #5 #6 #7
> smp: Brought up 1 node, 8 CPUs
> smpboot: Max logical packages: 1
> smpboot: Total of 8 processors activated (55872.08 BogoMIPS)
> node 0 initialised, 2719092 pages in 24ms
> devtmpfs: initialized
> x86/mm: Memory block size: 128MB

A second later:
> tsc: Refined TSC clocksource calibration: 3491.914 MHz
> clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x32557ae966b, max_idle_ns: 440795369289 ns

A further second later:
> clocksource: Switched to clocksource tsc
Comment 15 Sebastian Parschauer 2018-08-21 07:01:46 UTC
Created attachment 780265 [details]
Photo of screen with BIOS A27
Comment 16 Jiri Kosina 2018-08-21 07:06:43 UTC
Would it be possible for you to do the bisection in series.conf, on these patches? (the "nosmt" section).

I am reasonably certain that it's one of them causing the issue, so it'd be interesting to know which one exactly. The series should be bisectable without any issues.

Do you by any chance have SMT disabled in BIOS?

Thanks a lot.

        patches.arch/01-sched-smt-update-sched_smt_present-at-runtime.patch
        patches.arch/02-x86-smp-provide-topology_is_primary_thread.patch
        patches.arch/03-x86-topology-provide-topology_smt_supported.patch
        patches.arch/04-cpu-hotplug-split-do_cpu_down.patch
        patches.arch/04.1-cpu-hotplug-add-sysfs-state-interface.patch
        patches.arch/04.2-x86-topology-add-topology_max_smt_threads.patch
        patches.arch/04.3-x86-smpboot-do-not-use-smp_num_siblings-in-_max_logical_packages-calculation.patch
        patches.arch/05-cpu-hotplug-provide-knobs-to-control-smt.patch
        patches.arch/06-x86-cpu-remove-the-pointless-cpu-printout.patch
        patches.arch/07-x86-cpu-amd-remove-the-pointless-detect_ht-call.patch
        patches.arch/08-x86-cpu-common-provide-detect_ht_early.patch
        patches.arch/09-x86-cpu-topology-provide-detect_extended_topology_early.patch
        patches.arch/10-x86-cpu-intel-evaluate-smp_num_siblings-early.patch
        patches.arch/11-x86-cpu-amd-do-not-check-cpuid-max-ext-level-before-parsing-smp-info.patch
        patches.arch/12-x86-cpu-amd-evaluate-smp_num_siblings-early.patch
        patches.arch/14-x86-cpu-amd-move-topoext-reenablement-before-reading-smp_num_siblings.patch
        patches.arch/15-cpu-hotplug-boot-HT-siblings-at-least-once.patch
        patches.arch/16-cpu-hotplug-Online-siblings-when-SMT-control-is-turn.patch
Comment 17 Jiri Kosina 2018-08-21 07:11:55 UTC
FWIW all three reports we have so far are exactly the same CPU (family: 0x6, model: 0x3f, stepping: 0x2)
Comment 18 Borislav Petkov 2018-08-21 08:00:13 UTC
Btw, there's one more fix for the fix for the SMT off case:

bc2d8d262cba ("cpu/hotplug: Fix SMT supported evaluation")

Can you pls apply it ontop and test with it too?

Thx.
Comment 19 Takashi Iwai 2018-08-21 09:10:37 UTC
(In reply to Borislav Petkov from comment #18)
> Btw, there's one more fix for the fix for the SMT off case:
> 
> bc2d8d262cba ("cpu/hotplug: Fix SMT supported evaluation")
> 
> Can you pls apply it ontop and test with it too?

I'm building a test kernel with this patch in OBS home:tiwai:bsc1105271 repo.
Comment 20 Sebastian Parschauer 2018-08-21 09:18:08 UTC
Thanks, SMT is enabled. Expecting 4 cores and 8 HTs to come up.
Comment 21 Takashi Iwai 2018-08-21 09:56:19 UTC
(In reply to Takashi Iwai from comment #19)
> I'm building a test kernel with this patch in OBS home:tiwai:bsc1105271 repo.

The test kernel is ready:
  http://download.opensuse.org/repositories/home:/tiwai:/bsc1105271/standard/
Comment 22 Sebastian Parschauer 2018-08-21 10:14:39 UTC
(In reply to Takashi Iwai from comment #21)
> (In reply to Takashi Iwai from comment #19)
> > I'm building a test kernel with this patch in OBS home:tiwai:bsc1105271 repo.
> 
> The test kernel is ready:
>   http://download.opensuse.org/repositories/home:/tiwai:/bsc1105271/standard/

Same issue with this kernel-default-4.4.147-1.1.gb325bb2.x86_64.rpm. Did not help.
Comment 23 Jiri Kosina 2018-08-21 10:43:13 UTC
(In reply to Sebastian Parschauer from comment #22)

> Same issue with this kernel-default-4.4.147-1.1.gb325bb2.x86_64.rpm. Did not
> help.

I expected that a bit, as it's supposed to change behavior only if nosmt is on cmdline or SMT is disabled in BIOS.

Would you be able to do the bisect please?

My primary suspect would be

   patches.arch/15-cpu-hotplug-boot-HT-siblings-at-least-once.patch

so you can either start with that one, or do a proper bisect of those 18 patches.

Thanks again.
Comment 24 Sebastian Parschauer 2018-08-21 12:36:25 UTC
I'm on it.
Comment 25 Takashi Iwai 2018-08-21 14:01:31 UTC
For convenience, I built a kernel with the revert of the suspected 15-cpu-hotplug-*.patch, found in OBS home:tiwai:bsc1105271-revert-15 repo,
  http://download.opensuse.org/repositories/home:/tiwai:/bsc1105271-revert-15/standard/
Comment 26 Sebastian Parschauer 2018-08-21 15:21:32 UTC
(In reply to Takashi Iwai from comment #25)
> http://download.opensuse.org/repositories/home:/tiwai:/bsc1105271-revert-15/
> standard/

No luck with that. Problem persists. Will compile locally.
Comment 27 Sebastian Parschauer 2018-08-21 16:46:22 UTC
What do I have to change in the kernel config that it doesn't always recompile everything from scratch? TIA
This is really annoying for quick testing.
Comment 28 Takashi Iwai 2018-08-21 16:52:01 UTC
One standard way would be to reduce your config via "make localmodconfig".
Suppose you have .config file in the kernel tree, run "make localmodconfig" on the running system.  Then it'll be reduced only for the modules that are being used.

I'd recommend also modify CONFIG_LOCAL_VERSION to a different string for avoiding confusion / conflict.  When bisecting, for example, I change this string at each iteration or enable CONFIG_LOCALVERSION_AUTO.
Comment 29 Sebastian Parschauer 2018-08-22 07:21:30 UTC
Thanks, that helped. Removing everything until:
> patches.arch/05-cpu-hotplug-provide-knobs-to-control-smt.patch
is not a problem but the L1TF patches heavily depend on this one.

With the SMT patches removed until this one, the problem persists.
Comment 30 Jan Baier 2018-08-22 07:44:09 UTC
*** Bug 1105352 has been marked as a duplicate of this bug. ***
Comment 31 Sebastian Parschauer 2018-08-22 07:51:35 UTC
I managed to disable CONFIG_HOTPLUG_SMT by removing it from arch/Kconfig. Problem persists.
Comment 32 Jiri Kosina 2018-08-22 07:57:16 UTC
(In reply to Sebastian Parschauer from comment #29)
> Thanks, that helped. Removing everything until:
> > patches.arch/05-cpu-hotplug-provide-knobs-to-control-smt.patch
> is not a problem but the L1TF patches heavily depend on this one.

So could you please bisect the series then, including the L1TF patches (or just disable all them in one go, and see whether the problem persists, and if so, just bisect the nosmt series).

Of course you have to start bisection by disabling always 'higher-numbered' patches first, so that the patches still apply.

Thanks.
Comment 33 Sebastian Parschauer 2018-08-22 08:20:45 UTC
Now you need to help me again how to get from what kernel-source.git provides to what kernel.git provides. I've used kernel.git before and just used git revert.
Comment 34 Borislav Petkov 2018-08-22 08:27:36 UTC
(In reply to Sebastian Parschauer from comment #33)
> Now you need to help me again how to get from what kernel-source.git
> provides to what kernel.git provides. I've used kernel.git before and just
> used git revert.

Just comment out the patches in series.conf up to which you wanna test, sequence-patch.sh it, build and boot. Then you remove the comment of the next patch and repeat. Until you see the difference when booting.
Comment 35 Sebastian Parschauer 2018-08-22 08:36:58 UTC
Got it, was missing linux-4.4.tar.xz or setting LINUX_GIT.
Comment 36 Jiri Kosina 2018-08-22 08:42:09 UTC
(In reply to Jiri Kosina from comment #32)

> So could you please bisect the series then, including the L1TF patches (or
> just disable all them in one go, and see whether the problem persists, and
> if so, just bisect the nosmt series).
> 
> Of course you have to start bisection by disabling always 'higher-numbered'
> patches first, so that the patches still apply.

So acutally, to be completely on a safe side, I'd rather propose:

- you start by commenting out *all* the l1tf and nosmt patches

- if that kernel boots, it's clear that the issue is introduced
  by one of those patches, and start bisect-enabling them starting
  from the one with lowest number (so that they are still in proper
  series), until you identify the one that introduces the regression

- if even with all the nosmt and l1tf patches disabled the problem still
  persists, it's caused by some other change that happened (I find that rather
  unlikely, but let's not completely ditch that possiblity).

Thank you.
Comment 37 Sebastian Parschauer 2018-08-22 09:16:03 UTC
(In reply to Jiri Kosina from comment #16)

Almost found. It is one of these:
>         patches.arch/01-sched-smt-update-sched_smt_present-at-runtime.patch
>         patches.arch/02-x86-smp-provide-topology_is_primary_thread.patch
>         patches.arch/03-x86-topology-provide-topology_smt_supported.patch
>         patches.arch/04-cpu-hotplug-split-do_cpu_down.patch
>         patches.arch/04.1-cpu-hotplug-add-sysfs-state-interface.patch
>         patches.arch/04.2-x86-topology-add-topology_max_smt_threads.patch        
>         patches.arch/04.3-x86-smpboot-do-not-use-smp_num_siblings-in-
> _max_logical_packages-calculation.patch
>         patches.arch/05-cpu-hotplug-provide-knobs-to-control-smt.patch
Comment 38 Sebastian Parschauer 2018-08-22 09:33:52 UTC
It is directly the first patch of the series:
> patches.arch/01-sched-smt-update-sched_smt_present-at-runtime.patch

Without it, the system is booting.
Comment 39 Sebastian Parschauer 2018-08-22 09:36:00 UTC
*** Bug 1105385 has been marked as a duplicate of this bug. ***
Comment 40 Takashi Iwai 2018-08-22 11:22:51 UTC
(In reply to Sebastian Parschauer from comment #38)
> It is directly the first patch of the series:
> > patches.arch/01-sched-smt-update-sched_smt_present-at-runtime.patch
> 
> Without it, the system is booting.

So you just disabled this patch while keeping the rest, and it boots fine?
Comment 41 Jiri Kosina 2018-08-22 12:07:09 UTC
Created attachment 780406 [details]
[PATCH] process SMT static key even in CPU_STARTING hotplug event

Thanks a lot for the bisect.

Could you please try with this patch applied on kernel-source? It moves the static key check+enablement unconditionally on the hotplug event that appeared (I think we might be missing CPU_STARTING case here).
Comment 42 Sebastian Parschauer 2018-08-22 12:07:47 UTC
(In reply to Takashi Iwai from comment #40)
> So you just disabled this patch while keeping the rest, and it boots fine?

I've tested both ways based on the first bad KOTD commit f206194.

First way: disabling everything at "nosmt", "KVM", "SMT runtime control" and "fixes".

Second way: Only disable the patch patches.arch/01-sched-smt-update-sched_smt_present-at-runtime.patch.

The result is not perfect. Systemd times out while looking for my encrypted /data partition on 1TB HDD after I provide the pw for encrypted /home on NVMe. But I think this is config related.

I can try based on latest origin/SLE12-SP3 with the Leap 42.3 kernel-default config and only the one patch removed if you want.
Comment 43 Sebastian Parschauer 2018-08-22 12:59:48 UTC
(In reply to Sebastian Parschauer from comment #42)
> I can try based on latest origin/SLE12-SP3 with the Leap 42.3 kernel-default
> config and only the one patch removed if you want.

That method worked. System is coming up completely. I'm on commit ab97704c13 now. Will build two kernels next: One with the patch re-enabled and then one with the patch from comment 41 added at "fixes".
Comment 45 Jiri Kosina 2018-08-22 15:45:18 UTC
Created attachment 780447 [details]
[PATCH] Update static key in CPU notifier, not the sched one
Comment 47 Borislav Petkov 2018-08-22 16:17:44 UTC
Also, one more thing I'd like you to try:

Make sure you see something like:

[    0.545232] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.

during boot and when it freezes, wait a couple of minutes to see if the
watchdog fires with, hopefully, a helpful message.

If you can connect your box to another over serial to catch that whole
output would be even better. Otherwise we'll be staring at jpgs...

Thx.
Comment 48 Sebastian Parschauer 2018-08-22 17:24:22 UTC
Created attachment 780459 [details]
Serial port log from patch of comment 45

Your wish is my command. ;) I'm attaching the log of the kernel with the patch from comment 45 collected with minicom from my laptop. The picture on the screen is the same. So the patch did not help.

The scheduler code is spinning and printing call traces on serial port.

> [    0.487983] smpboot:  #2smpboot:  #3smpboot:  #4
> [    0.496242] BUG: scheduling while atomic: swapper/4/0/0x00000002
Comment 49 Sebastian Parschauer 2018-08-22 17:26:29 UTC
No NMI watchdog. Screen turns black after several minutes.
Comment 50 Sebastian Parschauer 2018-08-22 17:39:52 UTC
Created attachment 780460 [details]
Serial port log 4.4.143-65-default

I'm sure you know this one from bug 1105352 already.
Comment 51 Jiri Kosina 2018-08-22 17:51:26 UTC
Do you have the TSC desync messages (and turning TSC off) in the dmesg of the boot that works (with the 42.3 kernel with just the bisected patch removed)?
Comment 52 Sebastian Parschauer 2018-08-22 18:42:19 UTC
Created attachment 780465 [details]
Serial port log 4.12.14-lp150.12.16-default

(In reply to Sebastian Parschauer from comment #50)
Commit ab97704c13 shows the same behavior as 4.4.143-65-default - almost identical boot log.

But maybe this boot log of the Leap 15 kernel helps you. I only captured it until it asked for the first decryption password.
Comment 53 Sebastian Parschauer 2018-08-22 19:34:28 UTC
Created attachment 780477 [details]
Serial port logs 4.4.151 kernel ab97704c13 with and w/o bisected patch

I'm attaching the boot logs of kernel source commit ab97704c13 with and without the bisected patch (again booted until request for passphrase).

I don't see a significant change in the logs when comparing them with meld.
Comment 54 Sebastian Parschauer 2018-08-22 19:45:49 UTC
When comparing the 4.4 kernel with the Leap 15 kernel, then there is a change in the following line:

4.4:
> x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT

4.12:
> x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT

The second WC is changed to WP.
Comment 55 Borislav Petkov 2018-08-23 03:41:24 UTC
Hmmm.

Ok, a couple more things to try:

* does it boot if you supply "maxcpus=1" on the kernel command line?
Catch dmesg too pls.

* what happens if you simply comment out those two lines?

        if (cpumask_weight(cpu_smt_mask(cpu)) > 1)
                static_branch_enable(&sched_smt_present);

The scheduler might not know that it has SMT siblings but it should at
least boot.

Thx.
Comment 56 Borislav Petkov 2018-08-23 03:55:04 UTC
(In reply to Jiri Kosina from comment #51)
> Do you have the TSC desync messages (and turning TSC off) in the dmesg of
> the boot that works (with the 42.3 kernel with just the bisected patch
> removed)?

So they're not present in the 4.12 dmesg and it says:

[    0.000000] [Firmware Bug]: TSC ADJUST differs within socket(s), fixing all errors

and I know there was a major screwage in the TSC_ADJUST area and if BIOS is fumbling with it...

I don't see now that would cause the hang though. Maybe boot with

tsc=reliable

to disable the sync check...
Comment 57 Sebastian Parschauer 2018-08-23 05:50:25 UTC
Created attachment 780501 [details]
Log with maxcpus=1

It boots with maxcpus=1, maxcpus=4, but not with maxcpus=5.
Comment 58 Sebastian Parschauer 2018-08-23 05:55:44 UTC
Created attachment 780502 [details]
Log with tsc=reliable

System is coming up with tsc=reliable. So it really has something to do with the TSC.
Comment 60 Sebastian Parschauer 2018-08-23 06:43:22 UTC
Created attachment 780510 [details]
Log with the two lines commented out

The -spars4 kernel with the two lines commented out is also booting.
Comment 61 Takashi Iwai 2018-08-23 06:51:55 UTC
Could you check the kernel in IBS (not OBS) home:tiwai:test:sle12-sp3-smt-test?
Comment 62 Sebastian Parschauer 2018-08-23 07:06:35 UTC
(In reply to Takashi Iwai from comment #61)
> Could you check the kernel in IBS (not OBS)
> home:tiwai:test:sle12-sp3-smt-test?

Done, kernel-default-4.4.151-1.1.g6950079.x86_64.rpm does not boot. Same issue like with ab97704c13. Do you need the log?
Comment 63 Takashi Iwai 2018-08-23 07:25:49 UTC
(In reply to Sebastian Parschauer from comment #62)
> (In reply to Takashi Iwai from comment #61)
> > Could you check the kernel in IBS (not OBS)
> > home:tiwai:test:sle12-sp3-smt-test?
> 
> Done, kernel-default-4.4.151-1.1.g6950079.x86_64.rpm does not boot. Same
> issue like with ab97704c13. Do you need the log?

Not needed, thanks.
Comment 69 Borislav Petkov 2018-08-23 08:50:51 UTC
Created attachment 780527 [details]
test patch

Ok, please remove all test patches, apply this one and boot with

"debug initcall_debug ignore_loglevel log_buf_len=16M"

on the kernel command line and catch dmesg over serial again.

Thx.
Comment 73 Sebastian Parschauer 2018-08-24 06:16:31 UTC
Created attachment 780661 [details]
Log for debug patch c69

I'm attaching the requested log for comment 69.
Comment 74 Borislav Petkov 2018-08-24 08:12:00 UTC
Created attachment 780667 [details]
test patch 2

Ok, looks like that jump label call hangs somewhere. Looking at
upstream, we don't have

  d0646a6f5533 ("jump_label: Add RELEASE barrier after text changes")

so here's an updated patch.

Also, do you have lockdep enabled?

If not, please enable CONFIG_DEBUG_LOCK_ALLOC and CONFIG_PROVE_LOCKING and do
the exact same test as before.

Thx.
Comment 75 Jiri Kosina 2018-08-24 08:54:16 UTC
(In reply to Borislav Petkov from comment #74)
> Created attachment 780667 [details]
> test patch 2
> 
> Ok, looks like that jump label call hangs somewhere. Looking at
> upstream, we don't have
> 
>   d0646a6f5533 ("jump_label: Add RELEASE barrier after text changes")
> 
> so here's an updated patch.

I actually think the problem is elsewhere.

arch_jump_label_transform() calls get_online_cpus(), but we're calling this from CPU hotplug/bringup path while holding (I will have to double check that, but I am pretty sure at this point) CPU hotplug lock, and therefore arch_jump_label_transform() deadlocks on acquiring it.
Comment 77 Borislav Petkov 2018-08-24 11:12:17 UTC
(In reply to Jiri Kosina from comment #75)
> arch_jump_label_transform() calls get_online_cpus(), but we're calling this
> from CPU hotplug/bringup path while holding (I will have to double check
> that, but I am pretty sure at this point) CPU hotplug lock, and therefore
> arch_jump_label_transform() deadlocks on acquiring it.

Hmm, and the fix for that is:

f2545b2d4ce1 ("jump_label: Reorder hotplug lock and jump_label_lock")

but if I backport that, I'd need to backport the _cpuslocked() variants too.

And yes, why does it trigger only on those machines is beyond me too.
Comment 79 Borislav Petkov 2018-08-24 16:51:40 UTC
Created attachment 780747 [details]
test patch 3

Ok, here's a more involved version of the test patch, please try it. Also, add to the cmdline parameters earlyprintk=serial, ... as explained in Documentation/kernel-parameters.txt by filling in the serial port of your particular device.

Thx.
Comment 80 Sebastian Parschauer 2018-08-25 08:20:50 UTC
Let me summarize:
* use test patch 3 instead of test patch 1
* enable CONFIG_DEBUG_LOCK_ALLOC and CONFIG_PROVE_LOCKING in config
* boot with:
> debug initcall_debug ignore_loglevel log_buf_len=16M earlyprintk=serial,ttyS0,115200
added to the boot parameters for this kernel.

Kernel is building. Will test it and attach the log.
Comment 81 Sebastian Parschauer 2018-08-25 09:36:24 UTC
Created attachment 780775 [details]
Log for debug patch c79

Kernel does not boot. Lockdep detected a deadlock.
Comment 82 Borislav Petkov 2018-08-25 10:08:34 UTC
Good work, thanks!

This explains it all, lemme see if I can write it correctly:

tsc_init() enables the __use_tsc static key on the BSP and *that* grabs
jump_label_mutex before it grabs the hotplug lock while on another CPU
the CPU notifier runs which holds the hotplug lock already and then
enables another static key - the sched_smt_present - which causes the
lock inversion by trying to grab the jump_label_mutex first.

@jikos: the way I see it, there's no way around backporting the jump
label patches so that we don't grab that hotplug again in the notifier
call.

Or am I missing something?
Comment 83 Jiri Kosina 2018-08-28 06:37:29 UTC
(In reply to Borislav Petkov from comment #82)
>
> tsc_init() enables the __use_tsc static key on the BSP and *that* grabs
> jump_label_mutex before it grabs the hotplug lock while on another CPU
> the CPU notifier runs which holds the hotplug lock already and then
> enables another static key - the sched_smt_present - which causes the
> lock inversion by trying to grab the jump_label_mutex first.

Yeah, I agree with that analysis completely, thanks.

> 
> @jikos: the way I see it, there's no way around backporting the jump
> label patches so that we don't grab that hotplug again in the notifier
> call.
> 
> Or am I missing something?

I have been looking into the code a bit, and I really don't see any other way around this than factoring the hotplug lock out of jump label updating code :/
Comment 85 Borislav Petkov 2018-08-28 09:51:40 UTC
Ok, here's a completely untested kernel with the backported pile:

http://beta.suse.com/private/bpetkov/linux-4.4.152-jltest-x86.tar.xz

Sebastian, pls give it a try with the same cmdline args as previously:

"debug initcall_debug ignore_loglevel log_buf_len=16M earlyprintk=serial,ttyS0,115200"

You can install it by doing:

# tar xvfJ linux-4.4.152-jltest-x86.tar.xz -C /

as root and then

# mkinitrd

to regenerate the initrd.

Thx.
Comment 86 Sebastian Parschauer 2018-08-29 06:52:19 UTC
Created attachment 781104 [details]
Log jltest

Thanks, that kernel boots with and without the added boot parameters. Attaching the log with them. LGTM
Comment 87 Borislav Petkov 2018-08-29 07:02:14 UTC
Very nice, thanks for testing effort!

I'll queue the patches.
Comment 88 Sebastian Parschauer 2018-08-31 15:10:11 UTC
Hmm, can't find the branch. Would you mind if I would pick up the patches from your branch and apply them on top of the Leap 42.3 kernel in OBS?
Comment 97 Sebastian Parschauer 2018-09-11 10:17:11 UTC
Added to my personal Leap 42.3 fixes (traditional method without git):
> https://build.opensuse.org/project/show/home:sparschauer:leap42.3_fixes

Version: 4.4.143-65.1.1.FIX.1105271
Change: added the 8 jump label patches on top of Leap 42.3 kernel MU
Tested: yes, works
Supported: no

Thanks for all the help here!
Comment 99 Swamp Workflow Management 2018-09-11 15:36:15 UTC
This is an autogenerated message for OBS integration:
This bug (1105271) was mentioned in
https://build.opensuse.org/request/show/635004 42.3 / kernel-source
Comment 101 Swamp Workflow Management 2018-09-16 13:24:24 UTC
openSUSE-SU-2018:2738-1: An update that solves 14 vulnerabilities and has 93 fixes is now available.

Category: security (important)
Bug References: 1012382,1015342,1015343,1017967,1019695,1019699,1020412,1021121,1022604,1024361,1024365,1024376,1027968,1030552,1031492,1033962,1042286,1048317,1050431,1053685,1055014,1056596,1062604,1063646,1064232,1065364,1066223,1068032,1068075,1069138,1078921,1080157,1083663,1085042,1085536,1085539,1087092,1089066,1090888,1091171,1091860,1092903,1096254,1096748,1097105,1098253,1098822,1099597,1099810,1099832,1099922,1099999,1100000,1100001,1100132,1101822,1102346,1102486,1102517,1102715,1102797,1104485,1104683,1104897,1105271,1105292,1105296,1105322,1105323,1105392,1105396,1105524,1105536,1105769,1106016,1106105,1106185,1106191,1106229,1106271,1106275,1106276,1106278,1106281,1106283,1106369,1106509,1106511,1106697,1106929,1106934,1106995,1107060,1107078,1107319,1107320,1107689,1107735,1107937,1107966,963575,966170,966172,969470,969476,969477,970506
CVE References: CVE-2018-10902,CVE-2018-10938,CVE-2018-10940,CVE-2018-1128,CVE-2018-1129,CVE-2018-12896,CVE-2018-13093,CVE-2018-13094,CVE-2018-13095,CVE-2018-15572,CVE-2018-16658,CVE-2018-6554,CVE-2018-6555,CVE-2018-9363
Sources used:
openSUSE Leap 42.3 (src):    kernel-debug-4.4.155-68.1, kernel-default-4.4.155-68.1, kernel-docs-4.4.155-68.1, kernel-obs-build-4.4.155-68.1, kernel-obs-qa-4.4.155-68.1, kernel-source-4.4.155-68.1, kernel-syms-4.4.155-68.1, kernel-vanilla-4.4.155-68.1
Comment 103 Sebastian Parschauer 2018-09-20 14:58:53 UTC
The latest Leap 42.3 MU works fine. Thanks. Dropped my temporary fix now.

@Sebastian Vollath: Can you confirm for your workstation? Then please set this to verified - fixed. TIA
Comment 104 Swamp Workflow Management 2018-09-20 22:23:58 UTC
SUSE-SU-2018:2775-1: An update that solves 21 vulnerabilities and has 98 fixes is now available.

Category: security (important)
Bug References: 1012382,1015342,1015343,1017967,1019695,1019699,1020412,1021121,1022604,1024361,1024365,1024376,1027968,1030552,1031492,1033962,1042286,1048317,1050431,1053685,1055014,1056596,1062604,1063646,1064232,1065364,1066223,1068032,1068075,1069138,1078921,1080157,1083663,1085042,1085536,1085539,1086457,1087092,1089066,1090888,1091171,1091860,1096254,1096748,1097105,1098253,1098822,1099597,1099810,1099811,1099813,1099832,1099844,1099845,1099846,1099849,1099863,1099864,1099922,1099999,1100000,1100001,1100132,1101822,1101841,1102346,1102486,1102517,1102715,1102797,1103269,1103445,1103717,1104319,1104485,1104494,1104495,1104683,1104897,1105271,1105292,1105322,1105323,1105392,1105396,1105524,1105536,1105769,1106016,1106105,1106185,1106229,1106271,1106275,1106276,1106278,1106281,1106283,1106369,1106509,1106511,1106697,1106929,1106934,1106995,1107060,1107078,1107319,1107320,1107689,1107735,1107966,963575,966170,966172,969470,969476,969477,970506
CVE References: CVE-2018-10876,CVE-2018-10877,CVE-2018-10878,CVE-2018-10879,CVE-2018-10880,CVE-2018-10881,CVE-2018-10882,CVE-2018-10883,CVE-2018-10902,CVE-2018-10938,CVE-2018-1128,CVE-2018-1129,CVE-2018-12896,CVE-2018-13093,CVE-2018-13094,CVE-2018-13095,CVE-2018-15572,CVE-2018-16658,CVE-2018-6554,CVE-2018-6555,CVE-2018-9363
Sources used:
SUSE Linux Enterprise Live Patching 12-SP3 (src):    kgraft-patch-SLE12-SP3_Update_17-1-4.3.1
Comment 105 Swamp Workflow Management 2018-09-20 22:43:21 UTC
SUSE-SU-2018:2776-1: An update that solves 21 vulnerabilities and has 98 fixes is now available.

Category: security (important)
Bug References: 1012382,1015342,1015343,1017967,1019695,1019699,1020412,1021121,1022604,1024361,1024365,1024376,1027968,1030552,1031492,1033962,1042286,1048317,1050431,1053685,1055014,1056596,1062604,1063646,1064232,1065364,1066223,1068032,1068075,1069138,1078921,1080157,1083663,1085042,1085536,1085539,1086457,1087092,1089066,1090888,1091171,1091860,1096254,1096748,1097105,1098253,1098822,1099597,1099810,1099811,1099813,1099832,1099844,1099845,1099846,1099849,1099863,1099864,1099922,1099999,1100000,1100001,1100132,1101822,1101841,1102346,1102486,1102517,1102715,1102797,1103269,1103445,1103717,1104319,1104485,1104494,1104495,1104683,1104897,1105271,1105292,1105322,1105323,1105392,1105396,1105524,1105536,1105769,1106016,1106105,1106185,1106229,1106271,1106275,1106276,1106278,1106281,1106283,1106369,1106509,1106511,1106697,1106929,1106934,1106995,1107060,1107078,1107319,1107320,1107689,1107735,1107966,963575,966170,966172,969470,969476,969477,970506
CVE References: CVE-2018-10876,CVE-2018-10877,CVE-2018-10878,CVE-2018-10879,CVE-2018-10880,CVE-2018-10881,CVE-2018-10882,CVE-2018-10883,CVE-2018-10902,CVE-2018-10938,CVE-2018-1128,CVE-2018-1129,CVE-2018-12896,CVE-2018-13093,CVE-2018-13094,CVE-2018-13095,CVE-2018-15572,CVE-2018-16658,CVE-2018-6554,CVE-2018-6555,CVE-2018-9363
Sources used:
SUSE Linux Enterprise Workstation Extension 12-SP3 (src):    kernel-default-4.4.155-94.50.1
SUSE Linux Enterprise Software Development Kit 12-SP3 (src):    kernel-docs-4.4.155-94.50.1, kernel-obs-build-4.4.155-94.50.1
SUSE Linux Enterprise Server 12-SP3 (src):    kernel-default-4.4.155-94.50.1, kernel-source-4.4.155-94.50.1, kernel-syms-4.4.155-94.50.1
SUSE Linux Enterprise High Availability 12-SP3 (src):    kernel-default-4.4.155-94.50.1
SUSE Linux Enterprise Desktop 12-SP3 (src):    kernel-default-4.4.155-94.50.1, kernel-source-4.4.155-94.50.1, kernel-syms-4.4.155-94.50.1
SUSE CaaS Platform ALL (src):    kernel-default-4.4.155-94.50.1
SUSE CaaS Platform 3.0 (src):    kernel-default-4.4.155-94.50.1
Comment 106 Sebastian Parschauer 2018-09-21 06:30:14 UTC
@Jan: Is this fixed with the SLE kernel MU for you as well? TIA
Comment 107 Sebastian Vollath 2018-09-21 11:28:00 UTC
#comment103 : Sorry, I can't. Because one of my SSDs failed at the same time, and also given the fix for 4.4 I switched to Leap15 now (it's my workstation).
Comment 108 Jan Baier 2018-09-24 07:37:04 UTC
(In reply to Sebastian Parschauer from comment #106)
> @Jan: Is this fixed with the SLE kernel MU for you as well? TIA

I do not have the precisely same software setup anymore, but the latest kernel seems to work for me as well.
Comment 109 Swamp Workflow Management 2018-09-25 16:25:19 UTC
SUSE-SU-2018:2858-1: An update that solves 22 vulnerabilities and has 96 fixes is now available.

Category: security (important)
Bug References: 1012382,1015342,1015343,1017967,1019695,1019699,1020412,1021121,1022604,1024361,1024365,1024376,1027968,1030552,1033962,1042286,1048317,1050431,1053685,1055014,1056596,1062604,1063646,1064232,1065364,1066223,1068032,1068075,1069138,1078921,1080157,1083663,1085042,1085536,1085539,1086457,1087092,1089066,1090888,1091171,1091860,1092903,1096254,1096748,1097105,1098253,1098822,1099597,1099810,1099811,1099813,1099832,1099844,1099845,1099846,1099849,1099863,1099864,1099922,1099999,1100000,1100001,1100132,1101822,1101841,1102346,1102486,1102517,1102715,1102797,1103269,1103445,1104319,1104485,1104494,1104495,1104683,1104897,1105271,1105292,1105322,1105392,1105396,1105524,1105536,1105769,1106016,1106105,1106185,1106229,1106271,1106275,1106276,1106278,1106281,1106283,1106369,1106509,1106511,1106594,1106697,1106929,1106934,1106995,1107060,1107078,1107319,1107320,1107689,1107735,1107966,963575,966170,966172,969470,969476,969477,970506
CVE References: CVE-2018-10876,CVE-2018-10877,CVE-2018-10878,CVE-2018-10879,CVE-2018-10880,CVE-2018-10881,CVE-2018-10882,CVE-2018-10883,CVE-2018-10902,CVE-2018-10938,CVE-2018-10940,CVE-2018-1128,CVE-2018-1129,CVE-2018-12896,CVE-2018-13093,CVE-2018-13094,CVE-2018-13095,CVE-2018-15572,CVE-2018-16658,CVE-2018-6554,CVE-2018-6555,CVE-2018-9363
Sources used:
SUSE Linux Enterprise Software Development Kit 12-SP3 (src):    kernel-docs-azure-4.4.155-4.16.1
SUSE Linux Enterprise Server 12-SP3 (src):    kernel-azure-4.4.155-4.16.1, kernel-source-azure-4.4.155-4.16.1, kernel-syms-azure-4.4.155-4.16.1
Comment 110 Swamp Workflow Management 2018-09-25 19:22:17 UTC
SUSE-SU-2018:2862-1: An update that solves 12 vulnerabilities and has 83 fixes is now available.

Category: security (important)
Bug References: 1012382,1015342,1015343,1017967,1019695,1019699,1020412,1021121,1022604,1024361,1024365,1024376,1027968,1030552,1031492,1033962,1042286,1048317,1050431,1053685,1055014,1056596,1062604,1063646,1064232,1066223,1068032,1068075,1069138,1078921,1080157,1083663,1085042,1085536,1085539,1087092,1089066,1090888,1092903,1096748,1097105,1098822,1099597,1099810,1099832,1099922,1099999,1100000,1100001,1100132,1102346,1102486,1102517,1104485,1104683,1105271,1105296,1105322,1105323,1105392,1105396,1105524,1105536,1105769,1106016,1106105,1106185,1106191,1106229,1106271,1106275,1106276,1106278,1106281,1106283,1106369,1106509,1106511,1106697,1106929,1106934,1106995,1107060,1107078,1107319,1107320,1107689,1107735,1107966,963575,966170,966172,969470,969476,969477
CVE References: CVE-2018-10902,CVE-2018-10938,CVE-2018-1128,CVE-2018-1129,CVE-2018-12896,CVE-2018-13093,CVE-2018-13094,CVE-2018-13095,CVE-2018-15572,CVE-2018-16658,CVE-2018-6554,CVE-2018-6555
Sources used:
SUSE Linux Enterprise Real Time Extension 12-SP3 (src):    kernel-rt-4.4.155-3.23.1, kernel-rt_debug-4.4.155-3.23.1, kernel-source-rt-4.4.155-3.23.1, kernel-syms-rt-4.4.155-3.23.1