Bugzilla – Bug 1119532
After HBA reset all multipath devices are getting failed and IO stops with Multi Queue enabled
Last modified: 2022-03-04 21:09:32 UTC
Steps To Reproduce Bug - 1. Boot SLES12 SP4 and map 20 data luns from 3par targets. 2. Enable Multi Queue (scsi_mod.use_blk_mq=1 dm_mod.use_blk_mq=1) and reboot the system. 3. Verfied that MQ is enabled using below cli commands. linux:~ # cat /sys/module/scsi_mod/parameters/use_blk_mq Y linux:~ # cat /sys/class/scsi_host/host0/use_blk_mq 1 linux:~ # 4. Start IO using medusa tool. 5. Perform HBA reset and during this time the traffic is stopped and all the multipath devices are in faulty state. 6. multipath -ll cli is hanged after the IO failure. Note: With Single Queue this issue is not seen. IO is running fine during HBA reset. #### Initial Analysis ##### After reset of the port, driver updates SCSI layer on fc_port is online by calling "fc_remote_port_add()". The enteries in sysfs fc_remote_ports shows port_state as online. Had enabled logging at SCSI level using the following command "scsi_logging_level -s --mlcomplete 1 -T 7 -E 5 -S 7 -I 0 -a 0". # After Reset when LPFC updates SCSI layer when rport is online. SCSI stack is ONLY sending INQUIRY and REPORT_LUNS. We don't see any other commands being sent. # Target responds fine for these command. 2018-12-14T16:07:21.448998+05:30 linux kernel: [ 789.448821] scsi 7:0:0:0: scsi scan: INQUIRY successful with code 0x0 2018-12-14T16:07:21.449016+05:30 linux kernel: [ 789.448834] scsi 7:0:0:0: scsi scan: INQUIRY pass 2 length 172 2018-12-14T16:07:21.449019+05:30 linux kernel: [ 789.448855] lpfc 0000:08:00.0: 0:(0):3322 FCP cmd x12 <0/0> sid: x130900 did: x1a000 oxid: x102 Data: x4 x1408 x7 x5 9a cmd : ffff913b94881448 2018-12-14T16:07:21.449029+05:30 linux kernel: [ 789.448944] scsi 7:0:0:0: scsi scan: INQUIRY successful with code 0x0 2018-12-14T16:07:21.449031+05:30 linux kernel: [ 789.448954] scsi 7:0:0:0: scsi scan: peripheral device type of 31, no device added 2018-12-14T16:07:21.476076+05:30 linux kernel: [ 789.472164] scsi 7:0:0:0: scsi scan: Sending REPORT LUNS to (try 0) 2018-12-14T16:07:21.476094+05:30 linux kernel: [ 789.472182] lpfc 0000:08:00.0: 0:(0):3322 FCP cmd xa0 <0/0> sid: x130900 did: x1a000 oxid: x10e Data: x4 x1414 x9 x7 9a cmd : ffff913b94881448 2018-12-14T16:07:21.476097+05:30 linux kernel: [ 789.472287] lpfc 0000:08:00.0: 0:(0):9024 FCP command xa0 failed: x0 SNS x0 x0 Data: x8 xfd0 x0 x0 x0 2018-12-14T16:07:21.476150+05:30 linux kernel: [ 789.472312] scsi 7:0:0:0: scsi scan: REPORT LUNS successful (try 0) result 0x0 2018-12-14T16:07:21.476161+05:30 linux kernel: [ 789.472317] scsi 7:0:0:0: scsi scan: REPORT LUN scan 2018-12-14T16:07:21.476164+05:30 linux kernel: [ 789.472324] sd 7:0:0:12: scsi scan: device exists on 7:0:0:12 2018-12-14T16:07:21.476166+05:30 linux kernel: [ 789.472329] sd 7:0:0:13: scsi scan: device exists on 7:0:0:13 2018-12-14T16:07:21.476168+05:30 linux kernel: [ 789.472334] sd 7:0:0:14: scsi scan: device exists on 7:0:0:14 2018-12-14T16:07:21.476169+05:30 linux kernel: [ 789.472338] sd 7:0:0:15: scsi scan: device exists on 7:0:0:15 2018-12-14T16:07:21.476171+05:30 linux kernel: [ 789.472343] ses 7:0:0:254: scsi scan: device exists on 7:0:0:254 2018-12-14T16:07:21.508109+05:30 linux kernel: [ 789.504179] scsi 7:0:1:0: scsi scan: INQUIRY pass 1 length 36 2018-12-14T16:07:21.508128+05:30 linux kernel: [ 789.504197] lpfc 0000:08:00.0: 0:(0):3322 FCP cmd x12 <1/0> sid: x130900 did: x1a100 oxid: x10d Data: x5 x1413 x7 x5 9a cmd : ffff913b87e59948 2018-12-14T16:07:21.508131+05:30 linux kernel: [ 789.504301] scsi 7:0:1:0: scsi scan: INQUIRY successful with code 0x0 2018-12-14T16:07:21.508135+05:30 linux kernel: [ 789.504309] scsi 7:0:1:0: scsi scan: INQUIRY pass 2 length 172 2018-12-14T16:07:21.508150+05:30 linux kernel: [ 789.504327] lpfc 0000:08:00.0: 0:(0):3322 FCP cmd x12 <1/0> sid: x130900 did: x1a100 oxid: x11a Data: x5 x1420 x7 x5 9a cmd : ffff913b94881448 2018-12-14T16:07:21.508153+05:30 linux kernel: [ 789.504410] scsi 7:0:1:0: scsi scan: INQUIRY successful with code 0x0 2018-12-14T16:07:21.508155+05:30 linux kernel: [ 789.504417] scsi 7:0:1:0: scsi scan: peripheral device type of 31, no device added 2018-12-14T16:07:21.524114+05:30 linux kernel: [ 789.520193] scsi 7:0:1:0: scsi scan: Sending REPORT LUNS to (try 0) 2018-12-14T16:07:21.524133+05:30 linux kernel: [ 789.520211] lpfc 0000:08:00.0: 0:(0):3322 FCP cmd xa0 <1/0> sid: x130900 did: x1a100 oxid: x126 Data: x5 x142c x9 x7 9a cmd : ffff913b94881448 2018-12-14T16:07:21.524136+05:30 linux kernel: [ 789.520366] lpfc 0000:08:00.0: 0:(0):9024 FCP command xa0 failed: x0 SNS x0 x0 Data: x8 xfd0 x0 x0 x0 2018-12-14T16:07:21.524140+05:30 linux kernel: [ 789.520413] scsi 7:0:1:0: scsi scan: REPORT LUNS successful (try 0) result 0x0 2018-12-14T16:07:21.524155+05:30 linux kernel: [ 789.520417] scsi 7:0:1:0: scsi scan: REPORT LUN scan 2018-12-14T16:07:21.524157+05:30 linux kernel: [ 789.520422] sd 7:0:1:12: scsi scan: device exists on 7:0:1:12 2018-12-14T16:07:21.524159+05:30 linux kernel: [ 789.520425] sd 7:0:1:13: scsi scan: device exists on 7:0:1:13 2018-12-14T16:07:21.524161+05:30 linux kernel: [ 789.520428] sd 7:0:1:14: scsi scan: device exists on 7:0:1:14 2018-12-14T16:07:21.524163+05:30 linux kernel: [ 789.520430] sd 7:0:1:15: scsi scan: device exists on 7:0:1:15 2018-12-14T16:07:21.524164+05:30 linux kernel: [ 789.520434] ses 7:0:1:254: scsi scan: device exists on 7:0:1:254 Output of the command "lsscsi" [7:0:0:12] disk 3PARdata VV 3312 /dev/sdd [7:0:0:13] disk 3PARdata VV 3312 /dev/sde [7:0:0:14] disk 3PARdata VV 3312 /dev/sdf [7:0:0:15] disk 3PARdata VV 3312 /dev/sdg [7:0:0:254] enclosu 3PARdata SES 3312 - [7:0:1:12] disk 3PARdata VV 3312 /dev/sdh [7:0:1:13] disk 3PARdata VV 3312 /dev/sdi [7:0:1:14] disk 3PARdata VV 3312 /dev/sdj [7:0:1:15] disk 3PARdata VV 3312 /dev/sdk Multipath layer shows the paths as faulty after the reset. This leads to IO's not resuming. The sysfs entries in fc_remote_ports shows online, which points to driver updating upper layer about link up after reset command.
Please attach the full kernel message log.
Created attachment 792753 [details] var/log/message file of server and CLI ran. Provided information of kernel logs.
Created attachment 792817 [details] var/log/message file of server.
Hannes, we've attached the info, please follow up. thank you Laurie
Hannes, Would you please follow up, we are hitting this consistently in our test environment. thx Laurie
Hmm. This is a bit worrying: 2018-12-14T21:13:24.224987+05:30 linux multipathd[1518]: 360002ac000000000000040180001ede9: sdl - tur checker timed out which means that the TEST UNIT READY command didn't return in time. And slightly earlier we see: [ 379.259885] lpfc 0000:08:00.0: 0:2538 Received frame rctl:x22, type:x1, frame Data:22130900 00fffc13 01290000 28000000 1e09ffff 00000000 00000000 [ 379.259895] lpfc 0000:08:00.0: 0:(0):0929 FIND node DID Data: xffff96ad60cb8800 xfffc13 x0 x8001405 ffff96aa95b09c00 [ 379.259900] lpfc 0000:08:00.0: 0:(0):0112 ELS command x18 received from NPORT xfffc13 Data: x20 x800110 x130900 x130900 [ 379.259904] lpfc 0000:08:00.0: 0:(0):2422 ELS RDP Request dec len 12 tag x3 port_id 196608 len 4 [ 379.939911] lpfc 0000:08:00.0: 0:(0):0117 Xmit ELS response x2 to remote NPORT xfffc13 I/O tag: x1363, size: x154 port_state x20 rpi x5 fc_flag x800110 [ 379.939916] lpfc 0000:08:00.0: 0:(0):2171 Xmit RDP response tag x1363 xri xffff, did xfffc13, nlp_flag x0, nlp_state x8, rpi x5 [ 379.939958] lpfc 0000:08:00.0: 0:(0):0110 ELS response tag x1363 completes Data: x0 x0 x0 xfffc13 x0 x8 x5 [ 386.780110] lpfc 0000:08:00.1: 1:(0):0929 FIND node DID Data: xffff96ad5ba12800 x1a100 x80000004 x7000804 ffff96ad5bea3800 [ 387.224982] device-mapper: multipath: Failing path 8:176. It seems to indicate to me that the 'TEST UNIT READY' command haven't been completed/aborted correctly, causing the whole setup to get out of sync. Can you please check if the 'sdl' device in your log resides on the rport which received the LOGO above?
Hmm. Probably not. Ignore my previous comment.
Hannes - is there an analysis of this issue? Laurie
http://beta.suse.com/private/mwilck/bsc1119532 I've uploaded a test kernel there. Please check if the error occurs with that kernel, too.
Would it be possible to get similar logs for the (good) SQ case? Ideally, with both lpfc and scsi logging enabled.
Aborted commands (sense key 0b) start at 21:11:40 > 2018-12-14T21:11:40.652033+05:30 linux kernel: [ 283.651776] lpfc 0000:08:00.1: 1:(0):9024 FCP command x2a failed: x2 SNS x70000b00 x4bc40000 Data: x2 x0 x12 x0 x0 Note: I see no TUR (x00) commands logged as failed / aborted. Nport state transitions at 21:12:40, a minute later > 2018-12-14T21:12:42.288182+05:30 linux kernel: [ 345.284148] lpfc 0000:08:00.0: 0:(0):0904 NPort state transition ffff96ad27c55000 x01a000 x80000004: MAPPED -> NPR 10s later eventually the multipath checker detects failures > 2018-12-14T21:12:53.200716+05:30 linux kernel: [ 356.200679] device-mapper: multipath: Failing path 8:32. > 2018-12-14T21:12:53.200716+05:30 linux kernel: [ 356.200679] device-mapper: multipath: Failing path 8:32. > 2018-12-14T21:12:53.200708+05:30 linux multipathd[1518]: checker failed path 8:32 in map 360002ac000000000000040170001ede9 Just a second later at 12:12:54, ports are coming back to MAPPED state > 2018-12-14T21:12:54.156302+05:30 linux kernel: [ 357.155454] lpfc 0000:08:00.0: 0:(0):0904 NPort state transition ffff96ad27c55800 x01a100 x80000004: PRLI -> MAPPED 30s later, the TUR checker is timing out > 2018-12-14T21:13:24.224987+05:30 linux multipathd[1518]: 360002ac000000000000040180001ede9: sdl - tur checker timed out > 2018-12-14T21:13:24.225255+05:30 linux multipathd[1518]: checker failed path 8:176 in map 360002ac000000000000040180001ede9 > 2018-12-14T21:13:24.225404+05:30 linux multipathd[1518]: 360002ac000000000000040180001ede9: remaining active paths: 1 > 2018-12-14T21:13:24.225574+05:30 linux multipathd[1518]: sdl: mark as failed > 2018-12-14T21:13:24.228033+05:30 linux kernel: [ 387.224982] device-mapper: multipath: Failing path 8:176. And we see these messages, probably for PATH_FAILED events: > 2018-12-14T21:13:53.822063+05:30 linux systemd-udevd[994]: seq 3722 '/devices/virtual/block/dm-4' is taking a long time > 2018-12-14T21:13:53.822303+05:30 linux systemd-udevd[994]: seq 3728 '/devices/virtual/block/dm-11' is taking a long time The same events are killed later: > 2018-12-14T21:15:53.823066+05:30 linux systemd-udevd[994]: worker [3941] terminated by signal 9 (Killed) > 2018-12-14T21:15:53.823190+05:30 linux systemd-udevd[994]: worker [3941] failed while handling '/devices/virtual/block/dm-4' IO is obviously hanging on the reappeared devices. It could be the problem we fixed for bug 1119843 (comment 10), but I fail to understand why that would be related to multiqueue. @John, in the situation where tasks are blocked (30-60s after the controller reset has finished), could you run "echo w >/proc/sysrq-trigger"? It might also be helpful to set "verbosity 3" in /etc/multipath.conf.
Martin, We will try the test with the kernel you have provided in comment-10. Logging will be enabled for LPFC, SML and Multipath.
Hi Martin, We have updated to latest kernel image provided by you and retested it, but still we see issue during MQ enabled. Configured /etc/multipath.conf with verbosity 3 and captured logs. Shared the logs to the box location as the log size is large than. https://broadcom.ent.box.com/folder/70849907740 Thanks, Basavaraja.G
Basavara, I'm very sorry, this slipped through, and now the Box link above seems to have expired. Could you please upload it again?
Hi Martin, I have uploaded the logs and shared the box link to you. Let me know if your able to access it. Thanks Basava
(In reply to Basavaraja Gummanur from comment #16) > I have uploaded the logs and shared the box link to you. Let me know if your > able to access it. Yes. Thanks a lot.
2nd "multipath -ll" run hangs here: > Mar 22 05:00:03 | sdc: 13054 cyl, 255 heads, 63 sectors/track, start at 0 This would normally be followed by the serial no: > Mar 22 04:55:32 | sdc: 13054 cyl, 255 heads, 63 sectors/track, start at 0 > Mar 22 04:55:32 | sdc: serial = SGH733V48P The multipath command is in uninterruptible sleep in sg_io(): > 2019-03-22T05:00:08.644420+05:30 linux kernel: [ 455.644116] wait_for_completion_io+0xa3/0x110 > 2019-03-22T05:00:08.644421+05:30 linux kernel: [ 455.644119] ? wake_up_q+0x70/0x70 > 2019-03-22T05:00:08.644424+05:30 linux kernel: [ 455.644126] blk_execute_rq+0x73/0x80 > 2019-03-22T05:00:08.644425+05:30 linux kernel: [ 455.644129] sg_io+0x162/0x390 > 2019-03-22T05:00:08.644426+05:30 linux kernel: [ 455.644132] ? dev_printk+0x4d/0x50 > 2019-03-22T05:00:08.644427+05:30 linux kernel: [ 455.644136] scsi_cmd_ioctl+0x297/0x420 > 2019-03-22T05:00:08.644427+05:30 linux kernel: [ 455.644146] ? scsi_block_when_processing_errors+0xcd/0xd0 [scsi_mod] > 2019-03-22T05:00:08.644428+05:30 linux kernel: [ 455.644151] sd_ioctl+0xca/0x1a0 [sd_mod] > 2019-03-22T05:00:08.644433+05:30 linux kernel: [ 455.644154] blkdev_ioctl+0x872/0x910 > 2019-03-22T05:00:08.644434+05:30 linux kernel: [ 455.644158] block_ioctl+0x3d/0x40 > 2019-03-22T05:00:08.644435+05:30 linux kernel: [ 455.644161] do_vfs_ioctl+0x92/0x5e0 > 2019-03-22T05:00:08.644436+05:30 linux kernel: [ 455.644166] ? vfs_write+0x14b/0x1a0 > 2019-03-22T05:00:08.644436+05:30 linux kernel: [ 455.644169] SyS_ioctl+0x74/0x80 > 2019-03-22T05:00:08.644437+05:30 linux kernel: [ 455.644177] do_syscall_64+0x74/0x150 So multipath is trying to run an INQUIRY for VPD 80 via sg_ioctl, and hangs waiting for a response. This happens only if the inquiry data aren't found in sysfs. This is at time 455. At this time, except "multipath", there are 84 additional multipathd threads and 1 kworker in "D" state. The multipathd threads are also waiting for I/O completion on sg_ioctl. The kworker is scanning a FC target: > 2019-03-22T05:00:08.628920+05:30 linux kernel: [ 455.624350] kworker/u144:2 D 0 1124 2 0x00000000 > 2019-03-22T05:00:08.628936+05:30 linux kernel: [ 455.624362] Workqueue: scsi_wq_8 fc_scsi_scan_rport [scsi_transport_fc] > 2019-03-22T05:00:08.628940+05:30 linux kernel: [ 455.624363] Call Trace: > 2019-03-22T05:00:08.628973+05:30 linux kernel: [ 455.624422] scsi_report_lun_scan+0x183/0x410 [scsi_mod] > 2019-03-22T05:00:08.628976+05:30 linux kernel: [ 455.624432] __scsi_scan_target+0xeb/0x220 [scsi_mod] > 2019-03-22T05:00:08.628979+05:30 linux kernel: [ 455.624441] scsi_scan_target+0xff/0x110 [scsi_mod] > 2019-03-22T05:00:08.628982+05:30 linux kernel: [ 455.624447] fc_scsi_scan_rport+0xac/0xb0 [scsi_transport_fc] The same situation is observed 60s earlier, just that there is no multipath process yet, and the number of multipathd processes hanging is only 33. Again, we have one kworker scanning the target, but not the same one: > 2019-03-22T04:59:08.920570+05:30 linux kernel: [ 395.919262] kworker/u144:0 D > 2019-03-22T04:59:08.920575+05:30 linux kernel: [ 395.919278] 0 5 2 0x00000000 > 2019-03-22T04:59:08.920577+05:30 linux kernel: [ 395.919295] Workqueue: scsi_wq_7 fc_scsi_scan_rport [scsi_transport_fc] > 2019-03-22T04:59:08.920651+05:30 linux kernel: [ 395.919487] scsi_report_lun_scan+0x183/0x410 [scsi_mod] > 2019-03-22T04:59:08.920657+05:30 linux kernel: [ 395.919513] __scsi_scan_target+0xeb/0x220 [scsi_mod] > 2019-03-22T04:59:08.920659+05:30 linux kernel: [ 395.919532] scsi_scan_target+0xff/0x110 [scsi_mod] > 2019-03-22T04:59:08.920666+05:30 linux kernel: [ 395.919548] fc_scsi_scan_rport+0xac/0xb0 [scsi_transport_fc]
(In reply to Martin Wilck from comment #18) > > The same situation is observed 60s earlier, just that there is no multipath > process yet, and the number of multipathd processes hanging is only 33. At 455s (05:00:08), all multipathd threads that had been hanging at 395 are still hanging, and a lot more.
Can you please add a supportconfig of the affected system?
HBA reset on 10:00:00:00:c9:c5:bf:2 (presumably host 7): > 2019-03-22T04:58:48.368107+05:30 linux kernel: [ 375.367348] sd 7:0:0:0: scsi scan: device exists on 7:0:0:0 > 2019-03-22T04:58:48.368110+05:30 linux kernel: [ 375.367362] sd 7:0:0:0: scsi scan: Sending REPORT LUNS to (try 0) This command never seems to complete, otherwise we'd have seen a "REPORT LUNS failed" or "REPORT LUNS successful" message. The timeout for REPORT LUNS is 30s. Note also the hanging kworkers in scsi_report_lun_scan() in comment 18. What we do see next on this host is: > 2019-03-22T04:58:48.891407+05:30 linux kernel: [ 375.891368] sd 7:0:1:1: [sdy] sd_ioctl: disk=sdy, cmd=0x2285 > 2019-03-22T04:58:49.001973+05:30 linux multipathd[3157]: sdy: path state = running > 2019-03-22T04:58:49.002073+05:30 linux multipathd[3157]: sdy: get_state > 2019-03-22T04:58:49.002174+05:30 linux multipathd[3157]: 65:128: tur checker starting up > 2019-03-22T04:58:49.002275+05:30 linux multipathd[3157]: 65:128: tur checker still running This is an sg_ioctl command issued from multipathd. multipathd sees the device as "running" is sysfs although it's not currently present. The TUR command that it sends is never going to return. More precisely, it seems the command is never even *sent*, otherwise we'd have seen 'CDB: Test Unit Ready' output from scsi_log_send(). The same holds for the "REPORT LUNS" command above - there's no log trace of it ever being sent to the device.
The kworker stack is, again: > 2019-03-22T04:59:08.920570+05:30 linux kernel: [ 395.919262] kworker/u144:0 D > 2019-03-22T04:59:08.920575+05:30 linux kernel: [ 395.919278] 0 5 2 0x00000000 > 2019-03-22T04:59:08.920577+05:30 linux kernel: [ 395.919295] Workqueue: scsi_wq_7 fc_scsi_scan_rport [scsi_transport_fc] > > 2019-03-22T04:59:08.920624+05:30 linux kernel: [ 395.919389] io_schedule_timeout+0x19/0x40 > 2019-03-22T04:59:08.920626+05:30 linux kernel: [ 395.919397] wait_for_completion_io+0xa3/0x110 > 2019-03-22T04:59:08.920634+05:30 linux kernel: [ 395.919415] ? wake_up_q+0x70/0x70 > 2019-03-22T04:59:08.920636+05:30 linux kernel: [ 395.919425] blk_execute_rq+0x73/0x80 > 2019-03-22T04:59:08.920643+05:30 linux kernel: [ 395.919461] scsi_execute+0xb6/0x1e0 [scsi_mod] > 2019-03-22T04:59:08.920651+05:30 linux kernel: [ 395.919487] scsi_report_lun_scan+0x183/0x410 [scsi_mod] Contrary to what I said in comment 18, this exact stack still appears 60s later, for the same worker: > 2019-03-22T05:00:08.621750+05:30 linux kernel: [ 455.621154] kworker/u144:0 D > 2019-03-22T05:00:08.621766+05:30 linux kernel: [ 455.621571] Workqueue: scsi_wq_7 fc_scsi_scan_rport [scsi_transport_fc] > 2019-03-22T05:00:08.621803+05:30 linux kernel: [ 455.621660] io_schedule_timeout+0x19/0x40 > [...] At this time, the 30s request timeout should have fired but I don't see that (yet).
(In reply to Martin Wilck from comment #22) > At this time, the 30s request timeout should have fired but I don't see that > (yet). The timer is set in scsi_mq_prep_fn(). If it doesn't fire, the prep_fn was likely never run, which means that in all probability, the queue hadn't been run after inserting the command. That also matches the observation the sending of the command hadn't been logged. How is it possible that queue wasn't run? > static void __blk_mq_delay_run_hw_queue() { > ... > if (unlikely(blk_mq_hctx_stopped(hctx))) > return; blk_mq_hctx_stopped() is true after setting blk_mq_stop_hw_queue(), which (for SCSI devices), is only called from scsi_target_block(). scsi_target_block() is called from fc_remote_port_delete(). However this is not blk-mq specific. I still don't see why this problem would only occur with blk-mq.
(In reply to Martin Wilck from comment #23) > blk_mq_hctx_stopped() is true after setting blk_mq_stop_hw_queue(), which > (for SCSI devices), is only called from scsi_target_block(). > scsi_target_block() is called from fc_remote_port_delete(). ... and it would be unblocked either when the dev_loss_tmo is reached, or when the remote port is added again.
(In reply to Martin Wilck from comment #23) > However this is not blk-mq specific. I still don't see why this problem > would only occur with blk-mq. Indeed, in the non-mq case we have the same logic: > void __blk_run_queue(struct request_queue *q) > { > if (unlikely(blk_queue_stopped(q))) > return;
(In reply to Martin Wilck from comment #20) > Can you please add a supportconfig of the affected system? Hi Martin, We are able to collect the supportconfig before we start the IO with MQ enable. I have uploaded the same in box location below. https://broadcom.ent.box.com/folder/70849907740 After running IO we are not able to collect the support dump as it stops collecting at disk IO. Attached the screen shot in attachment section. Thanks, Basava
Created attachment 808007 [details] SupportconfigafterIORun
SupportConfig attached. in comment-27.
Adapter reset host7 / 05:00.0 > 2019-03-22T04:58:44.026920+05:30 linux kernel: [ 371.026683] lpfc 0000:05:00.0: 0:(0):3050 lpfc_board_mode set to offline > ... > 2019-03-22T04:58:46.500285+05:30 linux kernel: [ 373.500257] lpfc 0000:05:00.0: 0:(0):3050 lpfc_board_mode set to online > 2019-03-22T04:58:48.264048+05:30 linux kernel: [ 375.262221] lpfc 0000:05:00.0: 0:1303 Link Up Event x1 received Data: x1 x1 x20 x0 x0 x0 0 > ... > 2019-03-22T04:58:48.368099+05:30 linux kernel: [ 375.367217] lpfc 0000:05:00.0: 0:(0):0103 PRLI completes to NPort x01ab00 Data: x0 x142a00 x1 x0 > 2019-03-22T04:58:48.416349+05:30 linux kernel: [ 375.416208] lpfc 0000:05:00.0: 0:(0):0103 PRLI completes to NPort x01a100 Data: x0 x142a00 x0 x0 Hanging REPORT LUNS command is seen 20s later.
Interestingly, SCSI commands sent *after* the host reset still seem to complete successfully. This is the last one: > 2019-03-22T04:58:44.654436+05:30 linux kernel: [ 371.654368] sd 7:0:1:4: [sdab] tag#4147 Send: scmd 0xffff8eefec15e848 > 2019-03-22T04:58:44.654438+05:30 linux kernel: [ 371.654371] sd 7:0:1:4: [sdab] tag#4147 CDB: Write(10) 2a 00 00 00 1e 40 00 01 c0 00 > [...] > 2019-03-22T04:58:45.065813+05:30 linux kernel: [ 372.064507] sd 7:0:1:4: [sdab] tag#4147 Done: SUCCESS Result: hostbyte=DID_OK driverbyte=DRIVER_OK > 2019-03-22T04:58:45.065821+05:30 linux kernel: [ 372.064511] sd 7:0:1:4: [sdab] tag#4147 CDB: Write(10) 2a 00 00 00 1e 40 00 01 c0 00
Other commands fail with NEEDS_RETRY / hostbyte=DID_IMM_RETRY. This starts here: > 2019-03-22T04:58:44.036235+05:30 linux kernel: [ 371.036117] sd 7:0:0:5: [sdh] tag#6687 Done: NEEDS_RETRY Result: hostbyte=DID_IMM_RETRY driverbyte=DRIVER_OK and ends here, about 1.2s later: > 2019-03-22T04:58:45.212106+05:30 linux kernel: [ 372.208071] sd 7:0:1:20: [sdca] tag#4117 Done: NEEDS_RETRY Result: hostbyte=DID_IMM_RETRY driverbyte=DRIVER_OK This is the last completion of a SCSI command on host7 logged. The last command *sent* to the controller is sent right after this completion: Some c > 2019-03-22T04:58:45.212102+05:30 linux kernel: [ 372.208062] sd 7:0:1:20: [sdca] tag#4117 Send: scmd 0xffff8eefe46a2248 > 2019-03-22T04:58:45.212104+05:30 linux kernel: [ 372.208066] sd 7:0:1:20: [sdca] tag#4117 CDB: Read(10) 28 00 00 00 20 00 00 02 10 00 > 2019-03-22T04:58:45.212111+05:30 linux kernel: [ 372.208081] sd 7:0:1:20: [sdca] tag#4117 Inserting command ffff8eefe46a2248 into mlqueue This is the last sign of anything happening on this SCSI host at the midlayer and below. Shortly after, the lpfc controller logs > 2019-03-22T04:58:45.272170+05:30 linux kernel: [ 372.268233] lpfc 0000:05:00.0: 0:(0):2819 Abort outstanding I/O on NPort xfffc14 Data: x0 x0 x0 So I'd expect the SCSI commands to be aborted on this host, but that never seems to happen. Timeouts or error handler actions aren't observed, either. Some commands are repeated in quick succession with DID_IMM_RETRY. For example, tag#4117 on /dev/sdp gets 14 times NEED_RETRY in the 1.2s period noted above.
No other SCSI commands except READ(10) and WRITE(10) are logged after 04:56:58 (264.933). The next time the multipathd TUR checker was scheduled to run seems to have been after the host reset, thus the TUR commands are never logged.
I have a theory. Our kernel includes these commits: f660174e8 blk-mq: use the introduced blk_mq_unquiesce_queue() 66483a4a9f scsi: Introduce scsi_start_queue() 551eb598e scsi: Split scsi_internal_device_block() They got into 4.13-rc1 via the "block" and "scsi" trees, respectively. They were merged upstream in 9031114 Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi which is a non-trivial merge that replaced blk_mq_stop_hw_queues() with blk_mq_quiesce_queue_nowait() and blk_mq_start_stopped_hw_queues() with blk_mq_unquiesce_queue(). The changes from 9031114 were missing in our tree. This might indeed cause the quiesced queue not to be un-quiesced again, and only in the blk-mq case, as reported here.
New test kernel uploaded to http://beta.suse.com/private/mwilck/bsc1119532: kernel-default-4.12.14-2.1.g79fbabb.x86_64 Please report back. In case of failure, please provide similar logs like in comment 16.
Martin, We will run the test with new kernel and update. Thanks
Hi All, Already this kernel patch is available on the present OS where we are seeing this issue. Find the attachment of message seen during kernel rpm installation. Thanks, Basava
Created attachment 809076 [details] kernel update screenshot
(In reply to Basavaraja Gummanur from comment #36) > Already this kernel patch is available on the present OS where we are seeing > this issue. No, the message from zypper is misleading. Please install with "zypper install --force". This kind of version confusion can happen with unofficial test kernels.
Hi All, This issue is not seen after updating the kernel(kernel-default-4.12.14-2.1.g79fbabb.x86_64) provided by Suse. Hence closing this issue. IO runs fine with MQ enabled and performed port toggle. Thanks, Basava
Will keep this Bug open till an official drop with the fix is done.
Thanks for the feedback. Glad this helped. Hannes has pushed the fix to the SLE15 branch, thus it will show up in the forthcoming kernel maintenance update(s) of all affected SUSE distributions.
This is an autogenerated message for OBS integration: This bug (1119532) was mentioned in https://build.opensuse.org/request/show/714223 15.0 / kernel-source
SUSE-SU-2019:1829-1: An update that solves 11 vulnerabilities and has 71 fixes is now available. Category: security (important) Bug References: 1051510,1071995,1088047,1094555,1098633,1106383,1106751,1109137,1114279,1119532,1120423,1124167,1127155,1128432,1128902,1128910,1131645,1132154,1132390,1133401,1133738,1134303,1134395,1135296,1135556,1135642,1136157,1136598,1136922,1136935,1137103,1137194,1137429,1137625,1137728,1137884,1137995,1137996,1137998,1137999,1138000,1138002,1138003,1138005,1138006,1138007,1138008,1138009,1138010,1138011,1138012,1138013,1138014,1138015,1138016,1138017,1138018,1138019,1138291,1138293,1138374,1138375,1138589,1138719,1139771,1139782,1139865,1140133,1140328,1140405,1140424,1140428,1140575,1140577,1140637,1140658,1140715,1140719,1140726,1140727,1140728,1140814 CVE References: CVE-2018-16871,CVE-2018-20836,CVE-2019-10126,CVE-2019-10638,CVE-2019-10639,CVE-2019-11599,CVE-2019-12380,CVE-2019-12456,CVE-2019-12614,CVE-2019-12818,CVE-2019-12819 Sources used: SUSE Linux Enterprise Module for Public Cloud 15 (src): kernel-azure-4.12.14-5.33.1, kernel-source-azure-4.12.14-5.33.1, kernel-syms-azure-4.12.14-5.33.1 SUSE Linux Enterprise Module for Open Buildservice Development Tools 15-SP1 (src): kernel-azure-4.12.14-5.33.1, kernel-source-azure-4.12.14-5.33.1, kernel-syms-azure-4.12.14-5.33.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
This is an autogenerated message for OBS integration: This bug (1119532) was mentioned in https://build.opensuse.org/request/show/715440 15.1 / kernel-source
SUSE-SU-2019:1854-1: An update that solves 6 vulnerabilities and has 69 fixes is now available. Category: security (important) Bug References: 1051510,1071995,1088047,1098633,1103990,1103991,1103992,1106383,1109837,1111666,1112374,1114685,1119113,1119532,1120423,1125703,1128902,1130836,1131645,1132390,1133401,1133738,1134303,1134395,1135556,1135642,1135897,1136161,1136264,1136343,1136935,1137625,1137728,1138879,1139712,1139751,1139771,1139865,1140133,1140228,1140328,1140405,1140424,1140428,1140454,1140463,1140575,1140577,1140637,1140658,1140715,1140719,1140726,1140727,1140728,1140814,1140887,1140888,1140889,1140891,1140893,1140948,1140954,1140955,1140956,1140957,1140958,1140959,1140960,1140961,1140962,1140964,1140971,1140972,1140992 CVE References: CVE-2018-20836,CVE-2019-10126,CVE-2019-10638,CVE-2019-10639,CVE-2019-11599,CVE-2019-13233 Sources used: SUSE Linux Enterprise Workstation Extension 15-SP1 (src): kernel-default-4.12.14-197.10.1 SUSE Linux Enterprise Module for Open Buildservice Development Tools 15-SP1 (src): dtb-aarch64-4.12.14-197.10.1, kernel-debug-4.12.14-197.10.1, kernel-default-4.12.14-197.10.1, kernel-docs-4.12.14-197.10.1, kernel-kvmsmall-4.12.14-197.10.1, kernel-obs-qa-4.12.14-197.10.1, kernel-source-4.12.14-197.10.1, kernel-vanilla-4.12.14-197.10.1, kernel-zfcpdump-4.12.14-197.10.1 SUSE Linux Enterprise Module for Legacy Software 15-SP1 (src): kernel-default-4.12.14-197.10.1 SUSE Linux Enterprise Module for Development Tools 15-SP1 (src): kernel-docs-4.12.14-197.10.1, kernel-obs-build-4.12.14-197.10.1, kernel-source-4.12.14-197.10.1, kernel-syms-4.12.14-197.10.1 SUSE Linux Enterprise Module for Basesystem 15-SP1 (src): kernel-default-4.12.14-197.10.1, kernel-source-4.12.14-197.10.1, kernel-zfcpdump-4.12.14-197.10.1 SUSE Linux Enterprise High Availability 15-SP1 (src): kernel-default-4.12.14-197.10.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-SU-2019:1855-1: An update that solves 12 vulnerabilities and has 73 fixes is now available. Category: security (important) Bug References: 1051510,1061840,1065600,1071995,1088047,1094555,1098633,1106383,1106751,1109137,1114279,1119532,1120423,1124167,1127155,1128432,1128902,1128910,1131645,1132154,1132390,1133401,1133738,1134303,1134395,1135296,1135556,1135642,1136157,1136598,1136922,1136935,1137103,1137194,1137429,1137625,1137728,1137884,1137995,1137996,1137998,1137999,1138000,1138002,1138003,1138005,1138006,1138007,1138008,1138009,1138010,1138011,1138012,1138013,1138014,1138015,1138016,1138017,1138018,1138019,1138291,1138293,1138374,1138375,1138589,1138719,1139751,1139771,1139782,1139865,1140133,1140328,1140405,1140424,1140428,1140575,1140577,1140637,1140658,1140715,1140719,1140726,1140727,1140728,1140814 CVE References: CVE-2018-16871,CVE-2018-20836,CVE-2019-10126,CVE-2019-10638,CVE-2019-10639,CVE-2019-11478,CVE-2019-11599,CVE-2019-12380,CVE-2019-12456,CVE-2019-12614,CVE-2019-12818,CVE-2019-12819 Sources used: SUSE Linux Enterprise Workstation Extension 15 (src): kernel-default-4.12.14-150.27.1 SUSE Linux Enterprise Module for Open Buildservice Development Tools 15 (src): kernel-default-4.12.14-150.27.1, kernel-docs-4.12.14-150.27.1, kernel-obs-qa-4.12.14-150.27.1 SUSE Linux Enterprise Module for Legacy Software 15 (src): kernel-default-4.12.14-150.27.1 SUSE Linux Enterprise Module for Development Tools 15 (src): kernel-docs-4.12.14-150.27.1, kernel-obs-build-4.12.14-150.27.1, kernel-source-4.12.14-150.27.1, kernel-syms-4.12.14-150.27.1, kernel-vanilla-4.12.14-150.27.1 SUSE Linux Enterprise Module for Basesystem 15 (src): kernel-default-4.12.14-150.27.1, kernel-source-4.12.14-150.27.1, kernel-zfcpdump-4.12.14-150.27.1 SUSE Linux Enterprise High Availability 15 (src): kernel-default-4.12.14-150.27.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-SU-2019:1854-1: An update that solves 6 vulnerabilities and has 69 fixes is now available. Category: security (important) Bug References: 1051510,1071995,1088047,1098633,1103990,1103991,1103992,1106383,1109837,1111666,1112374,1114685,1119113,1119532,1120423,1125703,1128902,1130836,1131645,1132390,1133401,1133738,1134303,1134395,1135556,1135642,1135897,1136161,1136264,1136343,1136935,1137625,1137728,1138879,1139712,1139751,1139771,1139865,1140133,1140228,1140328,1140405,1140424,1140428,1140454,1140463,1140575,1140577,1140637,1140658,1140715,1140719,1140726,1140727,1140728,1140814,1140887,1140888,1140889,1140891,1140893,1140948,1140954,1140955,1140956,1140957,1140958,1140959,1140960,1140961,1140962,1140964,1140971,1140972,1140992 CVE References: CVE-2018-20836,CVE-2019-10126,CVE-2019-10638,CVE-2019-10639,CVE-2019-11599,CVE-2019-13233 Sources used: SUSE Linux Enterprise Workstation Extension 15-SP1 (src): kernel-default-4.12.14-197.10.1 SUSE Linux Enterprise Module for Open Buildservice Development Tools 15-SP1 (src): dtb-aarch64-4.12.14-197.10.1, kernel-debug-4.12.14-197.10.1, kernel-default-4.12.14-197.10.1, kernel-docs-4.12.14-197.10.1, kernel-kvmsmall-4.12.14-197.10.1, kernel-obs-qa-4.12.14-197.10.1, kernel-source-4.12.14-197.10.1, kernel-vanilla-4.12.14-197.10.1, kernel-zfcpdump-4.12.14-197.10.1 SUSE Linux Enterprise Module for Live Patching 15-SP1 (src): kernel-default-4.12.14-197.10.1, kernel-livepatch-SLE15-SP1_Update_3-1-3.3.1 SUSE Linux Enterprise Module for Legacy Software 15-SP1 (src): kernel-default-4.12.14-197.10.1 SUSE Linux Enterprise Module for Development Tools 15-SP1 (src): kernel-docs-4.12.14-197.10.1, kernel-obs-build-4.12.14-197.10.1, kernel-source-4.12.14-197.10.1, kernel-syms-4.12.14-197.10.1 SUSE Linux Enterprise Module for Basesystem 15-SP1 (src): kernel-default-4.12.14-197.10.1, kernel-source-4.12.14-197.10.1, kernel-zfcpdump-4.12.14-197.10.1 SUSE Linux Enterprise High Availability 15-SP1 (src): kernel-default-4.12.14-197.10.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-SU-2019:1851-1: An update that solves 11 vulnerabilities and has 77 fixes is now available. Category: security (important) Bug References: 1051510,1061840,1065600,1071995,1088047,1094555,1098633,1106383,1106751,1109137,1114279,1119532,1120423,1124167,1127155,1128432,1128902,1128910,1132154,1132390,1133401,1133738,1134303,1134395,1135296,1135556,1135642,1136157,1136811,1136922,1137103,1137194,1137221,1137366,1137429,1137625,1137728,1137884,1137995,1137996,1137998,1137999,1138000,1138002,1138003,1138005,1138006,1138007,1138008,1138009,1138010,1138011,1138012,1138013,1138014,1138015,1138016,1138017,1138018,1138019,1138291,1138293,1138374,1138375,1138589,1138719,1139751,1139771,1139782,1139865,1140133,1140328,1140405,1140424,1140428,1140575,1140577,1140637,1140658,1140715,1140719,1140726,1140727,1140728,1140814,1140948,821419,945811 CVE References: CVE-2018-16871,CVE-2018-20836,CVE-2019-10126,CVE-2019-10638,CVE-2019-10639,CVE-2019-11478,CVE-2019-11599,CVE-2019-12456,CVE-2019-12614,CVE-2019-12818,CVE-2019-12819 Sources used: SUSE Linux Enterprise Live Patching 12-SP4 (src): kgraft-patch-SLE12-SP4_Update_6-1-6.5.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-SU-2019:1855-1: An update that solves 12 vulnerabilities and has 73 fixes is now available. Category: security (important) Bug References: 1051510,1061840,1065600,1071995,1088047,1094555,1098633,1106383,1106751,1109137,1114279,1119532,1120423,1124167,1127155,1128432,1128902,1128910,1131645,1132154,1132390,1133401,1133738,1134303,1134395,1135296,1135556,1135642,1136157,1136598,1136922,1136935,1137103,1137194,1137429,1137625,1137728,1137884,1137995,1137996,1137998,1137999,1138000,1138002,1138003,1138005,1138006,1138007,1138008,1138009,1138010,1138011,1138012,1138013,1138014,1138015,1138016,1138017,1138018,1138019,1138291,1138293,1138374,1138375,1138589,1138719,1139751,1139771,1139782,1139865,1140133,1140328,1140405,1140424,1140428,1140575,1140577,1140637,1140658,1140715,1140719,1140726,1140727,1140728,1140814 CVE References: CVE-2018-16871,CVE-2018-20836,CVE-2019-10126,CVE-2019-10638,CVE-2019-10639,CVE-2019-11478,CVE-2019-11599,CVE-2019-12380,CVE-2019-12456,CVE-2019-12614,CVE-2019-12818,CVE-2019-12819 Sources used: SUSE Linux Enterprise Workstation Extension 15 (src): kernel-default-4.12.14-150.27.1 SUSE Linux Enterprise Module for Open Buildservice Development Tools 15 (src): kernel-default-4.12.14-150.27.1, kernel-docs-4.12.14-150.27.1, kernel-obs-qa-4.12.14-150.27.1 SUSE Linux Enterprise Module for Live Patching 15 (src): kernel-default-4.12.14-150.27.1, kernel-livepatch-SLE15_Update_12-1-1.5.1 SUSE Linux Enterprise Module for Legacy Software 15 (src): kernel-default-4.12.14-150.27.1 SUSE Linux Enterprise Module for Development Tools 15 (src): kernel-docs-4.12.14-150.27.1, kernel-obs-build-4.12.14-150.27.1, kernel-source-4.12.14-150.27.1, kernel-syms-4.12.14-150.27.1, kernel-vanilla-4.12.14-150.27.1 SUSE Linux Enterprise Module for Basesystem 15 (src): kernel-default-4.12.14-150.27.1, kernel-source-4.12.14-150.27.1, kernel-zfcpdump-4.12.14-150.27.1 SUSE Linux Enterprise High Availability 15 (src): kernel-default-4.12.14-150.27.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-SU-2019:1851-1: An update that solves 11 vulnerabilities and has 77 fixes is now available. Category: security (important) Bug References: 1051510,1061840,1065600,1071995,1088047,1094555,1098633,1106383,1106751,1109137,1114279,1119532,1120423,1124167,1127155,1128432,1128902,1128910,1132154,1132390,1133401,1133738,1134303,1134395,1135296,1135556,1135642,1136157,1136811,1136922,1137103,1137194,1137221,1137366,1137429,1137625,1137728,1137884,1137995,1137996,1137998,1137999,1138000,1138002,1138003,1138005,1138006,1138007,1138008,1138009,1138010,1138011,1138012,1138013,1138014,1138015,1138016,1138017,1138018,1138019,1138291,1138293,1138374,1138375,1138589,1138719,1139751,1139771,1139782,1139865,1140133,1140328,1140405,1140424,1140428,1140575,1140577,1140637,1140658,1140715,1140719,1140726,1140727,1140728,1140814,1140948,821419,945811 CVE References: CVE-2018-16871,CVE-2018-20836,CVE-2019-10126,CVE-2019-10638,CVE-2019-10639,CVE-2019-11478,CVE-2019-11599,CVE-2019-12456,CVE-2019-12614,CVE-2019-12818,CVE-2019-12819 Sources used: SUSE Linux Enterprise Workstation Extension 12-SP4 (src): kernel-default-4.12.14-95.24.1 SUSE Linux Enterprise Software Development Kit 12-SP4 (src): kernel-docs-4.12.14-95.24.1, kernel-obs-build-4.12.14-95.24.1 SUSE Linux Enterprise Server 12-SP4 (src): kernel-default-4.12.14-95.24.1, kernel-source-4.12.14-95.24.1, kernel-syms-4.12.14-95.24.1 SUSE Linux Enterprise Live Patching 12-SP4 (src): kgraft-patch-SLE12-SP4_Update_6-1-6.5.1 SUSE Linux Enterprise High Availability 12-SP4 (src): kernel-default-4.12.14-95.24.1 SUSE Linux Enterprise Desktop 12-SP4 (src): kernel-default-4.12.14-95.24.1, kernel-source-4.12.14-95.24.1, kernel-syms-4.12.14-95.24.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Closing.
openSUSE-SU-2019:1716-1: An update that solves 7 vulnerabilities and has 45 fixes is now available. Category: security (important) Bug References: 1051510,1071995,1088047,1094555,1098633,1106383,1106751,1109137,1114279,1119532,1120423,1124167,1127155,1128902,1128910,1131645,1132154,1132390,1133401,1133738,1134303,1134395,1135296,1135556,1135642,1136157,1136935,1137103,1137194,1137625,1137728,1137884,1138589,1138719,1139771,1139782,1139865,1140133,1140328,1140405,1140424,1140428,1140575,1140577,1140637,1140658,1140715,1140719,1140726,1140727,1140728,1140814 CVE References: CVE-2018-16871,CVE-2018-20836,CVE-2019-10126,CVE-2019-10638,CVE-2019-10639,CVE-2019-11599,CVE-2019-12614 Sources used: openSUSE Leap 15.0 (src): kernel-debug-4.12.14-lp150.12.67.1, kernel-default-4.12.14-lp150.12.67.1, kernel-docs-4.12.14-lp150.12.67.1, kernel-kvmsmall-4.12.14-lp150.12.67.1, kernel-obs-build-4.12.14-lp150.12.67.1, kernel-obs-qa-4.12.14-lp150.12.67.1, kernel-source-4.12.14-lp150.12.67.1, kernel-syms-4.12.14-lp150.12.67.1, kernel-vanilla-4.12.14-lp150.12.67.1
openSUSE-SU-2019:1757-1: An update that solves 9 vulnerabilities and has 82 fixes is now available. Category: security (important) Bug References: 1051510,1071995,1088047,1094555,1098633,1103990,1103991,1103992,1106383,1109837,1111666,1112374,1114279,1114685,1119113,1119532,1120423,1125703,1128902,1130836,1132390,1133401,1133738,1134303,1134395,1135296,1135556,1135642,1135897,1136156,1136157,1136161,1136264,1136271,1136333,1136343,1136462,1136935,1137103,1137194,1137366,1137625,1137728,1137884,1137985,1138263,1138589,1138681,1138719,1138732,1138879,1139712,1139771,1139865,1140133,1140228,1140328,1140405,1140424,1140428,1140454,1140463,1140575,1140577,1140637,1140658,1140715,1140719,1140726,1140727,1140728,1140814,1140887,1140888,1140889,1140891,1140893,1140948,1140954,1140955,1140956,1140957,1140958,1140959,1140960,1140961,1140962,1140964,1140971,1140972,1140992 CVE References: CVE-2018-16871,CVE-2018-20836,CVE-2019-10126,CVE-2019-10638,CVE-2019-10639,CVE-2019-11599,CVE-2019-12614,CVE-2019-12817,CVE-2019-13233 Sources used: openSUSE Leap 15.1 (src): kernel-debug-4.12.14-lp151.28.10.1, kernel-default-4.12.14-lp151.28.10.1, kernel-docs-4.12.14-lp151.28.10.1, kernel-kvmsmall-4.12.14-lp151.28.10.1, kernel-obs-build-4.12.14-lp151.28.10.1, kernel-obs-qa-4.12.14-lp151.28.10.1, kernel-source-4.12.14-lp151.28.10.1, kernel-syms-4.12.14-lp151.28.10.1, kernel-vanilla-4.12.14-lp151.28.10.1
SUSE-SU-2019:2069-1: An update that solves 18 vulnerabilities and has 157 fixes is now available. Category: security (important) Bug References: 1051510,1055117,1071995,1083647,1083710,1088047,1094555,1098633,1103990,1103991,1103992,1104745,1106383,1109837,1111666,1112374,1114279,1114685,1119113,1119222,1119532,1120423,1123080,1125703,1127034,1127315,1127611,1128432,1128902,1129770,1130836,1132390,1133021,1133401,1133738,1134090,1134097,1134390,1134395,1134399,1134730,1134738,1135153,1135296,1135335,1135556,1135642,1135897,1136156,1136157,1136161,1136217,1136264,1136271,1136333,1136342,1136343,1136345,1136348,1136460,1136461,1136462,1136467,1137103,1137194,1137224,1137366,1137429,1137458,1137534,1137535,1137584,1137586,1137609,1137625,1137728,1137811,1137827,1137884,1137985,1138263,1138291,1138293,1138336,1138374,1138375,1138589,1138681,1138719,1138732,1138874,1138879,1139358,1139619,1139712,1139751,1139771,1139865,1140133,1140139,1140228,1140322,1140328,1140405,1140424,1140428,1140454,1140463,1140559,1140575,1140577,1140637,1140652,1140658,1140676,1140715,1140719,1140726,1140727,1140728,1140814,1140887,1140888,1140889,1140891,1140893,1140903,1140945,1140948,1140954,1140955,1140956,1140957,1140958,1140959,1140960,1140961,1140962,1140964,1140971,1140972,1140992,1141312,1141401,1141402,1141452,1141453,1141454,1141478,1141558,1142023,1142052,1142083,1142112,1142115,1142119,1142220,1142221,1142265,1142350,1142351,1142354,1142359,1142450,1142623,1142673,1142701,1142868,1143003,1143105,1143185,1143189,1143191,1143209,1143507 CVE References: CVE-2018-16871,CVE-2018-20836,CVE-2018-20855,CVE-2019-10638,CVE-2019-10639,CVE-2019-1125,CVE-2019-11478,CVE-2019-11599,CVE-2019-11810,CVE-2019-12614,CVE-2019-12817,CVE-2019-12818,CVE-2019-12819,CVE-2019-13233,CVE-2019-13631,CVE-2019-13648,CVE-2019-14283,CVE-2019-14284 Sources used: SUSE Linux Enterprise Module for Public Cloud 15-SP1 (src): kernel-azure-4.12.14-8.13.1, kernel-source-azure-4.12.14-8.13.1, kernel-syms-azure-4.12.14-8.13.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-SU-2019:2430-1: An update that solves 45 vulnerabilities and has 474 fixes is now available. Category: security (important) Bug References: 1050242,1050549,1051510,1052904,1053043,1055117,1055121,1055186,1056787,1058115,1061840,1064802,1065600,1065729,1066129,1070872,1071995,1075020,1082387,1082555,1083647,1083710,1085535,1085536,1088047,1088804,1093389,1094555,1096003,1098633,1099658,1102247,1103186,1103259,1103990,1103991,1103992,1104745,1106011,1106284,1106383,1106751,1108193,1108838,1108937,1109837,1110946,1111331,1111666,1111696,1112063,1112128,1112178,1112374,1113722,1113956,1114279,1114427,1114542,1114638,1114685,1115688,1117114,1117158,1117561,1118139,1119113,1119222,1119532,1119680,1120091,1120318,1120423,1120566,1120843,1120902,1122767,1122776,1123080,1123454,1123663,1124503,1124839,1125703,1126206,1126356,1126704,1127034,1127175,1127315,1127371,1127374,1127611,1127616,1128052,1128415,1128432,1128544,1128902,1128904,1128971,1128979,1129138,1129273,1129693,1129770,1129845,1130195,1130425,1130527,1130567,1130579,1130699,1130836,1130937,1130972,1131326,1131427,1131438,1131451,1131467,1131488,1131530,1131565,1131574,1131587,1131645,1131659,1131673,1131847,1131848,1131851,1131900,1131934,1131935,1132044,1132219,1132226,1132227,1132365,1132368,1132369,1132370,1132372,1132373,1132384,1132390,1132397,1132402,1132403,1132404,1132405,1132407,1132411,1132412,1132413,1132414,1132426,1132527,1132531,1132555,1132558,1132561,1132562,1132563,1132564,1132570,1132571,1132572,1132589,1132618,1132673,1132681,1132726,1132828,1132894,1132943,1132982,1133005,1133016,1133021,1133094,1133095,1133115,1133149,1133176,1133188,1133190,1133311,1133320,1133401,1133486,1133529,1133547,1133584,1133593,1133612,1133616,1133667,1133668,1133672,1133674,1133675,1133698,1133702,1133731,1133738,1133769,1133772,1133774,1133778,1133779,1133780,1133825,1133850,1133851,1133852,1133897,1134090,1134097,1134160,1134162,1134199,1134200,1134201,1134202,1134203,1134204,1134205,1134223,1134303,1134354,1134390,1134393,1134395,1134397,1134399,1134459,1134460,1134461,1134597,1134600,1134607,1134618,1134651,1134671,1134730,1134738,1134743,1134760,1134806,1134810,1134813,1134848,1134936,1134945,1134946,1134947,1134948,1134949,1134950,1134951,1134952,1134953,1134972,1134974,1134975,1134980,1134981,1134983,1134987,1134989,1134990,1134994,1134995,1134998,1134999,1135006,1135007,1135008,1135018,1135021,1135024,1135026,1135027,1135028,1135029,1135031,1135033,1135034,1135035,1135036,1135037,1135038,1135039,1135041,1135042,1135044,1135045,1135046,1135047,1135049,1135051,1135052,1135053,1135055,1135056,1135058,1135100,1135120,1135153,1135278,1135281,1135296,1135309,1135312,1135314,1135315,1135316,1135320,1135323,1135330,1135335,1135492,1135542,1135556,1135603,1135642,1135661,1135758,1135897,1136156,1136157,1136161,1136188,1136206,1136215,1136217,1136264,1136271,1136333,1136342,1136343,1136345,1136347,1136348,1136353,1136424,1136428,1136430,1136432,1136434,1136435,1136438,1136439,1136456,1136460,1136461,1136462,1136467,1136469,1136477,1136478,1136498,1136573,1136586,1136598,1136881,1136922,1136935,1136978,1136990,1137103,1137151,1137152,1137153,1137162,1137194,1137201,1137224,1137232,1137233,1137236,1137366,1137372,1137429,1137444,1137458,1137534,1137535,1137584,1137586,1137609,1137625,1137728,1137739,1137752,1137811,1137827,1137884,1137985,1137995,1137996,1137998,1137999,1138000,1138002,1138003,1138005,1138006,1138007,1138008,1138009,1138010,1138011,1138012,1138013,1138014,1138015,1138016,1138017,1138018,1138019,1138263,1138291,1138293,1138336,1138374,1138375,1138589,1138681,1138719,1138732,1138874,1138879,1139358,1139619,1139712,1139751,1139771,1139865,1140133,1140139,1140228,1140322,1140328,1140405,1140424,1140428,1140454,1140463,1140559,1140575,1140577,1140637,1140652,1140658,1140676,1140715,1140719,1140726,1140727,1140728,1140814,1140887,1140888,1140889,1140891,1140893,1140903,1140945,1140948,1140954,1140955,1140956,1140957,1140958,1140959,1140960,1140961,1140962,1140964,1140971,1140972,1140992,1141312,1141401,1141402,1141452,1141453,1141454,1141478,1141558,1142023,1142052,1142083,1142112,1142115,1142119,1142220,1142221,1142254,1142350,1142351,1142354,1142359,1142450,1142623,1142673,1142701,1142868,1143003,1143045,1143105,1143185,1143189,1143191,1143209,1143507 CVE References: CVE-2017-5753,CVE-2018-12126,CVE-2018-12127,CVE-2018-12130,CVE-2018-16871,CVE-2018-16880,CVE-2018-20836,CVE-2018-20855,CVE-2018-7191,CVE-2019-10124,CVE-2019-10638,CVE-2019-10639,CVE-2019-11085,CVE-2019-11091,CVE-2019-1125,CVE-2019-11477,CVE-2019-11478,CVE-2019-11479,CVE-2019-11486,CVE-2019-11487,CVE-2019-11599,CVE-2019-11810,CVE-2019-11811,CVE-2019-11815,CVE-2019-11833,CVE-2019-11884,CVE-2019-12380,CVE-2019-12382,CVE-2019-12456,CVE-2019-12614,CVE-2019-12817,CVE-2019-12818,CVE-2019-12819,CVE-2019-13233,CVE-2019-13631,CVE-2019-13648,CVE-2019-14283,CVE-2019-14284,CVE-2019-3846,CVE-2019-3882,CVE-2019-5489,CVE-2019-8564,CVE-2019-9003,CVE-2019-9500,CVE-2019-9503 Sources used: SUSE Linux Enterprise Module for Realtime 15-SP1 (src): kernel-rt-4.12.14-14.8.1, kernel-rt_debug-4.12.14-14.8.1, kernel-source-rt-4.12.14-14.8.1, kernel-syms-rt-4.12.14-14.8.1 SUSE Linux Enterprise Module for Open Buildservice Development Tools 15-SP1 (src): kernel-rt-4.12.14-14.8.1, kernel-rt_debug-4.12.14-14.8.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-SU-2019:2756-1: An update that solves 44 vulnerabilities and has 368 fixes is now available. Category: security (important) Bug References: 1012382,1047238,1050911,1051510,1053043,1054914,1055117,1056686,1060662,1061840,1061843,1064597,1064701,1065600,1065729,1066369,1071009,1071306,1071995,1078248,1082555,1083647,1083710,1085030,1085536,1085539,1086103,1087092,1088047,1090734,1091171,1093205,1094555,1098633,1102097,1102247,1104902,1104967,1106061,1106284,1106383,1106434,1106751,1108382,1109137,1109158,1111666,1112178,1112894,1112899,1112902,1112903,1112905,1112906,1112907,1113722,1114279,1114542,1115688,1117158,1118139,1118689,1119086,1119222,1119532,1120423,1120566,1120876,1120902,1120937,1123034,1123080,1123105,1123959,1124167,1124370,1124503,1127034,1127155,1127315,1127988,1128432,1128902,1128910,1129424,1129519,1129664,1129770,1130972,1131107,1131281,1131304,1131565,1132154,1132390,1132686,1133021,1133401,1134097,1134291,1134303,1134390,1134671,1134881,1134882,1135219,1135296,1135335,1135556,1135642,1135661,1135897,1136157,1136261,1136811,1136896,1136935,1136990,1137069,1137162,1137221,1137366,1137372,1137429,1137444,1137458,1137534,1137535,1137584,1137586,1137609,1137625,1137728,1137739,1137752,1137811,1137827,1137865,1137884,1137959,1137995,1137996,1137998,1137999,1138000,1138002,1138003,1138005,1138006,1138007,1138008,1138009,1138010,1138011,1138012,1138013,1138014,1138015,1138016,1138017,1138018,1138019,1138374,1138375,1138539,1138589,1138719,1139020,1139021,1139101,1139500,1139771,1139782,1139865,1140012,1140133,1140139,1140155,1140322,1140328,1140405,1140424,1140426,1140428,1140487,1140637,1140652,1140658,1140715,1140719,1140726,1140727,1140728,1140814,1140887,1140888,1140889,1140891,1140893,1140903,1140945,1140948,1140954,1140955,1140956,1140957,1140958,1140959,1140960,1140961,1140962,1140964,1140971,1140972,1140992,1141013,1141401,1141402,1141450,1141452,1141453,1141454,1141478,1141543,1141554,1142019,1142076,1142109,1142112,1142117,1142118,1142119,1142129,1142220,1142221,1142350,1142351,1142354,1142359,1142450,1142496,1142541,1142635,1142685,1142701,1142857,1142868,1143003,1143105,1143185,1143300,1143466,1143507,1143765,1143841,1143843,1144123,1144333,1144474,1144518,1144718,1144813,1144880,1144886,1144912,1144920,1144979,1145010,1145024,1145051,1145059,1145189,1145235,1145300,1145302,1145388,1145389,1145390,1145391,1145392,1145393,1145394,1145395,1145396,1145397,1145408,1145409,1145661,1145678,1145687,1145920,1145922,1145934,1145937,1145940,1145941,1145942,1146042,1146074,1146084,1146163,1146285,1146346,1146351,1146352,1146361,1146376,1146378,1146381,1146391,1146399,1146413,1146425,1146512,1146514,1146516,1146519,1146524,1146526,1146529,1146531,1146540,1146543,1146547,1146550,1146575,1146589,1146664,1146678,1146938,1148031,1148032,1148033,1148034,1148035,1148093,1148133,1148192,1148196,1148198,1148202,1148303,1148363,1148379,1148394,1148527,1148574,1148616,1148617,1148619,1148698,1148712,1148859,1148868,1149053,1149083,1149104,1149105,1149106,1149197,1149214,1149224,1149313,1149325,1149376,1149413,1149418,1149424,1149446,1149522,1149527,1149539,1149552,1149555,1149591,1149602,1149612,1149626,1149651,1149652,1149713,1149940,1149959,1149963,1149976,1150025,1150033,1150112,1150381,1150423,1150562,1150727,1150860,1150861,1150933,1151350,1151610,1151667,1151671,1151891,1151955,1152024,1152025,1152026,1152161,1152325,1152457,1152460,1152466,1152972,1152974,1152975 CVE References: CVE-2017-18551,CVE-2017-18595,CVE-2018-20976,CVE-2018-21008,CVE-2019-10207,CVE-2019-11479,CVE-2019-14814,CVE-2019-14815,CVE-2019-14816,CVE-2019-14821,CVE-2019-14835,CVE-2019-15030,CVE-2019-15031,CVE-2019-15090,CVE-2019-15098,CVE-2019-15117,CVE-2019-15118,CVE-2019-15211,CVE-2019-15212,CVE-2019-15214,CVE-2019-15215,CVE-2019-15216,CVE-2019-15217,CVE-2019-15218,CVE-2019-15219,CVE-2019-15220,CVE-2019-15221,CVE-2019-15222,CVE-2019-15239,CVE-2019-15290,CVE-2019-15291,CVE-2019-15292,CVE-2019-15538,CVE-2019-15666,CVE-2019-15902,CVE-2019-15917,CVE-2019-15919,CVE-2019-15920,CVE-2019-15921,CVE-2019-15924,CVE-2019-15926,CVE-2019-15927,CVE-2019-9456,CVE-2019-9506 Sources used: SUSE Linux Enterprise Real Time Extension 12-SP4 (src): kernel-rt-4.12.14-8.6.1, kernel-rt_debug-4.12.14-8.6.1, kernel-source-rt-4.12.14-8.6.1, kernel-syms-rt-4.12.14-8.6.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.