Bug 1077388 - System locks up after running out of memory without using swap
System locks up after running out of memory without using swap
Status: RESOLVED WONTFIX
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Basesystem
Current
x86-64 Other
: P5 - None : Normal (vote)
: ---
Assigned To: E-mail List
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-01-24 10:42 UTC by Dennis Irrgang
Modified: 2019-11-29 14:59 UTC (History)
5 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dennis Irrgang 2018-01-24 10:42:20 UTC
I've run into this issue at least a dozen times now and already tried to get help in #suse on Freenode.

Whenever my system runs out of memory it locks up. The HDD activity LED is fully lit suggesting the system attempting to move data onto the swap file/partition. However this is unsuccessful, as the system continues to be unresponsive until a forced reboot.

The system also only barely seems to be using the swap to begin with. Almost the entire memory can be used already, but it won't start using swap until it's apparently too late.

I configured journalctl to be persistent, but there are no useful outputs that give an idea what might be causing the issue. The only 'useful' output I had once was the audio daemon reporting dropped packets in the stream to my bluetooth speaker.

The system is largely 'stock', a relatively fresh install from November on a fairly standard Lenovo T430.

Info:
Linux lenny.dir.li 4.14.14-1-default #1 SMP PREEMPT Wed Jan 17 09:26:10 UTC 2018 (eef6178) x86_64 x86_64 x86_64 GNU/Linux

lenny.dir.li                
    description: Notebook
    product: 2347A31 (LENOVO_MT_2347)
    vendor: LENOVO
    version: ThinkPad T430
    serial: PBGHHHD
    width: 4294967295 bits
    capabilities: smbios-2.7 dmi-2.7 smp vsyscall32
    configuration: administrator_password=disabled chassis=notebook family=ThinkPad T430 power-on_password=disabled sku=LENOVO_MT_2347 uuid=0125BC5A-1D52-CB11-BC35-AFEA165C4C79
  *-core
       description: Motherboard
       product: 2347A31
       vendor: LENOVO
       physical id: 0
       version: Not Defined
       serial: 1ZLMB27H3CX
       slot: Not Available
     *-cpu
          description: CPU
          product: Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz
          vendor: Intel Corp.
          physical id: 1
          bus info: cpu@0
          version: Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz
          serial: None
          slot: CPU Socket - U3E1
          size: 2991MHz
          capacity: 3300MHz
          width: 64 bits
          clock: 100MHz
          capabilities: x86-64 fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti retpoline tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts cpufreq
          configuration: cores=2 enabledcores=2 threads=4
        *-cache:0
             description: L1 cache
             physical id: 3
             slot: L1-Cache
             size: 32KiB
             capacity: 32KiB
             capabilities: internal write-through instruction
             configuration: level=1
        *-cache:1
             description: L2 cache
             physical id: 4
             slot: L2-Cache
             size: 256KiB
             capacity: 256KiB
             capabilities: internal write-through unified
             configuration: level=2
        *-cache:2
             description: L3 cache
             physical id: 5
             slot: L3-Cache
             size: 3MiB
             capacity: 3MiB
             capabilities: internal write-back unified
             configuration: level=3
     *-cache
          description: L1 cache
          physical id: 2
          slot: L1-Cache
          size: 32KiB
          capacity: 32KiB
          capabilities: internal write-through data
          configuration: level=1
     *-memory
          description: System Memory
          physical id: 7
          slot: System board or motherboard
          size: 8GiB
        *-bank:0
             description: SODIMM DDR3 Synchronous 1600 MHz (0,6 ns)
             product: HMT351S6CFR8C-PB
             vendor: Hynix/Hyundai
             physical id: 0
             serial: 3F135033
             slot: ChannelA-DIMM0
             size: 4GiB
             width: 64 bits
             clock: 1600MHz (0.6ns)
        *-bank:1
             description: SODIMM DDR3 Synchronous 1600 MHz (0,6 ns)
             product: HMT351S6CFR8C-PB
             vendor: Hynix/Hyundai
             physical id: 1
             serial: 04246841
             slot: ChannelB-DIMM0
             size: 4GiB
             width: 64 bits
             clock: 1600MHz (0.6ns)
     *-firmware
          description: BIOS
          vendor: LENOVO
          physical id: c
          version: G1ETB4WW (2.74 )
          date: 09/25/2017
          size: 128KiB
          capacity: 11MiB
          capabilities: pci pnp upgrade shadowing cdboot bootselect edd int13floppy720 int5printscreen int9keyboard int14serial int17printer int10video acpi usb biosbootspecification uefi
     *-pci
          description: Host bridge
          product: 3rd Gen Core processor DRAM Controller
          vendor: Intel Corporation
          physical id: 100
          bus info: pci@0000:00:00.0
          version: 09
          width: 32 bits
          clock: 33MHz
          configuration: driver=ivb_uncore
          resources: irq:0
        *-display
             description: VGA compatible controller
             product: 3rd Gen Core processor Graphics Controller
             vendor: Intel Corporation
             physical id: 2
             bus info: pci@0000:00:02.0
             version: 09
             width: 64 bits
             clock: 33MHz
             capabilities: msi pm vga_controller bus_master cap_list rom
             configuration: driver=i915 latency=0
             resources: irq:28 memory:f0000000-f03fffff memory:e0000000-efffffff ioport:6000(size=64) memory:c0000-dffff
        *-usb:0
             description: USB controller
             product: 7 Series/C210 Series Chipset Family USB xHCI Host Controller
             vendor: Intel Corporation
             physical id: 14
             bus info: pci@0000:00:14.0
             version: 04
             width: 64 bits
             clock: 33MHz
             capabilities: pm msi xhci bus_master cap_list
             configuration: driver=xhci_hcd latency=0
             resources: irq:27 memory:f2520000-f252ffff
           *-usbhost:0
                product: xHCI Host Controller
                vendor: Linux 4.14.14-1-default xhci-hcd
                physical id: 0
                bus info: usb@3
                logical name: usb3
                version: 4.14
                capabilities: usb-2.00
                configuration: driver=hub slots=4 speed=480Mbit/s
              *-usb
                   description: Mouse
                   product: G9 Laser Mouse
                   vendor: Logitech
                   physical id: 1
                   bus info: usb@3:1
                   version: 50.00
                   serial: 1167F71BED0029
                   capabilities: usb-2.00
                   configuration: driver=usbhid maxpower=98mA speed=12Mbit/s
           *-usbhost:1
                product: xHCI Host Controller
                vendor: Linux 4.14.14-1-default xhci-hcd
                physical id: 1
                bus info: usb@4
                logical name: usb4
                version: 4.14
                capabilities: usb-3.00
                configuration: driver=hub slots=4 speed=5000Mbit/s
        *-communication
             description: Communication controller
             product: 7 Series/C216 Chipset Family MEI Controller #1
             vendor: Intel Corporation
             physical id: 16
             bus info: pci@0000:00:16.0
             version: 04
             width: 64 bits
             clock: 33MHz
             capabilities: pm msi bus_master cap_list
             configuration: driver=mei_me latency=0
             resources: irq:29 memory:f2535000-f253500f
        *-network
             description: Ethernet interface
             product: 82579LM Gigabit Network Connection (Lewisville)
             vendor: Intel Corporation
             physical id: 19
             bus info: pci@0000:00:19.0
             logical name: enp0s25
             version: 04
             serial: 00:21:cc:ca:06:3b
             capacity: 1Gbit/s
             width: 32 bits
             clock: 33MHz
             capabilities: pm msi bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
             configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=3.2.6-k firmware=0.13-3 latency=0 link=no multicast=yes port=twisted pair
             resources: irq:31 memory:f2500000-f251ffff memory:f253b000-f253bfff ioport:6080(size=32)
        *-usb:1
             description: USB controller
             product: 7 Series/C216 Chipset Family USB Enhanced Host Controller #2
             vendor: Intel Corporation
             physical id: 1a
             bus info: pci@0000:00:1a.0
             version: 04
             width: 32 bits
             clock: 33MHz
             capabilities: pm debug ehci bus_master cap_list
             configuration: driver=ehci-pci latency=0
             resources: irq:16 memory:f253a000-f253a3ff
           *-usbhost
                product: EHCI Host Controller
                vendor: Linux 4.14.14-1-default ehci_hcd
                physical id: 1
                bus info: usb@1
                logical name: usb1
                version: 4.14
                capabilities: usb-2.00
                configuration: driver=hub slots=3 speed=480Mbit/s
              *-usb
                   description: USB hub
                   product: Integrated Rate Matching Hub
                   vendor: Intel Corp.
                   physical id: 1
                   bus info: usb@1:1
                   version: 0.00
                   capabilities: usb-2.00
                   configuration: driver=hub slots=6 speed=480Mbit/s
                 *-usb:0
                      description: Generic USB device
                      product: BCM20702A0
                      vendor: Broadcom Corp
                      physical id: 4
                      bus info: usb@1:1.4
                      version: 1.12
                      serial: E006E6B794F2
                      capabilities: usb-2.00
                      configuration: driver=btusb speed=12Mbit/s
                 *-usb:1
                      description: Video
                      product: Integrated Camera
                      vendor: Chicony Electronics Co., Ltd.
                      physical id: 6
                      bus info: usb@1:1.6
                      version: 5.18
                      capabilities: usb-2.00
                      configuration: driver=uvcvideo maxpower=200mA speed=480Mbit/s
        *-multimedia
             description: Audio device
             product: 7 Series/C216 Chipset Family High Definition Audio Controller
             vendor: Intel Corporation
             physical id: 1b
             bus info: pci@0000:00:1b.0
             version: 04
             width: 64 bits
             clock: 33MHz
             capabilities: pm msi pciexpress bus_master cap_list
             configuration: driver=snd_hda_intel latency=0
             resources: irq:30 memory:f2530000-f2533fff
        *-pci:0
             description: PCI bridge
             product: 7 Series/C216 Chipset Family PCI Express Root Port 1
             vendor: Intel Corporation
             physical id: 1c
             bus info: pci@0000:00:1c.0
             version: c4
             width: 32 bits
             clock: 33MHz
             capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:16 ioport:5000(size=4096) memory:f1d00000-f24fffff ioport:f0400000(size=8388608)
           *-generic
                description: System peripheral
                product: MMC/SD Host Controller
                vendor: Ricoh Co Ltd
                physical id: 0
                bus info: pci@0000:02:00.0
                version: 07
                width: 32 bits
                clock: 33MHz
                capabilities: msi pm pciexpress bus_master cap_list
                configuration: driver=sdhci-pci latency=0
                resources: irq:16 memory:f1d00000-f1d000ff
        *-pci:1
             description: PCI bridge
             product: 7 Series/C210 Series Chipset Family PCI Express Root Port 2
             vendor: Intel Corporation
             physical id: 1c.1
             bus info: pci@0000:00:1c.1
             version: c4
             width: 32 bits
             clock: 33MHz
             capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:17 memory:f1c00000-f1cfffff
           *-network
                description: Wireless interface
                product: Centrino Advanced-N 6205 [Taylor Peak]
                vendor: Intel Corporation
                physical id: 0
                bus info: pci@0000:03:00.0
                logical name: wlp3s0
                version: 34
                serial: 8c:70:5a:9d:35:08
                width: 64 bits
                clock: 33MHz
                capabilities: pm msi pciexpress bus_master cap_list ethernet physical wireless
                configuration: broadcast=yes driver=iwlwifi driverversion=4.14.14-1-default firmware=18.168.6.1 ip=192.168.178.53 latency=0 link=yes multicast=yes wireless=IEEE 802.11
                resources: irq:32 memory:f1c00000-f1c01fff
        *-pci:2
             description: PCI bridge
             product: 7 Series/C210 Series Chipset Family PCI Express Root Port 3
             vendor: Intel Corporation
             physical id: 1c.2
             bus info: pci@0000:00:1c.2
             version: c4
             width: 32 bits
             clock: 33MHz
             capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:18 ioport:4000(size=4096) memory:f1400000-f1bfffff ioport:f0c00000(size=8388608)
        *-usb:2
             description: USB controller
             product: 7 Series/C216 Chipset Family USB Enhanced Host Controller #1
             vendor: Intel Corporation
             physical id: 1d
             bus info: pci@0000:00:1d.0
             version: 04
             width: 32 bits
             clock: 33MHz
             capabilities: pm debug ehci bus_master cap_list
             configuration: driver=ehci-pci latency=0
             resources: irq:23 memory:f2539000-f25393ff
           *-usbhost
                product: EHCI Host Controller
                vendor: Linux 4.14.14-1-default ehci_hcd
                physical id: 1
                bus info: usb@2
                logical name: usb2
                version: 4.14
                capabilities: usb-2.00
                configuration: driver=hub slots=3 speed=480Mbit/s
              *-usb
                   description: USB hub
                   product: Integrated Rate Matching Hub
                   vendor: Intel Corp.
                   physical id: 1
                   bus info: usb@2:1
                   version: 0.00
                   capabilities: usb-2.00
                   configuration: driver=hub slots=8 speed=480Mbit/s
        *-isa
             description: ISA bridge
             product: QM77 Express Chipset LPC Controller
             vendor: Intel Corporation
             physical id: 1f
             bus info: pci@0000:00:1f.0
             version: 04
             width: 32 bits
             clock: 33MHz
             capabilities: isa bus_master cap_list
             configuration: driver=lpc_ich latency=0
             resources: irq:0
        *-storage
             description: SATA controller
             product: 7 Series Chipset Family 6-port SATA Controller [AHCI mode]
             vendor: Intel Corporation
             physical id: 1f.2
             bus info: pci@0000:00:1f.2
             version: 04
             width: 32 bits
             clock: 66MHz
             capabilities: storage msi pm ahci_1.0 bus_master cap_list
             configuration: driver=ahci latency=0
             resources: irq:26 ioport:60a8(size=8) ioport:60b4(size=4) ioport:60a0(size=8) ioport:60b0(size=4) ioport:6060(size=32) memory:f2538000-f25387ff
        *-serial
             description: SMBus
             product: 7 Series/C216 Chipset Family SMBus Controller
             vendor: Intel Corporation
             physical id: 1f.3
             bus info: pci@0000:00:1f.3
             version: 04
             width: 64 bits
             clock: 33MHz
             configuration: driver=i801_smbus latency=0
             resources: irq:18 memory:f2534000-f25340ff ioport:efa0(size=32)
     *-scsi:0
          physical id: 0
          logical name: scsi0
          capabilities: emulated
        *-disk
             description: ATA Disk
             product: INTEL SSDSC2BW24
             physical id: 0.0.0
             bus info: scsi@0:0.0.0
             logical name: /dev/sda
             version: LE1i
             serial: CVCV22610069240CGN
             size: 223GiB (240GB)
             capabilities: gpt-1.00 partitioned partitioned:gpt
             configuration: ansiversion=5 guid=09af3522-df01-4256-bbd5-d10f372f0a20 logicalsectorsize=512 sectorsize=512
           *-volume:0
                description: Windows FAT volume
                vendor: mkfs.fat
                physical id: 1
                bus info: scsi@0:0.0.0,1
                logical name: /dev/sda1
                logical name: /boot/efi
                version: FAT16
                serial: 84d2-3dff
                size: 154MiB
                capacity: 155MiB
                capabilities: boot fat initialized
                configuration: FATs=2 filesystem=fat mount.fstype=vfat mount.options=rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro name=primary state=mounted
           *-volume:1
                description: EFI partition
                physical id: 2
                bus info: scsi@0:0.0.0,2
                logical name: /dev/sda2
                serial: 52b71cc4-6905-4db7-be4a-2f41f0bd6e91
                capacity: 156MiB
                configuration: name=primary
           *-volume:2
                description: LVM Physical Volume
                vendor: Linux
                physical id: 3
                bus info: scsi@0:0.0.0,3
                logical name: /dev/sda3
                serial: 43de8d2d-2541-4e88-8230-bdfd5ae13634
                size: 223GiB
                capacity: 223GiB
                width: 256 bits
                capabilities: multi encrypted luks initialized
                configuration: bits=256 cipher=aes filesystem=luks hash=sha256 mode=xts-plain64 name=primary version=1
     *-scsi:1
          physical id: 3
          logical name: scsi1
          capabilities: emulated
        *-cdrom
             description: DVD-RAM writer
             product: DVDRAM GT50N
             vendor: HL-DT-ST
             physical id: 0.0.0
             bus info: scsi@1:0.0.0
             logical name: /dev/cdrom
             logical name: /dev/cdrw
             logical name: /dev/dvd
             logical name: /dev/dvdrw
             logical name: /dev/sr0
             version: LT20
             capabilities: removable audio cd-r cd-rw dvd dvd-r dvd-ram
             configuration: ansiversion=5 status=nodisc
  *-battery
       product: 45N1173
       vendor: SANYO
       physical id: 1
       slot: Rear
       capacity: 93960mWh
       configuration: voltage=10,8V
Comment 1 Karl Cheng 2018-01-30 13:25:15 UTC
I've experienced this before too with a just over 1 year install of Tumbleweed. Computer is an HP Elitebook 9470m laptop (i5-3317U CPU, single 8GB RAM).

Sometimes I have to resort to using Alt+SysRq+F to manually trigger an OOM, though nowadays I haven't ran into this as often as before. (You'll need to change /proc/sys/kernel/sysrq to enable this.)


Not sure if this is something specific to the system setup on openSUSE or an upstream kernel problem though.
Comment 2 Chenzi Cao 2018-02-01 10:09:47 UTC
Assign to kernel maintainers to take a look, please feel free to reassign whenever necessary, thanks.
Comment 3 Takashi Iwai 2018-02-14 13:46:37 UTC
Put some mm guys into the game, hopefully can give some better insight...
Comment 4 Michal Hocko 2018-02-14 16:31:46 UTC
I am afraid you are seeing a memory reclaim trashing behavior. This basically means the the remaining reclaimable memory (page cache as the swap is not available) is most likely the executed code which gets reclaimed and refaulted back. Our out-of-memory detection relies on the fact that there is no reclaimable memory but you might be unlucky to never hit that condition because the clean page cache is too easy to drop and re-read from the storage to make an impression of the reclaim forward progress.

The sad news is that this is a long unsolved problem. And we do not have a good answer for it in the upstream kernel either. Hitting the condition might differ between different kernel versions but the fundamental problem in the OOM detection is still there.
Comment 5 Borislav Petkov 2019-11-29 14:59:12 UTC
Looks forgotten. Feel free to reopen if still of interest.