Bug 913695

Summary: possible memory leak in kernel 3.16.7
Product: [openSUSE] openSUSE Distribution Reporter: Miroslav Beneš <mbenes>
Component: KernelAssignee: Michal Hocko <mhocko>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: jslaby, mbenes, mhocko, mhocko
Version: 13.2   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: meminfo

Description Miroslav Beneš 2015-01-19 14:12:12 UTC
openSUSE 13.2 with the latest kernel 3.16.7-7 got to the state where there was not enough free memory available. Swap was full and no process held the memory at first glance. The system ran for several weeks. There was no page allocation failure or out of memory message in the log.

/proc/meminfo, slabinfo and zoneinfo are attached with the list of processes. 

According to Michal Hocko it could be fixed by 5ddacbe92b806cd5b4f8f154e8e46ac267fff55c in upstream kernel. It is not present in 3.16.7 version.

Unfortunately I had to turn off the computer due to power down in the building and cannot reproduce it again.
Comment 1 Miroslav Beneš 2015-01-19 14:12:35 UTC
Created attachment 620006 [details]
Comment 2 Miroslav Beneš 2015-01-19 14:13:05 UTC
Created attachment 620007 [details]
Comment 3 Miroslav Beneš 2015-01-19 14:13:22 UTC
Created attachment 620008 [details]
Comment 4 Miroslav Beneš 2015-01-19 14:14:04 UTC
Created attachment 620009 [details]
Comment 5 Michal Hocko 2015-01-19 16:15:07 UTC
MemTotal:        8131708 kB
MemFree:         1889300 kB
Buffers:           23424 kB
SwapCached:        31980 kB
Active:          3174752 kB
Inactive:        1886984 kB
Unevictable:          80 kB
Mlocked:              80 kB
SwapTotal:       2103292 kB
SwapFree:              8 kB
Slab:             782384 kB
KernelStack:        8144 kB
PageTables:        41880 kB

echo $((8131708-(1889300+23424+31980+3174752+1886984+80+782384+8144+41880)))

So almost 300M is missing somewhere. This might be a signal of memory leak or some kernel component allocates from the page allocator directly and uses that memory.

5ddacbe92b80 (mm: free compound page with correct order) sounds like a good fit as well because you've told me (off-bugzilla) that the machine was running for a long time and you were running kvm with 4GB of memory many times. I can imagine that THP zero page would be mapped many times in that load and so the leak could build up continually until it starts getting noticable.

I will push the patch to the git even though we are not 100% sure this is the real fix for this issue. Having /proc/vmstat would be ideal because we could check thp_zero_page_alloc which is not present in meminfo nor zoneinfo.
Comment 6 Michal Hocko 2015-01-19 16:25:42 UTC
pushed to openSUSE-13.2 branch with a note that the culprit might be different but the fix is addressing a real leak anyway and as the stable kernel for 3.16 is dead already we need it anyway.
Comment 7 Michal Hocko 2015-01-19 16:32:33 UTC
pushed the patch to openSUSE-13.1 as well. SLE branches are fine (SLE12 has the fix from the stable, SLE11-SP3 and older do not have 97ae17497e99 which introduced the issue).
Comment 8 Miroslav Beneš 2015-02-02 15:21:23 UTC
So I am in the same situation after two weeks. This is the relevant part of /proc/vmstat

thp_fault_alloc 9500
thp_fault_fallback 107474
thp_collapse_alloc 3284
thp_collapse_alloc_failed 17798
thp_split 2167
thp_zero_page_alloc 21
thp_zero_page_alloc_failed 1751

I am going to install kernel with the fix and see what happens.
Comment 9 Swamp Workflow Management 2015-04-13 12:13:55 UTC
openSUSE-SU-2015:0713-1: An update that solves 13 vulnerabilities and has 52 fixes is now available.

Category: security (important)
Bug References: 867199,893428,895797,900811,901925,903589,903640,904899,905681,907039,907818,907988,908582,908588,908589,908592,908593,908594,908596,908598,908603,908604,908605,908606,908608,908610,908612,909077,909078,909477,909634,910150,910322,910440,911311,911325,911326,911356,911438,911578,911835,912061,912202,912429,912705,913059,913466,913695,914175,915425,915454,915456,915577,915858,916608,917830,917839,918954,918970,919463,920581,920604,921313,922542,922944
CVE References: CVE-2014-8134,CVE-2014-8160,CVE-2014-8559,CVE-2014-9419,CVE-2014-9420,CVE-2014-9428,CVE-2014-9529,CVE-2014-9584,CVE-2014-9585,CVE-2015-0777,CVE-2015-1421,CVE-2015-1593,CVE-2015-2150
Sources used:
openSUSE 13.2 (src):    bbswitch-0.8-3.6.6, cloop-2.639-14.6.6, crash-7.0.8-6.6, hdjmod-1.28-18.7.6, ipset-6.23-6.6, kernel-docs-3.16.7-13.2, kernel-obs-build-3.16.7-13.7, kernel-obs-qa-3.16.7-13.1, kernel-obs-qa-xen-3.16.7-13.1, kernel-source-3.16.7-13.1, kernel-syms-3.16.7-13.1, pcfclock-0.44-260.6.2, vhba-kmp-20140629-2.6.2, virtualbox-4.3.20-10.2, xen-4.4.1_08-12.2, xtables-addons-2.6-6.2
Comment 10 Swamp Workflow Management 2015-04-13 12:19:38 UTC
openSUSE-SU-2015:0714-1: An update that solves 11 vulnerabilities and has 5 fixes is now available.

Category: security (important)
Bug References: 903640,904899,907988,909078,910150,911325,911326,912202,912654,912705,913059,913695,914175,915322,917839,920901
CVE References: CVE-2014-7822,CVE-2014-8134,CVE-2014-8160,CVE-2014-8173,CVE-2014-8559,CVE-2014-9419,CVE-2014-9420,CVE-2014-9529,CVE-2014-9584,CVE-2014-9585,CVE-2015-1593
Sources used:
openSUSE 13.1 (src):    cloop-2.639-11.19.1, crash-7.0.2-2.19.1, hdjmod-1.28-16.19.1, ipset-6.21.1-2.23.1, iscsitarget-, kernel-docs-3.11.10-29.2, kernel-source-3.11.10-29.1, kernel-syms-3.11.10-29.1, ndiswrapper-1.58-19.1, pcfclock-0.44-258.19.1, vhba-kmp-20130607-2.20.1, virtualbox-4.2.28-2.28.1, xen-4.3.3_04-37.1, xtables-addons-2.3-2.19.1
Comment 11 Jiri Slaby 2015-05-20 07:15:03 UTC
Any updates with the updated kernel?
Comment 12 Miroslav Beneš 2015-05-20 07:56:47 UTC
It seems to be ok now. My current uptime is 36 days and I do not see the mentioned problems anymore. I think this can be closed as resolved/fixed. Thanks a lot.
Comment 13 Michal Hocko 2015-05-20 08:03:40 UTC
Thanks for the feedback. Closing...