Bug 1114589 - Add earlyoom package to the system
Add earlyoom package to the system
Status: RESOLVED FEATURE
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Basesystem
Current
Other Other
: P5 - None : Normal (vote)
: ---
Assigned To: Tomáš Chvátal
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-11-04 15:01 UTC by Aleksey Kontsevich
Modified: 2020-02-02 22:17 UTC (History)
4 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
earlyoom.spec (807 bytes, text/x-rpm-spec)
2019-01-06 11:18 UTC, Aleksey Kontsevich
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Aleksey Kontsevich 2018-11-04 15:01:15 UTC
Please add earlyoom package to the system, strange such important daemon does not exist in SUSE repos.

https://github.com/rfjakob/earlyoom
Comment 1 Tomáš Chvátal 2019-01-04 09:55:04 UTC
This is not an issue, rahter package request, as such you are welcome to use https://build.opensuse.org/ to generate such package. You can base it on Fedora package from here:

https://src.fedoraproject.org/cgit/rpms/earlyoom.git/tree/earlyoom.spec
Comment 2 Aleksey Kontsevich 2019-01-06 11:05:10 UTC
(In reply to Tomáš Chvátal from comment #1)
> This is not an issue, rahter package request, as such you are welcome to use
> https://build.opensuse.org/ to generate such package. You can base it on
> Fedora package from here:
> 
> https://src.fedoraproject.org/cgit/rpms/earlyoom.git/tree/earlyoom.spec

That spec does not work in SUSE. There is already created working one: https://build.opensuse.org/package/view_file/home:ps1260/earlyoom/earlyoom.spec?expand=1 - why just not to add it to official repos?
Comment 3 Aleksey Kontsevich 2019-01-06 11:17:58 UTC
(In reply to Aleksey Kontsevich from comment #2)
> There is already created working one:
> https://build.opensuse.org/package/view_file/home:ps1260/earlyoom/earlyoom.
> spec?expand=1 - why just not to add it to official repos?

I've fixed above spec to register earlyoom as a service. See attached spec below.
Comment 4 Aleksey Kontsevich 2019-01-06 11:18:40 UTC
Created attachment 793718 [details]
earlyoom.spec
Comment 5 Aleksey Kontsevich 2019-01-06 11:19:18 UTC
(In reply to Aleksey Kontsevich from comment #4)
> Created attachment 793718 [details]
> earlyoom.spec

Sources:  https://github.com/rfjakob/earlyoom/archive/v1.2.tar.gz
Comment 6 Tomáš Chvátal 2019-07-03 11:14:54 UTC
We are not going to do the package for you, you have to integrate it yourself if you want it using the workflows like everyone else.

https://en.opensuse.org/openSUSE:Factory_development_model
Comment 7 Aleksey Kontsevich 2020-01-06 02:53:24 UTC
What about this package: https://software.opensuse.org/download/package?package=earlyoom&project=home%3Aembar-%3ALietukas - works fine for me. Whether specs are correct there and could be included in Factory?
Comment 8 Aleksey Kontsevich 2020-01-06 02:58:24 UTC
They are going to include earlyoom into Fedora 32 by default:

- https://www.mail-archive.com/devel-announce@lists.fedoraproject.org/msg02139.html
- https://fedoraproject.org/wiki/Changes/EnableEarlyoom
- https://pagure.io/fedora-workstation/issue/119

Think it is good for openSUSE to do the same as this problem appears quite often and very annoying and non-maintainable for now. Thanks!
Comment 9 Sasi Olin 2020-01-06 10:20:31 UTC
Sure, please submit the package to factory and the required changes to the desktop patterns.
Comment 10 Aleksey Kontsevich 2020-01-06 11:09:31 UTC
(In reply to Stasiek Michalski from comment #9)
> Sure, please submit the package to factory and the required changes to the
> desktop patterns.

Never did this before. :) Any instructions? Thanks!
Comment 11 Sasi Olin 2020-01-06 11:54:37 UTC
Hop on irc://chat.freenode.net/opensuse-packaging, https://matrix.to/#/#opensuse-packaging:matrix.org or https://discord.gg/opensuse channel packaging and we can help you get this going :P
Comment 12 Mindaugas Baranauskas 2020-01-06 14:16:19 UTC
Early OOM package now lives in 
https://build.opensuse.org/package/show/Base:System/earlyoom
Comment 13 Aleksey Kontsevich 2020-01-06 15:44:12 UTC
(In reply to Mindaugas Baranauskas from comment #12)
> Early OOM package now lives in 
> https://build.opensuse.org/package/show/Base:System/earlyoom

Thanks Mindaugas much!!!
Comment 14 Mindaugas Baranauskas 2020-01-06 16:30:57 UTC
After some clean up, submit request opened for openSUSE:Factory
https://build.opensuse.org/request/show/761212
Comment 15 William Brown 2020-01-06 22:50:38 UTC
This may be a diversion from the packaging request. Thanks Mindaugas for helping with this!

Alekesey, you mentioned you needed some instructions here. Did you find that https://en.opensuse.org/openSUSE:Factory_development_model wasn't helpful as a resource? What kind of instructions did you think were missing and what would have helped you as "documentation" to enable you to do the contribution yourself? How can we make the process clearer? 

As much as I appreciate Stasiek's offer to help in chat, sadly that doesn't scale, and has timezone and other accessibility problems, so we really should look at how we could improve documentation of how to contribute and make information locality and focus.

Thanks,
Comment 16 Aleksey Kontsevich 2020-01-07 08:23:57 UTC
(In reply to William Brown from comment #15)
> This may be a diversion from the packaging request. Thanks Mindaugas for
> helping with this!

No, not a diversion, I'd asked Mindaugas to do this.

> Alekesey, you mentioned you needed some instructions here. Did you find that
> https://en.opensuse.org/openSUSE:Factory_development_model wasn't helpful as
> a resource? What kind of instructions did you think were missing and what
> would have helped you as "documentation" to enable you to do the
> contribution yourself? How can we make the process clearer? 

I just thought it will take much time for me as I never did this before, need to understand OBS workflow, etc. So I just asked Mindaugas to do this as it it his package, it already works and good enough and he has experience to do this: https://build.opensuse.org/package/show/home%3Aembar-%3ALietukas/earlyoom#comment-1132736

And thanks Mindaugas he responded to this immediately.
Comment 17 William Brown 2020-01-07 22:58:08 UTC
Do you mind if I send you some questions to your email about this topic? The comment that stands out to me is:

> I just thought it will take much time for me as I never did this before,
> need to understand OBS workflow, etc

So I fear we have an unclear or intimidating process, and I would like to ask you some more about "how to make it easier" to help you and others in the future. 

Thanks!
Comment 18 Mindaugas Baranauskas 2020-01-10 15:31:58 UTC
Done with https://build.opensuse.org/request/show/761913
Comment 19 Aleksey Kontsevich 2020-01-10 15:56:29 UTC
(In reply to Mindaugas Baranauskas from comment #18)
> Done with https://build.opensuse.org/request/show/761913

Hurray! Great! Thanks a lot!
Comment 20 Mindaugas Baranauskas 2020-01-10 16:12:02 UTC
Although now this package exist in default openSUSE Factory repository, 
it does not mean it will be pre-installed (auto-installed) with some pattern.
Comment 21 Aleksey Kontsevich 2020-02-01 16:09:37 UTC
(In reply to Mindaugas Baranauskas from comment #20)
> Although now this package exist in default openSUSE Factory repository, 
> it does not mean it will be pre-installed (auto-installed) with some pattern.

Mindaugas, however earlyoom does not work for me: my custom app has memory leak and it eats all the memory very fast and earlyoom does not kill it. After this I forced to kill it manually then to kill and restart plasma and firefox as they unloaded to swap and become unresponsive. Very inconvenient! Any ideas? Change some settings or earlyoom just unable to kill some apps?
Comment 22 Mindaugas Baranauskas 2020-02-01 16:17:22 UTC
EarlyOOM allows to set your own conditions in /etc/sysconfig/earlyoom , e.g.
you can set different percentage (or KB) for RAM and for SWAP,
also set whitelist and blacklist of programs. 

See configuration example in https://build.opensuse.org/package/view_file/openSUSE:Factory/earlyoom/earlyoom.sysconfig

-r option is for checking interval (seconds)
-m option is for RAM critical level (%) [or -M for KiB]
-s option is for SWAP critical level (%) [or -S for KiB]
more info: `man earlyoom`

If you will find optimal values, we can set them as default
Comment 23 Aleksey Kontsevich 2020-02-01 16:44:21 UTC
(In reply to Mindaugas Baranauskas from comment #22)
> EarlyOOM allows to set your own conditions in /etc/sysconfig/earlyoom , e.g.
> you can set different percentage (or KB) for RAM and for SWAP,
> also set whitelist and blacklist of programs. 
> 
> See configuration example in
> https://build.opensuse.org/package/view_file/openSUSE:Factory/earlyoom/
> earlyoom.sysconfig
> 
> -r option is for checking interval (seconds)
> -m option is for RAM critical level (%) [or -M for KiB]
> -s option is for SWAP critical level (%) [or -S for KiB]
> more info: `man earlyoom`
> 
> If you will find optimal values, we can set them as default

-r is 3
-m set to 10 - 1.6 GB however it goes significantly below this level (about 700 MB) and system become unresponsive.

if I do not want to swap at all, set -s to 0 or to 100?

What happened to Linux?! In 2.x versions swap works perfect unloaded applications to swap and put them back immediately like a charm. Now forced to kill an app and restart it or even reboot PC to have it back. What is wrong here? Any way to fix?
Comment 24 Aleksey Kontsevich 2020-02-01 16:50:13 UTC
Set -s 100 and added -P switch.
Comment 25 Aleksey Kontsevich 2020-02-01 16:51:50 UTC
(In reply to Aleksey Kontsevich from comment #24)
> Set -s 100 and added -p switch.

   -s PERCENT[,KILL_PERCENT]
       set free swap minimum to PERCENT of total (default 10 %).  Send SIGKILL if at or below KILL_PERCENT (default PERCENT/2), otherwise SIGTERM.

       You can use -s 100 to have earlyoom effectively ignore swap usage: Processes are killed once available memory drops below the configured minimum, no  matter much swap is free.

 -p
       Increase earlyoom’s priority: set niceness of earlyoom to -20 and oom_score_adj to -1000
Comment 26 Aleksey Kontsevich 2020-02-01 16:59:26 UTC
Hmmm, I see need to set both: SIG_KILL, and SET_TERM percents, that is why it did not work for me.

>-s PERCENT[,KILL_PERCENT]
>       set free swap minimum to PERCENT of total (default 10 %).  Send SIGKILL >if at or below KILL_PERCENT (default PERCENT/2), otherwise SIGTERM.

So for me correct parameters will be:

>EARLYOOM_ARGS="-p -r 3 -m 11,10 -s 100,100 -n --avoid '(^|/)(systemd|Xorg|X|Xwayland|xdm|ssdm|kdm|gdm|lightdm|ssh|yast|yast2|y2controlcenter|zypper|rpm)$' --prefer '(^|/)(java|firefox|waterfox|chromium|chrome|opera|ffmpeg|vlc|akregator|thumbnail.so)$'"

Mindaugas could You fix earlyoom.sysconfig and set above parameters or similar, please?
Comment 27 Aleksey Kontsevich 2020-02-01 17:01:21 UTC
Now I see this in logs:

sending SIGTERM when mem <= 11 % and swap <= 100 %,
        SIGKILL when mem <= 10 % and swap <= 100 %

Previously was
        SIGKILL when mem <= 5 % and swap <= 50 %

Which did not work for me.
Comment 28 Mindaugas Baranauskas 2020-02-01 17:13:28 UTC
(In reply to Aleksey Kontsevich from comment #24)
> Set -s 100 and added -P switch.

"-s 100" This is too much restrictive.
"-p" we can add.
Comment 29 Mindaugas Baranauskas 2020-02-01 17:16:20 UTC
(In reply to Aleksey Kontsevich from comment #23) 
> -r is 3
> -m set to 10 - 1.6 GB however it goes significantly below this level (about
> 700 MB) and system become unresponsive.

Should work (for SIGTERM) if both conditions are met: both RAM and SWAP bellow critical level. If only one type of memory is nearly fully used, EarlyOom will do nothing.
Comment 30 Aleksey Kontsevich 2020-02-01 17:35:53 UTC
(In reply to Mindaugas Baranauskas from comment #28)
> (In reply to Aleksey Kontsevich from comment #24)
> > Set -s 100 and added -P switch.
> 
> "-s 100" This is too much restrictive.

No at all! As I can't get system to work after swap: kernel does not unload applications from swap  - forced to reboot. Also start swapping does not makes much sense moreover with 20% as it will take hours until single application will fill 16 GB of the swap and kernel will load/unload it all the time.

If single app - takes all the memory it should be killed.

And what happened with linux?! Why swap starts to work so badly like I said in  Comment 23?

> "-p" we can add.

Thanks!
Comment 31 Aleksey Kontsevich 2020-02-01 17:44:33 UTC
Now earlyoom works fine for me - killed the app. I mean need to specify both values in  earlyoom.sysconfig:

-m PERCENT[,KILL_PERCENT]
-s PERCENT[,KILL_PERCENT]

And add comments about both values from man page and possibility of -s 100 switch for users to avoid confusion. Thanks much!
Comment 32 Mindaugas Baranauskas 2020-02-01 18:16:51 UTC
However, this is purpose of SWAP to use it as extended RAM. If you don't need SWAP at all (e.g. your hard disk toooooo slooowwww), you can format SWAP partition into some user-usable partition format like Ext4,NTFS etc
Comment 33 Aleksey Kontsevich 2020-02-01 18:28:16 UTC
(In reply to Mindaugas Baranauskas from comment #32)
> However, this is purpose of SWAP to use it as extended RAM. If you don't
> need SWAP at all (e.g. your hard disk toooooo slooowwww), you can format
> SWAP partition into some user-usable partition format like Ext4,NTFS etc

too slow?! Are You kidding? It is not an SSD but newest WD with 250 MB/s speed. The problem is application constantly grows in size because of memory leak and eats all the memory and swap, and while other applications are relatively small unloading them to swap is useless. And after app is killed and I can't get unloaded apps to work - they are unresponsive: whether kernel or KDE fails to tell kernel to unload them so forced to reboot machine. Which was not the case in 2.x versions of kernel where I was able to set swat 2-3 times bigger than memory, while now 1:1 proportion is useless. So trying to understand what is wrong here: kernel settings, KDE or whatever?
Comment 34 Mindaugas Baranauskas 2020-02-01 20:26:21 UTC
(In reply to Aleksey Kontsevich from comment #33)
> ... in 2.x versions of kernel where I was able
> to set swat 2-3 times bigger than memory, while now 1:1 proportion is
> useless. So trying to understand what is wrong here: kernel settings, KDE or
> whatever?

I don't know. You may ask in openSUSE forum or in opensuse-support
mailing list <opensuse-support@opensuse.org> - maybe someone will respond
Comment 35 Mindaugas Baranauskas 2020-02-01 20:31:57 UTC
What is name of problematic program?
You can add it into blacklist after  --prefer option.
e.g. --prefer '(^|/)(java|firefox|chromium|chrome|opera|akregator|thumbnail.so)
Comment 36 Aleksey Kontsevich 2020-02-01 21:36:40 UTC
> I don't know. You may ask in openSUSE forum or in opensuse-support
> mailing list <opensuse-support@opensuse.org> - maybe someone will respond

Thanks: https://forums.opensuse.org/showthread.php/538958-Strange-swap-behavior
Comment 37 Aleksey Kontsevich 2020-02-02 22:17:54 UTC
(In reply to Mindaugas Baranauskas from comment #28)
> "-s 100" This is too much restrictive.

Hi Mindaugas, so this is confirmed upstream bug since kernel 4.7: https://forums.opensuse.org/showthread.php/538958-Strange-swap-behavior?p=2926144#post2926144

So I think 
> -s 100,100
switch is completely correct here to set by default.