Bugzilla – Bug 1112824
Severe Performance Degradation with update to latest tumbleweed (20181018)
Last modified: 2020-06-26 14:39:14 UTC
Created attachment 786777 [details] journalctl output There is a severe performance degradation when upgrading to the latest version of tumbleweed. Mouse movements are extremely jittery, typing is choppy, slow, and will often repeat characters which were only typed once. Behavior is consistent with all things utilizing gnome-shell version 3.30.0 and 3.30.1 (attempted to use GNOME_factory repository). Performance is bad enough to render the entire desktop environment virtually unusable. Journalctl Logs: ================ (See attached file). System Information: =================== (See attached file).
Created attachment 786778 [details] System Information
Same here. Dell XPS 13 9360 It doesn't seem kernel related, both kernel 4.18.14-1 and 4.18.13-1 have same symptoms on snapshot 20181018 For now i'm back at snapshot 20181015, because it's unusable in this state
Same here: Dell Optiplex 790 Running GNOME on XOrg Tried running without extensions, but it is the same.
We just prepared some fixes in the devel project GNOME:Factory - from first tests, you will need the updates for gjs (i.e. gjs-1.54.2-163.1.x86_64.rpm and libgjs0-1.54.2-163.1.x86_64.rpm) and gnome-shell (gnome-shell-3.30.1-393.1.x86_64.rpm) (release numbers could change when rebuilds happen, those are meant as 'minimum numbers') All packages are the GNOME:Factory repo http://download.opensuse.org/repositories/GNOME:/Factory/openSUSE_Factory If anybody can do further tests and confirm this also being fixed for you, this would be appreciated
I installed the following packages and the responsiveness has greatly improved: gjs-1.54.2-163.1.x86_64.rpm gnome-shell-3.30.1-393.1.x86_64.rpm gnome-shell-browser-plugin-3.30.1-393.1.x86_64.rpm gnome-shell-calendar-3.30.1-393.1.x86_64.rpm gnome-shell-devel-3.30.1-393.1.x86_64.rpm gnome-shell-lang-3.30.1-393.1.noarch.rpm libgjs0-1.54.2-163.1.x86_64.rpm The CPU spikes are gone from the gnome-shell process as monitored via top -p $(pidof gnome-shell| tr ' ' ',') . I'm intentionally not removing the needinfo flag as my laptop did not have smooth animations before the update either so I'm not sure if the problem is entirely gone.
I also installed these packages, and it's working fine now
Installing gnome-shell, libgjs0, and gjs helps (a lot!) but I still see severe choppiness when the shell is in application overview mode. For instance, when I try clicking on the YaST2 Modules, it takes of the order of 5 seconds to unfold. When in application overview mode, mouse movement also becomes choppy generally. Plus, video playing is still choppy -- not as much as it was with TW 20181018 but nowhere near as smooth as 3.28.x levels before that.
I am on wayland btw.
I can confirm comment #7.
Hey, Comment 5 works for me aswell! Huge improvement!
Running GNOME on XOrg with Nvidia blob After installing packages from comment 4 gjs-1.54.2-163.1.x86_64.rpm gnome-shell-3.30.1-393.1.x86_64.rpm gnome-shell-calendar-3.30.1-393.1.x86_64.rpm gnome-shell-lang-3.30.1-393.1.noarch.rpm libgjs0-1.54.2-163.1.x86_64.rpm the performance is restored.
I can also confirm a huge difference. But I do not have the feeling it's already back to normal. Looking at top gnome-shell still causes quite high CPU utilization on my i7-7500. I would expect still a lot less load. But at least it's somewhat usable again.
After installing the suggested packages, there seems to be a major performance increase in the standard UI, but when using the activities overview, there is still a lot of lag and slowness. It makes it much more usable, but it still needs work.
The Wayland session is still unusable, though much better than before, it is significantly worse than 3.28.
https://bugzilla.opensuse.org/show_bug.cgi?id=1112824#c4 +1 works on me
Can confirm that the packages pushed on the 24th fixed the worst of it, but there are still periodic freezes.
Now in 20181022 update, nautillus is too slow to open but everything else is fine.
*** Bug 1114587 has been marked as a duplicate of this bug. ***
I observe the same problems even with 20181029 snapshot https://youtu.be/l_fZvjqnuYM https://youtu.be/6vMaKK346No http://paste.opensuse.org/34989715
In fact, the problem appeared from gnome 3.26, only there it was not much noticeable, but with each release gnome it got worse, now it is practically impossible to use it, and this problem is only in opensuse
Created attachment 790009 [details] perf gnome 3.30.2 I checked the utility perf, at the output, he got this conclusion, for some reason the unknown has a lot of shared object
I think it is worth noting that this does not happen in Ubuntu 18.10 or Fedora 29, which use gnome 3.30.
It seems the problem is only with Intel processors, only openSUSE, there is no problem with other distributions, I checked
(In reply to Dead Mozay from comment #23) > It seems the problem is only with Intel processors, only openSUSE, there is > no problem with other distributions, I checked Does that mean Intel CPUs have the problem and AMD CPUs don't? Or does it mean systems using the Intel iGPU have the problem and systems using discrete AMD / NVidia GPUs don't?
(In reply to Robert Munteanu from comment #24) > (In reply to Dead Mozay from comment #23) > > It seems the problem is only with Intel processors, only openSUSE, there is > > no problem with other distributions, I checked > > Does that mean Intel CPUs have the problem and AMD CPUs don't? Or does it > mean systems using the Intel iGPU have the problem and systems using > discrete AMD / NVidia GPUs don't? There is a problem with Intel CPU, various benchmarks show a drop in performance in multi-core testing, sometimes decent, just checked with other distributions, there is no such thing. I also tried both the integrated graphics and the discrete, and quite good, I have gtx 1070, the difference with the intel hd is no
I also reassembled all the components that could have a negative impact on the performance of the gnome, and 3d, xorg, mesa, gnome-shell, mutter, gjs with patches fedora, even compile the kernel from the source code is fresher without patches, it seems to be better, but sometimes it slowed down anyway, apparently somewhere else there is a problem, I could not find anything else
in general, as far as I can tell, the problem is not in the gnome, but somewhere in the depths of the system packages. after compiling the kernel from the source code, the problem with hyper-threading disappeared, the gnome with the enabled hyper-threading showed slideshows instead of animations
(In reply to Dead Mozay from comment #27) > in general, as far as I can tell, the problem is not in the gnome, but > somewhere in the depths of the system packages. after compiling the kernel > from the source code, the problem with hyper-threading disappeared, the > gnome with the enabled hyper-threading showed slideshows instead of > animations Interesting. Is this the same kernel version as in Tumbleweed? Did you use the same config? This might help tracking down the root cause.
(In reply to Robert Munteanu from comment #28) > (In reply to Dead Mozay from comment #27) > > in general, as far as I can tell, the problem is not in the gnome, but > > somewhere in the depths of the system packages. after compiling the kernel > > from the source code, the problem with hyper-threading disappeared, the > > gnome with the enabled hyper-threading showed slideshows instead of > > animations > > Interesting. Is this the same kernel version as in Tumbleweed? Did you use > the same config? This might help tracking down the root cause. I compiled 4.19.2, but the problem appeared a long time ago, the configuration used the standard one, without changes, I wanted to compare it to Fedora config, but there it is missing, you need to get it from makemenu, only all hands in any way will not reach to save a config. I tried to identify the problem, but it did not work, all my attempts were unsuccessful there is little experience and knowledge, if you tell me what you need, I will try to help
Created attachment 790294 [details] kernel config It seems that it was possible to solve the problem, at least on the Intel HD animations, they became smooth, so that someone else would check it, you need to rebuild the kernel with the attached config
(In reply to Dead Mozay from comment #30) > Created attachment 790294 [details] > kernel config > > It seems that it was possible to solve the problem, at least on the Intel HD > animations, they became smooth, so that someone else would check it, you > need to rebuild the kernel with the attached config Thanks for the config! I tried to compare it with the one shipped in TW, but unfortunately there are too many differences: $ zcat /proc/config.gz | diff - kconfig-suggested | wc -l 4621 I'll try and test your assumption that this is specific to Intel CPUs soon though, I have an AMD machine that I have not updated yet.
(In reply to Robert Munteanu from comment #31) > (In reply to Dead Mozay from comment #30) > > Created attachment 790294 [details] > > kernel config > > > > It seems that it was possible to solve the problem, at least on the Intel HD > > animations, they became smooth, so that someone else would check it, you > > need to rebuild the kernel with the attached config > > > Thanks for the config! I tried to compare it with the one shipped in TW, but > unfortunately there are too many differences: > > $ zcat /proc/config.gz | diff - kconfig-suggested | wc -l > 4621 > > I'll try and test your assumption that this is specific to Intel CPUs soon > though, I have an AMD machine that I have not updated yet. It's my pleasure I just concluded, everyone who said that they had no problems with the gnome, everyone CPU AMD. There is also a possibility that a patch is guilty, and everything is fine with the config, I compiled it manually without patches. PS And yes, the problem really disappeared, I launched several programs, a browser with a lot of tabs, the animations remained smooth, there are jerks, but usual for a gnome and not like they were before
There is a possibility that the spectre mitigations have something to do with all this, and or hyperthreading/smt. I tested my system on the impact of those mitigations. As a reference I used a mouse click on "Show applications" item on the gnome-shell menu, running wayland. Watching if the application icons transition smoothly from the corner into the main desktop area. Hardware: Dell XPS 13 9360 Snapshot 20181118 Kernel 4.19.2-1 My subjective results 1. boot with default kernel parameters Transition is barely noticeable, not smooth for sure 2. boot with kernel parameters: pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier Transition is very smooth 3. boot with kernel parameters: spectre_v2=off Transition is smooth This parameter seems to have the biggest impact of all the tested parameters. Now I read about the new recently disclosed spectre v2 vulnerabilities and the STIBP mitigation. Some voices say that we might as well disable hyperthreading/smt instead of these new mitigations. 4. boot with default kernel parameters, and hyperthreading disabled in BIOS Transitions are smooth. For the moment I will leave all mitigations enabled and hyperthreading disabled. For the record, this is an Intel Kabylake cpu
(In reply to Michiel Janssens from comment #33) > There is a possibility that the spectre mitigations have something to do > with all this, and or hyperthreading/smt. > > 3. boot with kernel parameters: > spectre_v2=off > Transition is smooth > This parameter seems to have the biggest impact of all the tested parameters. > Early days -- minutes, actually -- but this seems to help a lot in my case too (Intel i7-7*** processors). There are still some animation choppiness on Application overview, but nothing like before.
(In reply to Michiel Janssens from comment #33) > 3. boot with kernel parameters: > spectre_v2=off > Transition is smooth > This parameter seems to have the biggest impact of all the tested parameters. > Thanks a LOT for figuring this out. But reading above that only openSUSE is affected - I'd assume that Ubuntu and Fedora are also shipping with these mitigations enabled by default, no?
(In reply to Michiel Janssens from comment #33) > There is a possibility that the spectre mitigations have something to do > with all this, and or hyperthreading/smt. > I tested my system on the impact of those mitigations. > As a reference I used a mouse click on "Show applications" item on the > gnome-shell menu, running wayland. Watching if the application icons > transition smoothly from the corner into the main desktop area. > Hardware: Dell XPS 13 9360 > Snapshot 20181118 > Kernel 4.19.2-1 > > My subjective results > 1. boot with default kernel parameters > Transition is barely noticeable, not smooth for sure > > 2. boot with kernel parameters: > pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier > Transition is very smooth > > 3. boot with kernel parameters: > spectre_v2=off > Transition is smooth > This parameter seems to have the biggest impact of all the tested parameters. > > Now I read about the new recently disclosed spectre v2 vulnerabilities and > the STIBP mitigation. Some voices say that we might as well disable > hyperthreading/smt instead of these new mitigations. > > 4. boot with default kernel parameters, and hyperthreading disabled in BIOS > Transitions are smooth. > > For the moment I will leave all mitigations enabled and hyperthreading > disabled. > For the record, this is an Intel Kabylake cpu Some more data points: - for my Lenovo W541 laptop - i7-4810MQ with iGPU, none of these helped. The animations are maybe slightly faster, but still laggy - after updating my desktop ( Ryzen 5 2600X, Nvidia 1070, Samsung EVO 970 NVME SSD ) from Gnome 3.28, the animations are smooth, but I feel I slight stutter when pressing buttons for the first time, e.g. the applications button. Maybe I'm imagining, I don't now. What I think would be useful is for those affected and for which the various boot options worked to clarify whether they were using a discrete GPU or the integrated GPU.
(In reply to Robert Munteanu from comment #36) > What I think would be useful is for those affected and for which the various > boot options worked to clarify whether they were using a discrete GPU or the > integrated GPU. In my case integrated GPU: Intel® HD Graphics 620 (Kaby Lake GT2).
Oddly, I just now booted to kernel 4.18.12 -- the version of kernel with which GNOME 3.28.x was performing really well. With no changes to the default boot parameters, GNOME 3.30.2's performance is again awful, stuttering. It seems GNOME 3.30.x + recent kernel = problem.
I can confirm that there is an improvement with spectre mitigations off. It is not as good as I would like, it stutters quite a bit, but it runs better I have an i7-6500U.
disabled pti=off spectre_v2=off l1tf=off the problem does not solve, I checked it first. Here is the animation with rebuild kernel https://youtu.be/MdX2GM79w30, Protection against these vulnerabilities is enabled.
(In reply to Martin Wilck from comment #35) > (In reply to Michiel Janssens from comment #33) > > 3. boot with kernel parameters: > > spectre_v2=off > > Transition is smooth > > This parameter seems to have the biggest impact of all the tested parameters. > > > > Thanks a LOT for figuring this out. But reading above that only openSUSE is > affected - I'd assume that Ubuntu and Fedora are also shipping with these > mitigations enabled by default, no? So I had a look upon fedora29. - kernel version 4.18.17 - gnome version 3.30.2 hyperthreading on and it seems spectre protection is also turned on So presumably the only difference we have is that kernel version is different. What do you think?
(In reply to Zhiyuan Gao from comment #41) > (In reply to Martin Wilck from comment #35) > > (In reply to Michiel Janssens from comment #33) > > > 3. boot with kernel parameters: > > > spectre_v2=off > > > Transition is smooth > > > This parameter seems to have the biggest impact of all the tested parameters. > > > > > > > Thanks a LOT for figuring this out. But reading above that only openSUSE is > > affected - I'd assume that Ubuntu and Fedora are also shipping with these > > mitigations enabled by default, no? > > So I had a look upon fedora29. > - kernel version 4.18.17 > - gnome version 3.30.2 > hyperthreading on and it seems spectre protection is also turned on > > So presumably the only difference we have is that kernel version is > different. > > What do you think? Fedora kernel 4.19.2 there are no problems either
(In reply to Atri Bhattacharya from comment #37) > (In reply to Robert Munteanu from comment #36) > > What I think would be useful is for those affected and for which the various > > boot options worked to clarify whether they were using a discrete GPU or the > > integrated GPU. > > In my case integrated GPU: Intel® HD Graphics 620 (Kaby Lake GT2). I have the same iGPU. Intel HD 620 KL GT2 on i7-7500U cpu. Also have to mention that I've mainly tested with 2 connected 24inch external monitors, so the gpu has to work harder and artifacts are easier to spot.
Intel UHD 620 on n i7-8550u. I disabled all spectre patches in 4.19.2 and it's better but still sluggish. Whatever the real problem is it was worsened by these patches. Seeing as fedora has the patches and no issues, there is something else that is broken. Someone suggested the config is to blame or an old suse patch.
After several days of tests with a compiled kernel without patches, I can say that this partially solved the problem, the application review is smooth, but the activites still sometimes slows down, just when I launch the application from Dash, there are also signatures sometimes, while reading Twitter yesterday I came across several posts about The fact that the brakes appeared everywhere when the activities was activated. perhaps this is due to patches spectre v2, now they are just deciding what to do with them, Linus suggests that they be removed from the code at all, he considers it unwise to use them in the kernel, referring to the fact that it will be necessary for someone to disable the SMT in the bios. Corrections promise in GNOME 3.31, backports may be done, but this is not accurate.
and also found out that there are no patches STIBP in Fedora, in openSUSE this patch is there, maybe that's the problem
(In reply to Dead Mozay from comment #46) > and also found out that there are no patches STIBP in Fedora, in openSUSE > this patch is there, maybe that's the problem This would make sense if the performance regressions were only observed with 4.19.2 and newer, where STIBP was first enabled in openSUSE. But people have started reporting this with earlier kernels. If people are running 4.19.2, STIBP likely contributes to the slowness.
So what info/logs would be helpful to fix this?
(In reply to Benjamin Sabatini from comment #48) > So what info/logs would be helpful to fix this? problem fixed? I just have a self-assembly kernel, at the same time there was an update to the gnome, now I’m not sure what exactly improved the performance
I tried kernel-vanilla and I found that it runs even better than kernel-default with spectre mitigations off. Can anybody replicate my findings?
(In reply to Nicolás Luciano Bértolo from comment #50) > I tried kernel-vanilla and I found that it runs even better than > kernel-default with spectre mitigations off. Can anybody replicate my > findings? in 4.19.4 a patch has already been adopted that disables stibp, by the way stibp was enabled in 4.14, but yes, performance improves with vanilla kernel
(In reply to Dead Mozay from comment #51) > (In reply to Nicolás Luciano Bértolo from comment #50) > > I tried kernel-vanilla and I found that it runs even better than > > kernel-default with spectre mitigations off. Can anybody replicate my > > findings? > > in 4.19.4 a patch has already been adopted that disables stibp, by the way > stibp was enabled in 4.14, but yes, performance improves with vanilla kernel So problem not fixed in the default kernel. As many have said several times, the problem was pre 4.19.2. In other words, it is not because of STIBP.
(In reply to Nicolás Luciano Bértolo from comment #50) > I tried kernel-vanilla and I found that it runs even better than > kernel-default with spectre mitigations off. Can anybody replicate my > findings? For my system they give similar behavior, so a lot better then just kernel-default.
Today I bought a new laptop, installed openSUSE Tumbleweed with GNOME, looks bad, installed kernel-vanilla, It got a little better, tried to turn off all the patches on spectre, meltdown and l1tf (pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier) It became a little better, but still bad, on the workstation with the enabled hyper threading performance is terrible too, if you turn off HT, it becomes much better, Unfortunately on the laptop HT can not be disabled. Who has the ability to turn off HT, check how you will behave GNOME. kernel options do not help, you need to disable the BIOS. Again, this problem is only in openSUSE.
(In reply to Nicolás Luciano Bértolo from comment #50) > I tried kernel-vanilla and I found that it runs even better than > kernel-default with spectre mitigations off. Can anybody replicate my > findings? vanilla kernel + spectre fixes enabled works well for me. Disabling spectre is even better.
In general, everything that I tried, does not help, rebuild the kernel without patches, compile the kernel from appstrim, disable spectre, rebuild gnome-shell, gjs, xorg, mesa, mutter, try kernel-vanilla, it all partially solved the problem, but still not as it should be. The best option is disable Hyper threading, only on my laptop I have no such opportunity, on my PC it is possible,. installed openSUSE Leap, with enabled Hyper threading the problem is there (GNOME 3.26). Installed Fedora, GNOME works perfectly, despite being turned Hyper threading or not. I have no more ideas.
Is there any news on this issue? With each update it gets worse on some hardware configurations. Nothing else helps, even disabling hyper-threading also partially solves the problem. Unfortunately, not everywhere you can turn it off.
Boris, could you take a look? This seems relevant with the stuff you know well... Put Jiri on Cc, too.
Can anyone of the gazillion people on this bugzilla build the upstream v4.20-rc5 kernel and test with it? We have disabled STIBP et al upstream and will backport this to our kernels eventually. Thx.
(In reply to Borislav Petkov from comment #59) > Can anyone of the gazillion people on this bugzilla build the upstream > v4.20-rc5 kernel and test with it? > > We have disabled STIBP et al upstream and will backport this to our kernels > eventually. > > Thx. disabling stibp does not solve the problem, I tried, I also deleted a number of patches that could affect performance, nothing changes. I also found that the problem also exists in kernel 4.12, and problems started with it
(In reply to Borislav Petkov from comment #59) > Can anyone of the gazillion people on this bugzilla build the upstream > v4.20-rc5 kernel and test with it? FWIW, OBS Kernel:HEAD repo already contains the latest upstream 4.20-rc5: https://download.opensuse.org/repositories/Kernel:/HEAD/standard/ You can try kernel-vanilla from that repo.
(In reply to Dead Mozay from comment #60) > disabling stibp does not solve the problem, I tried, I also deleted a number > of patches that could affect performance, nothing changes. I also found that > the problem also exists in kernel 4.12, and problems started with it Does that mean that if you use something else besides gnome, your performance regression is gone?
(In reply to Borislav Petkov from comment #62) > Does that mean that if you use something else besides gnome, your > performance regression is gone? yes, it seems to be an interaction between GNOME and the kernel
(In reply to Dominique Leuenberger from comment #63) > yes, it seems to be an interaction between GNOME and the kernel If the performance regression goes away when you use a different desktop env than gnome, then it is clearly a gnome issue. Otherwise, you'd have to be more specific about this "interaction". Maybe perf-profile a typical workload which causes the regression, etc. Thx.
(In reply to Takashi Iwai from comment #61) > (In reply to Borislav Petkov from comment #59) > > Can anyone of the gazillion people on this bugzilla build the upstream > > v4.20-rc5 kernel and test with it? > > FWIW, OBS Kernel:HEAD repo already contains the latest upstream 4.20-rc5: > https://download.opensuse.org/repositories/Kernel:/HEAD/standard/ > You can try kernel-vanilla from that repo. nothing changed
(In reply to Dead Mozay from comment #65) > nothing changed If booting that same kernel with spectre_v2=off spectre_v2_user=off spec_store_bypass_disable=off l1tf=off on the kernel command line still doesn't change anything, then the problem is somewhere else.
(In reply to Borislav Petkov from comment #64) > (In reply to Dominique Leuenberger from comment #63) > > yes, it seems to be an interaction between GNOME and the kernel > > If the performance regression goes away when you use a different desktop > env than gnome, then it is clearly a gnome issue. > > Otherwise, you'd have to be more specific about this "interaction". > Maybe perf-profile a typical workload which causes the regression, etc. > > Thx. it is possible, and it is also possible if the CPU will not work correctly, problems will also be observed in the gnome, the gnome is very dependent on how the CPU works. At the beginning of the search for a problem, I did synthetic tests, they showed a drop in multi-core tests, which in turn could well cause a similar problem in the GNOME
(In reply to Dead Mozay from comment #67) > it is possible, and it is also possible if the CPU will not work correctly, What does that mean? > problems will also be observed in the gnome, the gnome is very dependent on > how the CPU works. s/gnome/every program/ > At the beginning of the search for a problem, I did synthetic tests, > they showed a drop in multi-core tests, which in turn could well cause > a similar problem in the GNOME I have no clue what that means.
Created attachment 791977 [details] Tests I'm talking about it, openSUSE boot with pti=off spectre_v2=off l1tf=off nospec_store_bypass_disable no_stf_barrier, Fedora with default settings. In Fedora no problem even with enabled hyper-threading.
(In reply to Borislav Petkov from comment #66) > (In reply to Dead Mozay from comment #65) > > nothing changed > > If booting that same kernel with > > spectre_v2=off spectre_v2_user=off spec_store_bypass_disable=off l1tf=off > > on the kernel command line still doesn't change anything, then the > problem is somewhere else. I can confirm that those parameters help a lot. But in my tests I couldn't find a difference between 4.19 and 4.20. 4.20 with mitigations enabled is just as bad as 4.19-default.
(In reply to Dead Mozay from comment #69) > I'm talking about it, openSUSE boot with pti=off spectre_v2=off l1tf=off > nospec_store_bypass_disable no_stf_barrier, Fedora with default settings. In > Fedora no problem even with enabled hyper-threading. Can you upload full dmesg from the Fedora kernel and also do $ grep . /sys/devices/system/cpu/vulnerabilities/* > vuln.log on it and upload that vuln.log file too? Thx.
(In reply to Nicolás Luciano Bértolo from comment #70) > I can confirm that those parameters help a lot. But in my tests I couldn't > find a difference between 4.19 and 4.20. 4.20 with mitigations enabled is > just as bad as 4.19-default. Well, the mitigations do cost and we've tried to make them as unpunishing as possible. For example, my workstation with 20-rc5 has: /sys/devices/system/cpu/vulnerabilities/l1tf:Not affected /sys/devices/system/cpu/vulnerabilities/meltdown:Not affected /sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Mitigation: Speculative Store Bypass disabled via prctl and seccomp /sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: __user pointer sanitization /sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full AMD retpoline, IBPB: conditional, STIBP: disabled, RSB filling which means that SSB will be enabled only for applications which request it (prctl and seccomp), spectre_v2 is mitigated by retpolines (a lot cheaper than IBRS) and the indirect branch predictor barrier happens when switching between different user processes of which one can be a malicious one. And this is the default setting. The whole idea behind having all those cmdline options was for people who don't want to sacrifice performance and would like to disable the security mitigations. So the ultimate decision will be with the user. In any case, the default case does enable a *sensible* set of the mitigations but they are not for free(!) and depend on the workload. I hope I'm making sense here.
(In reply to Borislav Petkov from comment #72) > (In reply to Nicolás Luciano Bértolo from comment #70) > > I can confirm that those parameters help a lot. But in my tests I couldn't > > find a difference between 4.19 and 4.20. 4.20 with mitigations enabled is > > just as bad as 4.19-default. > > Well, the mitigations do cost and we've tried to make them as > unpunishing as possible. For example, my workstation with 20-rc5 has: > > /sys/devices/system/cpu/vulnerabilities/l1tf:Not affected > /sys/devices/system/cpu/vulnerabilities/meltdown:Not affected > /sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Mitigation: > Speculative Store Bypass disabled via prctl and seccomp > /sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: __user > pointer sanitization > /sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full AMD > retpoline, IBPB: conditional, STIBP: disabled, RSB filling > > which means that SSB will be enabled only for applications which request > it (prctl and seccomp), spectre_v2 is mitigated by retpolines (a lot > cheaper than IBRS) and the indirect branch predictor barrier happens > when switching between different user processes of which one can be a > malicious one. > > And this is the default setting. > > The whole idea behind having all those cmdline options was for people > who don't want to sacrifice performance and would like to disable the > security mitigations. So the ultimate decision will be with the user. > > In any case, the default case does enable a *sensible* set of the > mitigations but they are not for free(!) and depend on the workload. > > I hope I'm making sense here. To me you are making sense, thank you for looking into this. I have the same result as https://bugzilla.opensuse.org/show_bug.cgi?id=1112824#c70 However, I also tested kernel-vanilla, from Head and current 4.19.5. Kernel-vanilla gives similar or almost similar result as with mitigations off as you mentioned in https://bugzilla.opensuse.org/show_bug.cgi?id=1112824#c66 But in my understanding kernel-vanilla has all default mitigations on, as running latest spectre-meltdown-checker says my system is not vulnerable. So it seems to me there must be a difference in the kernels. 4.19.5-1-vanilla: /sys/devices/system/cpu/vulnerabilities/l1tf Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable /sys/devices/system/cpu/vulnerabilities/meltdown Mitigation: PTI /sys/devices/system/cpu/vulnerabilities/spec_store_bypass Mitigation: Speculative Store Bypass disabled via prctl and seccomp /sys/devices/system/cpu/vulnerabilities/spectre_v1 Mitigation: __user pointer sanitization /sys/devices/system/cpu/vulnerabilities/spectre_v2 Mitigation: Full generic retpoline, IBPB, IBRS_FW 4.19.5-1-default: /sys/devices/system/cpu/vulnerabilities/l1tf Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable /sys/devices/system/cpu/vulnerabilities/meltdown Mitigation: PTI /sys/devices/system/cpu/vulnerabilities/spec_store_bypass Mitigation: Speculative Store Bypass disabled via prctl and seccomp /sys/devices/system/cpu/vulnerabilities/spectre_v1 Mitigation: __user pointer sanitization /sys/devices/system/cpu/vulnerabilities/spectre_v2 Mitigation: Indirect Branch Restricted Speculation, IBPB, IBRS_FW Apparently on my Intel system both kernels have different spectre_v2 mitigations. Kernel-default is using IBRS, which as you say is more expensive than retpoline, which is used by kernel-vanilla.
I've booted kernel-default with parameters spectre_v2=retpoline,generic Now I have the same performance as kernel-vanilla without parameters. 4.19.5-1-default with spectre_v2=retpoline,generic: /sys/devices/system/cpu/vulnerabilities/spectre_v2 Mitigation: Full generic retpoline, IBPB, IBRS_FW
Created attachment 792011 [details] Logs (In reply to Borislav Petkov from comment #71) > (In reply to Dead Mozay from comment #69) > > I'm talking about it, openSUSE boot with pti=off spectre_v2=off l1tf=off > > nospec_store_bypass_disable no_stf_barrier, Fedora with default settings. In > > Fedora no problem even with enabled hyper-threading. > > Can you upload full dmesg from the Fedora kernel and also do > > $ grep . /sys/devices/system/cpu/vulnerabilities/* > vuln.log > > on it and upload that vuln.log file too? > > Thx. Sure
(In reply to Michiel Janssens from comment #73) > Apparently on my Intel system both kernels have different spectre_v2 > mitigations. > Kernel-default is using IBRS, which as you say is more expensive than > retpoline, which is used by kernel-vanilla. Yes, kernel-default has our SUSE patches which are part of SLE and have this additional IBRS enablement which is not upstream and thus vanilla doesn't have it. IBRS is a heavy hammer and didn't get accepted upstream but we took it. Which is going to be replaced by enhanced IBRS which should be lighter but it is still being rolled out and I don't know whether older, already released machines can even get it through microcode. For details, see: https://software.intel.com/sites/default/files/managed/c5/63/336996-Speculative-Execution-Side-Channel-Mitigations.pdf where all the different mitigation mechanisms are explained. Now, it is debatable whether a Skylake class machine which needs IBRS to be fully mitigated is even exploitable when only retpolines are enabled. It has been said that running a spectre v2 exploit on a machine only with retpolines and not IBRS is very very hard to do. Thus, many people are unwilling to pay the performance penalty and revert to retpolines. IOW, if you boot with spectre_v2=retpoline on kernel-default, you should be getting close to vanilla. All IMHO, of course.
(In reply to Michiel Janssens from comment #74) > Now I have the same performance as kernel-vanilla without parameters. ... and you've confirmed that! :-)
(In reply to Borislav Petkov from comment #76) > (In reply to Michiel Janssens from comment #73) > > > Apparently on my Intel system both kernels have different spectre_v2 > > mitigations. > > Kernel-default is using IBRS, which as you say is more expensive than > > retpoline, which is used by kernel-vanilla. > > Yes, kernel-default has our SUSE patches which are part of SLE and have > this additional IBRS enablement which is not upstream and thus vanilla > doesn't have it. > > IBRS is a heavy hammer and didn't get accepted upstream but we took it. > Which is going to be replaced by enhanced IBRS which should be lighter > but it is still being rolled out and I don't know whether older, already > released machines can even get it through microcode. For details, see: > > https://software.intel.com/sites/default/files/managed/c5/63/336996- > Speculative-Execution-Side-Channel-Mitigations.pdf > > where all the different mitigation mechanisms are explained. > > Now, it is debatable whether a Skylake class machine which needs IBRS to > be fully mitigated is even exploitable when only retpolines are enabled. > It has been said that running a spectre v2 exploit on a machine only > with retpolines and not IBRS is very very hard to do. Thus, many people > are unwilling to pay the performance penalty and revert to retpolines. > IOW, if you boot with spectre_v2=retpoline on kernel-default, you should > be getting close to vanilla. > > All IMHO, of course. I build a kernel 4.19.5 without IBRS patches, nothing has changed, although I do not exclude the possibility that I could do something wrong, or skip. https://build.opensuse.org/package/show/home:Dead_Mozay:Kernel/kernel-default?expand=0
(In reply to Dead Mozay from comment #75) > Created attachment 792011 [details] > Logs Yes, you're comparing apples with oranges: 1. Fedora is 4.18.16-300.fc29.x86_64 which has some patches - who knows what - all the distros do ship their own tweaks ontop. 2. 4.20.0-rc5-1.g2ccaf30-vanilla which is the upstream kernel (perhaps?) and does not have those patches. All this says is that Fedora's kernel is somewhat better, provided the benchmarks are sensible. I have no clue what yours do. If you want to compare security mitigations, you need to take the same kernel and do two runs, once with the mitigation enabled and once with the mitigation disabled.
(In reply to Dead Mozay from comment #78) > I build a kernel 4.19.5 without IBRS patches, nothing has changed, although > I do not exclude the possibility that I could do something wrong, or skip. That's likely. The expectation is that without IBRS you should be seeing a performance uplift.
(In reply to Borislav Petkov from comment #79) > (In reply to Dead Mozay from comment #75) > > Created attachment 792011 [details] > > Logs > > Yes, you're comparing apples with oranges: > > 1. Fedora is 4.18.16-300.fc29.x86_64 which has some patches - who knows > what - all the distros do ship their own tweaks ontop. > > 2. 4.20.0-rc5-1.g2ccaf30-vanilla which is the upstream kernel (perhaps?) > and does not have those patches. > > All this says is that Fedora's kernel is somewhat better, provided the > benchmarks are sensible. I have no clue what yours do. > > If you want to compare security mitigations, you need to take the same > kernel and do two runs, once with the mitigation enabled and once with > the mitigation disabled. with the same kernel, the results are the same, just where it is installed there is no possibility to use other kernels because of the proprietary drivers nvidia. in fedora vanilla kernel, without patches, at least so maintainers maintain I tried to install fedora on this laptop, it works fine even with the kernel 4.19.4 which was at that time
(In reply to Dead Mozay from comment #81) > (In reply to Borislav Petkov from comment #79) > > (In reply to Dead Mozay from comment #75) > > > Created attachment 792011 [details] > > > Logs > > > > Yes, you're comparing apples with oranges: > > > > 1. Fedora is 4.18.16-300.fc29.x86_64 which has some patches - who knows > > what - all the distros do ship their own tweaks ontop. > > > > 2. 4.20.0-rc5-1.g2ccaf30-vanilla which is the upstream kernel (perhaps?) > > and does not have those patches. > > > > All this says is that Fedora's kernel is somewhat better, provided the > > benchmarks are sensible. I have no clue what yours do. > > > > If you want to compare security mitigations, you need to take the same > > kernel and do two runs, once with the mitigation enabled and once with > > the mitigation disabled. > > with the same kernel, the results are the same, just where it is installed > there is no possibility to use other kernels because of the proprietary > drivers nvidia. > in fedora vanilla kernel, without patches, at least so maintainers maintain > I tried to install fedora on this laptop, it works fine even with the kernel > 4.19.4 which was at that time Are you testing Fedora kernel with openSUSE user-space stuff? Or are you testing Fedora user-space? If Fedora kernel works better with openSUSE user-space, then the point should be either Fedora's downstream patch or the difference of the kernel configuration. For the latter case, you can try to build the upstream kernel with Fedora kernel config and see whether it works.
> If you want to compare security mitigations, you need to take the same kernel > and do two runs, once with the mitigation enabled and once with the mitigation > disabled. I performed tests with enabled and disabled parameters, the result in fedora is practically the same, in openSUSE the result differs, but it was at the time of the release of Leap 15, you need to make new tests, but I can only on the current kernels of both systems
(In reply to Takashi Iwai from comment #82) > (In reply to Dead Mozay from comment #81) > > (In reply to Borislav Petkov from comment #79) > > > (In reply to Dead Mozay from comment #75) > > > > Created attachment 792011 [details] > > > > Logs > > > > > > Yes, you're comparing apples with oranges: > > > > > > 1. Fedora is 4.18.16-300.fc29.x86_64 which has some patches - who knows > > > what - all the distros do ship their own tweaks ontop. > > > > > > 2. 4.20.0-rc5-1.g2ccaf30-vanilla which is the upstream kernel (perhaps?) > > > and does not have those patches. > > > > > > All this says is that Fedora's kernel is somewhat better, provided the > > > benchmarks are sensible. I have no clue what yours do. > > > > > > If you want to compare security mitigations, you need to take the same > > > kernel and do two runs, once with the mitigation enabled and once with > > > the mitigation disabled. > > > > with the same kernel, the results are the same, just where it is installed > > there is no possibility to use other kernels because of the proprietary > > drivers nvidia. > > in fedora vanilla kernel, without patches, at least so maintainers maintain > > I tried to install fedora on this laptop, it works fine even with the kernel > > 4.19.4 which was at that time > > Are you testing Fedora kernel with openSUSE user-space stuff? Or are you > testing Fedora user-space? > > If Fedora kernel works better with openSUSE user-space, then the point > should be either Fedora's downstream patch or the difference of the kernel > configuration. > > For the latter case, you can try to build the upstream kernel with Fedora > kernel config and see whether it works. I tried to build the kernel with the fedora config, I even wrote about it somewhere above, it works better, but there are still some problems,
(In reply to Dead Mozay from comment #84) > > Are you testing Fedora kernel with openSUSE user-space stuff? Or are you > > testing Fedora user-space? > > > > If Fedora kernel works better with openSUSE user-space, then the point > > should be either Fedora's downstream patch or the difference of the kernel > > configuration. > > > > For the latter case, you can try to build the upstream kernel with Fedora > > kernel config and see whether it works. > > I tried to build the kernel with the fedora config, I even wrote about it > somewhere above, it works better, but there are still some problems, If you still have the same performance problem with a self-built upstream kernel using the Fedora config, then it's really something else than kernel. If something is *improved* by the Fedora kernel config, though, we'd like to see the difference. You can try the following: - Boot with Fedora config kernel, copy /proc/config.gz to .config on 4.19.x Linux kernel tree, and run "make localmodconfig". This will give you a minimal set of kernel config for the currently running kernel. Save this config to somewhere. - Similarly, boot with openSUSE kernel, and do the same. Now you get two kernel configs you can directly compare.
(In reply to Takashi Iwai from comment #85) > (In reply to Dead Mozay from comment #84) > > > Are you testing Fedora kernel with openSUSE user-space stuff? Or are you > > > testing Fedora user-space? > > > > > > If Fedora kernel works better with openSUSE user-space, then the point > > > should be either Fedora's downstream patch or the difference of the kernel > > > configuration. > > > > > > For the latter case, you can try to build the upstream kernel with Fedora > > > kernel config and see whether it works. > > > > I tried to build the kernel with the fedora config, I even wrote about it > > somewhere above, it works better, but there are still some problems, > > If you still have the same performance problem with a self-built upstream > kernel using the Fedora config, then it's really something else than kernel. > > If something is *improved* by the Fedora kernel config, though, we'd like to > see the difference. You can try the following: > > - Boot with Fedora config kernel, copy /proc/config.gz to .config on 4.19.x > Linux kernel tree, and run "make localmodconfig". This will give you a > minimal set of kernel config for the currently running kernel. Save this > config to somewhere. > > - Similarly, boot with openSUSE kernel, and do the same. Now you get two > kernel configs you can directly compare. https://bugzilla.opensuse.org/show_bug.cgi?id=1112824#c30 this is fedora config, I only enabled support BTRFS
That config contains all stuff, so it's not clear which different matters. That's why I asked to create minimal configs so that we can compare 1:1.
(In reply to Takashi Iwai from comment #87) > That config contains all stuff, so it's not clear which different matters. > That's why I asked to create minimal configs so that we can compare 1:1. unfortunately in Fedora CONFIG_IKCONFIG_PROC=n
Created attachment 792088 [details] New tests performed the same tests, and so about since the release openSUSE Leap 15.0, only in Leap does disabling SMT in BIOS solve the problem
and if I get the config via makemenu and then run make localmodconfig?
(In reply to Dead Mozay from comment #56) > In general, everything that I tried, does not help, rebuild the kernel > without patches, compile the kernel from appstrim, disable spectre, rebuild > gnome-shell, gjs, xorg, mesa, mutter, try kernel-vanilla, it all partially > solved the problem, but still not as it should be. The best option is > disable Hyper threading, only on my laptop I have no such opportunity, on my > PC it is possible,. installed openSUSE Leap, with enabled Hyper threading > the problem is there (GNOME 3.26). Installed Fedora, GNOME works perfectly, > despite being turned Hyper threading or not. I have no more ideas. You can disable smt via kernel parameters. see: https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html
(In reply to Michiel Janssens from comment #91) > (In reply to Dead Mozay from comment #56) > > In general, everything that I tried, does not help, rebuild the kernel > > without patches, compile the kernel from appstrim, disable spectre, rebuild > > gnome-shell, gjs, xorg, mesa, mutter, try kernel-vanilla, it all partially > > solved the problem, but still not as it should be. The best option is > > disable Hyper threading, only on my laptop I have no such opportunity, on my > > PC it is possible,. installed openSUSE Leap, with enabled Hyper threading > > the problem is there (GNOME 3.26). Installed Fedora, GNOME works perfectly, > > despite being turned Hyper threading or not. I have no more ideas. > > You can disable smt via kernel parameters. > see: > https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html > https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html Disable can be, but that it should not be, is not the solution
(In reply to Dead Mozay from comment #92) > Disable can be, but that it should not be, is not the solution For testing and comparison purpose.
(In reply to Michiel Janssens from comment #93) > (In reply to Dead Mozay from comment #92) > > Disable can be, but that it should not be, is not the solution > > For testing and comparison purpose. for testing, need leap, I have TW, then it no longer helps
yet with the option nosmt=force got better at openSUSE Tumbleweed
(In reply to Dead Mozay from comment #88) > (In reply to Takashi Iwai from comment #87) > > That config contains all stuff, so it's not clear which different matters. > > That's why I asked to create minimal configs so that we can compare 1:1. > > unfortunately in Fedora CONFIG_IKCONFIG_PROC=n Then copy Fedora kernel config manually to .config on kernel source directory, then run "make localmodconfig". It's only about copying from the proper source.
Created attachment 792119 [details] Configs (In reply to Takashi Iwai from comment #96) > (In reply to Dead Mozay from comment #88) > > (In reply to Takashi Iwai from comment #87) > > > That config contains all stuff, so it's not clear which different matters. > > > That's why I asked to create minimal configs so that we can compare 1:1. > > > > unfortunately in Fedora CONFIG_IKCONFIG_PROC=n > > Then copy Fedora kernel config manually to .config on kernel source > directory, then run "make localmodconfig". It's only about copying from the > proper source. Made. Maybe I didn’t do that, it issued 2 requests, I answered both NEW
Any updates on this? I thought I was going crazy because my fresh install was sluggish as hell. Also running TW on a T480 with an Intel i5 8250u.
(In reply to Dennis Irrgang from comment #98) > Any updates on this? I thought I was going crazy because my fresh install > was sluggish as hell. > > Also running TW on a T480 with an Intel i5 8250u. not yet, so nobody understood what the problem
(In reply to Dennis Irrgang from comment #98) > Any updates on this? I thought I was going crazy because my fresh install > was sluggish as hell. > > Also running TW on a T480 with an Intel i5 8250u. Please explain in a detailed, step-by-step manner what you're doing so that I can try to reproduce it here. What machine, what image, what kernel, etc. Everything that you think is relevant for someone else to know in order to be able to reproduce the problem - whatever "problem" that is. Thx.
FWIW, I can compare with Ubuntu on similar hardware (Kaby Lake R quad core, UHD 620), and even though the kernel parameters mentioned here - or using balance_performance for energy_performance_preference - make it smoother on TW, it's still slow compared. As a subjective measurement in FullHD resolution the super key animation to Activities should be butter smooth with no hickup or steps in either zoom out or zoom in with eg 3 windows. Currently the performance for me is about the same with FullHD on TW with all the kernel tweaks as it is with 4K resolution on another distro without the kernel tweaks. In my case however the comparison in case was against a patched GNOME 3.28.3, so it's not apples to apples. Both testbeds here are using X, not Wayland which seems to perform worse on both. So even though eg the kernel parameters seem to help to an extent, it does not seem it could be said "it's kernel", as also indicated by the fact how GNOME on default Fedora kernel performs just fine too. At the moment unfortunately I don't have anything more to give, I hope to find the culprit together though. You may want to set balance_performance instead of balance_power which was not mentioned here before, but that also is not the silver bullet in any sense. At one point that was critically needed for gnome-shell's optimal user experience but I believe that was fixed in 3.30.
Quickly looking at the logs of comment 97, the main differences between two kernel configs are: CONFIG_NO_HZ_FULL=y for Fedora CONFIG_NO_HZ_IDLE=y for TW CONFIG_PREEMPT_VOLUNTARY=y for Fedora CONFIG_PREEMPT=y for TW CONFIG_SCHED_AUTOGROUP=y for Fedora CONFIG_SCHED_AUTOGROUP=n for TW CONFIG_SLUB=y for Fedora CONFIG_SLAB=y for TW The last one is unlikely influencing on the performance, though. The rest are subtle drivers and debugging configs. So I'm building a TW kernel with the config changes above to follow Fedora in OBS home:tiwai:bsc1112824 repo. The build should finish in an hour or so. Please give it a try and see whether if this really improves things as advertised here.
(In reply to Borislav Petkov from comment #100) > (In reply to Dennis Irrgang from comment #98) > > Any updates on this? I thought I was going crazy because my fresh install > > was sluggish as hell. > > > > Also running TW on a T480 with an Intel i5 8250u. > > Please explain in a detailed, step-by-step manner what you're doing so that > I can try to reproduce it here. What machine, what image, what kernel, etc. > Everything that you think is relevant for someone else to know in order to > be able to reproduce the problem - whatever "problem" that is. > > Thx. The problem appears to be exactly the one talked about in this bug report. Abysmal performance of transition animations in GNOME. I'm happy to help in any way I can, though I'm not sure if I can bring anything new to the table. As I said I'm running the most recent snapshot of TW using GNOME on a Lenovo T480 with the Intel i7 8250U CPU and the integrated Intel UHD graphics card. Due to a similar bug in Wayland (Wayland freezes every so often, causing runaway stuck-keys) I had switched back to X11, though now I'm experiencing the same bug of this ticket instead.
(In reply to Dennis Irrgang from comment #103) > The problem appears to be exactly the one talked about in this bug report. > Abysmal performance of transition animations in GNOME. I'm happy to help in > any way I can, though I'm not sure if I can bring anything new to the table. You can test the latest upstream kernel from here: http://kernel.suse.com/packages/master and see if that gives some of your performance back. STIBP et al has been reworked upstream and our kernels will get it soon. Thx.
(In reply to Takashi Iwai from comment #102) > Quickly looking at the logs of comment 97, the main differences between two > kernel configs are: > > CONFIG_NO_HZ_FULL=y for Fedora > CONFIG_NO_HZ_IDLE=y for TW > > CONFIG_PREEMPT_VOLUNTARY=y for Fedora > CONFIG_PREEMPT=y for TW > > CONFIG_SCHED_AUTOGROUP=y for Fedora > CONFIG_SCHED_AUTOGROUP=n for TW > > CONFIG_SLUB=y for Fedora > CONFIG_SLAB=y for TW > > The last one is unlikely influencing on the performance, though. > The rest are subtle drivers and debugging configs. > > So I'm building a TW kernel with the config changes above to follow Fedora > in OBS home:tiwai:bsc1112824 repo. The build should finish in an hour or so. > > Please give it a try and see whether if this really improves things as > advertised here. I feel the gnome has become a bit more responsive, but I tried the free Nvidia drivers, it could also be optimization nouveau driver in the kernel, This should be checked on the Intel graphics, unfortunately I have no such opportunity now
(In reply to Takashi Iwai from comment #102) > So I'm building a TW kernel with the config changes above to follow Fedora > in OBS home:tiwai:bsc1112824 repo. The build should finish in an hour or so. > > Please give it a try and see whether if this really improves things as > advertised here. I don't see difference on my Intel machine with the 4.19.8-1.g8bc7ab5-default nosmt=force ie disabling hyperthreading is something that does seem to make a real tangible difference for me. With it the most annoying lags are gone. The effect is the same both with 4.19.7 and the one from home:tiwai:bsc1112824, so in other words the changed configuration there does not seem to affect this behavior. This is based on 10+ boots.
(In reply to Timo Jyrinki from comment #106) > (In reply to Takashi Iwai from comment #102) > > So I'm building a TW kernel with the config changes above to follow Fedora > > in OBS home:tiwai:bsc1112824 repo. The build should finish in an hour or so. > > > > Please give it a try and see whether if this really improves things as > > advertised here. > > I don't see difference on my Intel machine with the 4.19.8-1.g8bc7ab5-default > > nosmt=force ie disabling hyperthreading is something that does seem to make > a real tangible difference for me. With it the most annoying lags are gone. > The effect is the same both with 4.19.7 and the one from > home:tiwai:bsc1112824, so in other words the changed configuration there > does not seem to affect this behavior. This is based on 10+ boots. Just to be sure: did you already test the kernel in OBS Kernel:HEAD, which is based on the latest 4.20-rc?
(In reply to Takashi Iwai from comment #107) > Just to be sure: did you already test the kernel in OBS Kernel:HEAD, which > is based on the latest 4.20-rc? Now I have (4.20.0-rc6-2.g91eea17-default). It seems the situation remains the same even there. With hyperthreading enabled overall user experience is bad due to lagginess, with nosmt=force it's usable.
(In reply to Timo Jyrinki from comment #108) > Now I have (4.20.0-rc6-2.g91eea17-default). It seems the situation remains > the same even there. With hyperthreading enabled overall user experience is > bad due to lagginess, with nosmt=force it's usable. Does booting with l1tf=off help? Does booting with spectre_v2=retpoline help? Can you try a different desktop environment besides gnome and see how it behaves there? Thx.
(In reply to Borislav Petkov from comment #109) > (In reply to Timo Jyrinki from comment #108) > > Now I have (4.20.0-rc6-2.g91eea17-default). It seems the situation remains > > the same even there. With hyperthreading enabled overall user experience is > > bad due to lagginess, with nosmt=force it's usable. > > Does booting with l1tf=off help? > > Does booting with spectre_v2=retpoline help? > > Can you try a different desktop environment besides gnome and see how it > behaves there? > > Thx. disabling patches on the 4.19.7 kernel no longer affects anything, just made several reboots with different patch shutdown options, and without shutting down at all, the result is the same. At the moment, my performance is only affected by disabling hyper-threading.
> disabling patches on the 4.19.7 kernel no longer affects anything, just made > several reboots with different patch shutdown options, and without shutting > down at all, the result is the same. At the moment, my performance is only > affected by disabling hyper-threading. This does not match my experience. Using spectre_v2=retpoline makes the default kernel perform just as well as the vanilla kernel. I'm using kernel 4.19.7-1-default.
(In reply to Nicolás Luciano Bértolo from comment #111) > > disabling patches on the 4.19.7 kernel no longer affects anything, just made > > several reboots with different patch shutdown options, and without shutting > > down at all, the result is the same. At the moment, my performance is only > > affected by disabling hyper-threading. > > This does not match my experience. Using spectre_v2=retpoline makes the > default kernel perform just as well as the vanilla kernel. > > I'm using kernel 4.19.7-1-default. Perhaps spectre_v2 = retpoline affects the work with Intel HD graphics, i have Nvidia with proprietary drivers, for me this option is useless.
(In reply to Nicolás Luciano Bértolo from comment #111) > > disabling patches on the 4.19.7 kernel no longer affects anything, just made > > several reboots with different patch shutdown options, and without shutting > > down at all, the result is the same. At the moment, my performance is only > > affected by disabling hyper-threading. > > This does not match my experience. Using spectre_v2=retpoline makes the > default kernel perform just as well as the vanilla kernel. > > I'm using kernel 4.19.7-1-default. Checked on my laptop http://paste.opensuse.org/view//92547497, the difference really is with the parameter spectre_v2=retpoline, but still this is not enough, a conflict of patches may have occurred somewhere, I saw a package with a bunch of patches somewhere, and a few were very old, 6 years ago were created, most likely they have not been needed for a long time, and now such patches can cause only problems
Probably related to this bug. I noticed that when simply moving a window around gnoem-shell uses 100% of its current core. Interestingly the same happens simply while typing in Telegram. I noticed the latter because every time I typed, a video running in a separate window would begin to stutter. This is while using the X-Server.
There may also be a time component to this. It seems like performance is degrading over time. After a reboot performance is 'usable', but after a few hours of working it becomes notably worse.
(In reply to Dennis Irrgang from comment #115) > There may also be a time component to this. It seems like performance is > degrading over time. After a reboot performance is 'usable', but after a few > hours of working it becomes notably worse. At the expense of overall performance, I'm not sure, but the GNOME starts to fail more, if the computer is running for a long time without rebooting. What I had enough of my mind, I already checked, it remains to hope for real gurus. but in general, similar problems with animations appeared in Fedora, but in openSUSE, they are more noticeable and annoying, for many, but I do not exclude the possibility that the lags are due to the fact that I have Fedora installed on the old HDD, and openSUSE on NVME SSD, but problems are still more on openSUSE.
In the meantime, http://kernel.suse.com/packages/stable has 4.20.2 which should have all the latest mitigations and improvements so if people wanna test...
I'm testing 4.20 kernel now. For me nothing has changed. Still animations are choppy, windows are opening delayed. My specs: Intel i7-7500U, Intel HD graphics 620, 8GB ram.
(In reply to Adam Szyszko from comment #118) > I'm testing 4.20 kernel now. For me nothing has changed. Still animations > are choppy, windows are opening delayed. > My specs: Intel i7-7500U, Intel HD graphics 620, 8GB ram. similarly, I did not notice any changes except one, all options for disabling protection spectre V2 can be removed. Performance no longer suffers when security is enabled, but with the gnome this problem did not solve.
And now can anyone try with a different desktop environment besides gnome?
(In reply to Borislav Petkov from comment #120) > And now can anyone try with a different desktop environment besides gnome? some people say that everything is ok there, perhaps because they have nothing to compare with, but when I checked the bug I checked with plasma5 I also noticed some problems in animations there, but I’ll not say that the root of the problems in the gnome will be, in plasma5, through the release, they break something, into this the moment there is something broken too, when choosing the engine, the opengl3 animations work poorly, changing the version to the old one helps a little, but problems remain anyway, this problem is in plasma5, at the moment it is not possible to conduct such a check, you need to wait when the problem is corrected in plasma5. And there are no more sane desktop environments where you can see how graphics acceleration works and other things.
Ok, bouncing back for reassignment. If there's anything the kernel can do, I'm all ears.
(In reply to Borislav Petkov from comment #122) > Ok, bouncing back for reassignment. If there's anything the kernel can do, > I'm all ears. at the moment I see only one problem related to the kernel, the SMT enabled in the BIOS still spoils the performance, again this is most noticeable in the gnome, otherwise it seems to me that it is indistinguishable from the vanilla kernel, I don’t observe any early performance losses, the rest apparently need to look elsewhere. It is necessary to do an audit of all system packages, as I said earlier there are packages with very old patches, it is possible that somewhere such a patch caused some kind of conflict.
Are you using GNOME on a wayland session or an XOrg session?
(In reply to Atri Bhattacharya from comment #124) > Are you using GNOME on a wayland session or an XOrg session? XOrg on desktop, wayland on laptop
I've lost track about this issue during the last weeks but I'm still facing Gnome performance which is really hard to work with on my laptop also with latest TW snapshot. I'm supposed to run Gnome under Xorg. At least that is what my session selector says but on the other hand when I look at the process list I see Xwayland running and not Xorg. (Looks like a bug to me) My system is a Core-i7-7500 (Quad) but the Gnome shell performance is really hard to work with. All window/desktop actions (moving windows, switches workspaces etc) are quite laggy and not smooth at all. This started in October and only got better a bit. Compared to before it's really slow. Considering switching to Xfce or something for the hope it'll be better.
(In reply to Wolfgang Rosenauer from comment #126) > I've lost track about this issue during the last weeks but I'm still facing > Gnome performance which is really hard to work with on my laptop also with > latest TW snapshot. > > I'm supposed to run Gnome under Xorg. At least that is what my session > selector says but on the other hand when I look at the process list I see > Xwayland running and not Xorg. (Looks like a bug to me) > > My system is a Core-i7-7500 (Quad) but the Gnome shell performance is really > hard to work with. All window/desktop actions (moving windows, switches > workspaces etc) are quite laggy and not smooth at all. This started in > October and only got better a bit. Compared to before it's really slow. > Considering switching to Xfce or something for the hope it'll be better. echo $XDG_SESSION_TYPE will show what session you have X11 or Wayland. To activate the X11 session, you need to edit the file /etc/gdm/custom.conf, remove comment from the line WaylandEnable=false and restart session.
XDG_SESSION_TYPE is "x11" wolfgang@ox:~> ps ax | grep X 1834 tty7 Sl+ 0:00 /usr/bin/Xwayland :1024 -rootless -terminate -accessx -core -auth /run/user/464/mutter/Xauthority -listen 4 -listen 5 -displayfd 6 3393 tty2 Sl+ 0:02 /usr/bin/X vt2 -displayfd 3 -auth /run/user/1000/gdm/Xauthority -background none -noreset -keeptty -verbose 3 wolfgang@ox:~> grep -i wayland /etc/gdm/custom.conf #WaylandEnable=false What is the verdict for that output? Wayland or Xorg? I guess "wayland" but XDG_SESSION_TYPE does not reflect that.
(In reply to Wolfgang Rosenauer from comment #128) > XDG_SESSION_TYPE is "x11" > wolfgang@ox:~> ps ax | grep X > 1834 tty7 Sl+ 0:00 /usr/bin/Xwayland :1024 -rootless -terminate > -accessx -core -auth /run/user/464/mutter/Xauthority -listen 4 -listen 5 > -displayfd 6 > 3393 tty2 Sl+ 0:02 /usr/bin/X vt2 -displayfd 3 -auth > /run/user/1000/gdm/Xauthority -background none -noreset -keeptty -verbose 3 > wolfgang@ox:~> grep -i wayland /etc/gdm/custom.conf > #WaylandEnable=false > > What is the verdict for that output? > Wayland or Xorg? I guess "wayland" but XDG_SESSION_TYPE does not reflect > that. you need to edit the file /etc/gdm/custom.conf, remove comment from the line WaylandEnable=false and restart session.
Did that now. Now there is no Xwayland running anymore. And the desktop animations are now certainly smoother than with wayland running but but also still stutter more than expected. gnome-shell CPU consumption still very visible.
(In reply to Wolfgang Rosenauer from comment #130) > Did that now. Now there is no Xwayland running anymore. > And the desktop animations are now certainly smoother than with wayland > running but but also still stutter more than expected. > gnome-shell CPU consumption still very visible. the consumption of CPU resources for the gnome is normal, this is not the reason
I am on latest Tumbleweed with a T460s - i7 6600 The only thing that works for me to resolve the performance issues is to disable hyperthreading by adding nosmt=force boot parameter for now.
tried nosmt=force now as well. It really feels slightly better. I wouldn't say I'm back to the performance from mid last year but it comes close. So for me force-disabling wayland and nosmt=force helped quite a lot.
I'm really confused why were are assuming this is a kernel issue? What is GNOME doing with the kernel that could cause this? I just encountered two other users in a Discord server that I frequent that have this same exact issue (GNOME is extremely sluggish and they have Intel CPUs), but only with GNOME, and only on Tumbleweed. All other DEs work fine. All other distros with GNOME 3.30 work fine. There's gotta be some patch we apply to GNOME or *something* along those lines causing this... it seems extremely unlikely that it would be the kernel as these users have no issue using KDE Plasma or Xfce on the latest Tumbleweed snapshot (20190126). GNOME runs much slower for both of them on Tumbleweed than on Arch or Fedora, but only GNOME. As I said above, what would could be going with the kernel that would *only* affect GNOME?
(In reply to simon izor from comment #134) > I'm really confused why were are assuming this is a kernel issue? What is > GNOME doing with the kernel that could cause this? > > I just encountered two other users in a Discord server that I frequent that > have this same exact issue (GNOME is extremely sluggish and they have Intel > CPUs), but only with GNOME, and only on Tumbleweed. All other DEs work > fine. All other distros with GNOME 3.30 work fine. > > There's gotta be some patch we apply to GNOME or *something* along those > lines causing this... it seems extremely unlikely that it would be the > kernel as these users have no issue using KDE Plasma or Xfce on the latest > Tumbleweed snapshot (20190126). GNOME runs much slower for both of them on > Tumbleweed than on Arch or Fedora, but only GNOME. As I said above, what > would could be going with the kernel that would *only* affect GNOME? Everything is very difficult, as I wrote above, the gnome is very dependent on how well the processor works in the operating system, any error in the kernel can affect its operation, which in turn can lead to different regressions, so the possibility of such exclusion was it is impossible, for example, by disabling the spectre protection, it started to work better, patches were applied to the kernel, not the gnome, there were such conclusions from here, currently disabling patches does not change anything at all, and he again started to lag after the latest updates. > GPU <- [ [ Cogl + Cairo ] <- [ GDK + Clutter ] <- GTK+ ] <- application So GNOM works, need to check these packages, I don’t have enough skills to check it I tried plasma 5, it also lags me, not like a gnome, but it’s still noticeable
I personally am using an Intel i7 6700T, and I have not noticed any slowdowns in my use of Tumbleweed. I am using Lumina as my DE (not in the official repositories), but if this were a kernel problem, I would think it would happen pretty much anywhere. I've got 4 Electron apps, Falkon, mpv playing a Twitch stream, a terminal, and some stuff sitting in my tray, but Tumbleweed feels just as fast to me as it has since the mitigations for spectre and such were implemented. I did notice *some* slowdown when that happened, but nothing major... mostly just with package management. I don't use GNOME myself, so I cannot comment on how well that runs, but the other users who have experienced GNOME being sluggish on Tumbleweed have had zero complaints about other DEs not being up to par. One of them just recently switched from Arch. I would think sluggish performance would have been something they noticed right away if it affected all DEs and not just when they decided to install GNOME to try it out after the other user was complaing about how slow it was.
Simon, I agree with you it's not just a kernel issue. And Dead Mozay said it's quite complicated. I've tried to analyze, but not yet get a full view. But basically speaking - there are a lot of issues inside GNOME (gjs, gnome-shell and probably mutter are affected), but these issues become more visible with kernel issue. spectre_v2=retpoline boot option really helps, but some lags still there. Finally I gave up with tumbleweed, there are too many packages updated the same time and it makes it harder to find out the exact root cause, especially because as I said I'm sure it's not only one issue. I just finished building some packages related to GNOME with the fixes I think should help, but I did it for Leap 15 and still testing it. Leap 15 has different issues in GNOME, but it seems easier to fix. For example one of the patch fixing tweener.js errors in journal, not just suppressing the messages as it was done with gdm options in some builds, but really fix the behavior of using objects. I'm still testing on my system, but if somebody wants to check it too, I'll be happy. But please note, the fixes is for Leap 15.0 only.
It seems to me that I have good news, maybe I found and fixed the problem, now I sent the correction to my repository, as soon as I’m ready I will provide a link for those who want to test
It seems it was possible to partially solve the problem, it does not fix everything, but the GNOME has become more responsive. > sudo zypper addrepo https://download.opensuse.org/repositories/home:Dead_Mozay:GNOME/openSUSE_Tumbleweed/home:Dead_Mozay:GNOME.repo > sudo zypper refresh > sudo zypper install -f libmozjs-60 > reboot Who will check, please write
still have bugs in glib2, in functions > g_main_context_check > g_main_context_prepare ibus > g_source_iter_next libicu-suse63_1 and some other components, in glib2 there were patches that caused some problems, I rebuilt glib2 without these patches, some of the problems were gone, Glib2 is most affected, need to fix the rest, but then I don’t know how. glib2 is partially fixed in my repository, you need to update all glib2 packages from it, the result is a little better
Created attachment 795858 [details] Without my fixes It seems this will not end, corrected one, got out another, need to rebuild all the components of GNOME, and not only GNOME, there is a whole lot, including some of the kernel. If anyone has a desire, you can start sysperf yourself, perform all operations where there are lags, and see the result
After my own testing, I've proposed changing to a kernel that uses VOLUNTARY preemption https://lists.opensuse.org/opensuse-kernel/2019-02/msg00005.html Anyone on Tumbleweed wishing to see if they can reproduce my findings can give the kernel I tried a go at https://download.opensuse.org/repositories/home:/favogt:/twvolun/standard/x86_64/kernel-default-4.20.6-1.1.g463cfd2.x86_64.rpm
Hi Richard. It seems nothing has changed with your kernel. I found that it's quite easy to check if the issue presents on the system. Just to use geekbench. The single core score is very dependent on spectre_v2=retpoline option. Here is the result with spectre_v2=retpoline: https://browser.geekbench.com/v4/cpu/11952293 And here is the result with default mitigation for my CPU (IBRS): https://browser.geekbench.com/v4/cpu/11952353 3981 vs 3140 in single core score. Multicore difference is insignificant. The behavior with micro lags in GNOME is the same with the default kernel from Tumbleweed and from your repo. Microlags are present with IBRS only. On my other system Lenovo T470p i5-7440HQ there is no difference in geekbench score with IBRS or retpoline options.
Dead Mozay: I've tried the gnome packages from your repo. I didn't find any difference in performance with the default packages. Microlags are still there with IBRS spectre_v2 mitigation.
Obviously there is no silver bullet. This is no single bug, but attributed by different factors. We couldn't address all of them so far, and it's enough to cause the significant problem... For TW, I guess the majority of regressions would be fixed by CONFIG_VOLUNTARY_PREEMPTY and the user-space side fixes. But there must be many remaining machines that still suffer from the performance issue, unfortunately.
(In reply to Richard Brown from comment #143) > After my own testing, I've proposed changing to a kernel that uses VOLUNTARY > preemption > > https://lists.opensuse.org/opensuse-kernel/2019-02/msg00005.html > > Anyone on Tumbleweed wishing to see if they can reproduce my findings can > give the kernel I tried a go at > https://download.opensuse.org/repositories/home:/favogt:/twvolun/standard/ > x86_64/kernel-default-4.20.6-1.1.g463cfd2.x86_64.rpm I tried this kernel on my laptop and ran a test in sysprof this is the initial data https://cloud.rfo13.ru/index.php/s/sQpk72nRXSESYpL This is after the kernel update. https://cloud.rfo13.ru/index.php/s/gCcaAHYNDopAdMd and this is after the update libmozjs-60 from my repository https://cloud.rfo13.ru/index.php/s/b8XDLsS3mAfFXrJ there is definitely progress, but visually almost nothing changes. I also stumbled upon such tests https://www.phoronix.com/scan.php?page=article&item=dellxps-9380-linux&num=1 Perhaps the author of the article is right, and the problem lies in energy consumption.
Thanks everyone for your continued investigations. Let's not despair, the solutions are there somewhere :) (In reply to Dead Mozay from comment #151) > there is definitely progress, but visually almost nothing changes. In your going through GNOME stack patches, did you find anything suspicious we're carrying that might be causing performance drops and should be dropped if not upstreamed? And dropping non-upstreamable patches is a good idea overall. I wouldn't (necessarily, unless having fun of course) spend time on doing things other distributions aren't doing, but finding the differences. > I also stumbled upon such tests > https://www.phoronix.com/scan.php?page=article&item=dellxps-9380-linux&num=1 > Perhaps the author of the article is right, and the problem lies in energy > consumption. Surely that's yet another part of the equation, but I've also tweaked my performance policies to be balance_performance which matches other distributions. And I've disabled hyperthreading, and using spectre_v2=retpoline. All of that makes things better, but not comparable to other distros. In addition to /sys/devices/system/cpu/*/*/energy_performance_preference (permanent via /etc/default/tlp) it'd be welcome to know if there are other knobs to turn that could be affecting what Phoronix found, but previously I haven't bumped to others that would be as important. I'd also be interested in trying out the voluntary preemption, to not rule out that and since that's clearly something that is not as easy to test (no runtime switch). Optimizing for smooth desktop usability seems to be really complex these days, and it seems that differing from what others do always has a negative impact due to that complexity. That's a bit sad state of affairs, and lack of performance work resources put to the desktop, but what can we do. It's nice though that Phoronix confirms that we're not all crazy. Sometimes user perceptions are biased or wrong, especially when it comes to "did it help?" if it's not a huge difference.
(In reply to Timo Jyrinki from comment #152) > Thanks everyone for your continued investigations. Let's not despair, the > solutions are there somewhere :) > > (In reply to Dead Mozay from comment #151) > > there is definitely progress, but visually almost nothing changes. > > In your going through GNOME stack patches, did you find anything suspicious > we're carrying that might be causing performance drops and should be dropped > if not upstreamed? And dropping non-upstreamable patches is a good idea > overall. Unfortunately, I did not find anything concrete, except for the regressions in mozjs60, which are corrected by a regular update to a more recent version of this library, I also noticed a strange sqlite behavior, but after a while it disappeared, apparently then some background operations went, and the tracker-miners also behave the same way, after turning off the PC, it began to work quietly, when turned on, I constantly howled even in idle time. but all this did not lead to the desired result, Is that update mozjs fixed some animations > I wouldn't (necessarily, unless having fun of course) spend time on doing > things other distributions aren't doing, but finding the differences. > > Surely that's yet another part of the equation, but I've also tweaked my > performance policies to be balance_performance which matches other > distributions. And I've disabled hyperthreading, and using > spectre_v2=retpoline. All of that makes things better, but not comparable to > other distros. For my PC and laptop, only the disabling of hyper threading is relevant now, the rest is no longer affected > It's nice though that Phoronix confirms that we're not all crazy. I agree, when I started saying almost a year ago that there was a problem in GNOME, no one believed me, everyone claimed that everything was fine, then I decided that the problem could be in my hardware, but later at the presentation on the release day Leap 15 showed a laptop with installed Leap 15, and there it was already clear that lags are present, these lags were caused by the enabled SMT, then everything started with them, the performance drop was already noticeable in GNOME 3.28, but there is also an interesting point, GNOME 3.26 and 3.28 are susceptible to memory leakage but openSUSE is the only Linux distribution the tributary was at that time where there were no memory leaks in GNOME 3.28, but there was a performance drop, I then considered that a patch was corrected for leaks, but slightly degrading performance, I hoped that with the next release it would be better, but as it turned out worse
One thing that is different in TW compared to Fedora, Ubuntu, etc. is btrfs as root filesystem. It is known that gnome-shell likes to perform io operations in the rendering thread, so btrfs may be too slow for that? This is just speculation. I wonder if anyone that is using ext4 as root fs can repro this behaviour.
(In reply to Nicolás Luciano Bértolo from comment #154) > One thing that is different in TW compared to Fedora, Ubuntu, etc. is btrfs > as root filesystem. > > It is known that gnome-shell likes to perform io operations in the rendering > thread, so btrfs may be too slow for that? This is just speculation. > > I wonder if anyone that is using ext4 as root fs can repro this behaviour. The file system does not play a role, it was checked and there ext4
I have ext4 on all af my systems. This issue doesn't depend on file system at all. I have my solution for Leap 15.0 I'll be happy if it could help for somebody other than just me: https://download.opensuse.org/repositories/home:/vzhestkov:/leap15-gnome-update/openSUSE_Leap_15.0/ Just add the repo and make dist upgrade.
Invited to join as a non impacted users :-) I share some informations : I'm using plasma5 with xorg and proprio nvidia drivers I didn't find or perceive any slowdown on my desktop in the last months. I've tested geekbench and the result are there with a plain normal TW (up to date) https://browser.geekbench.com/v4/cpu/12014162 What I always apply is patch all hardware firmware for all component on my hardware some have direct relation with spectre/meldown and as my cpu is a xeon it is also preboot patched with kernel-firmware when the lastest Dell bios doesn't contain all fixes.
(In reply to Bruno Friedmann from comment #157) > Invited to join as a non impacted users :-) > > I share some informations : > I'm using plasma5 with xorg and proprio nvidia drivers > I didn't find or perceive any slowdown on my desktop in the last months. > > I've tested geekbench and the result are there with a plain normal TW (up to > date) > https://browser.geekbench.com/v4/cpu/12014162 > > What I always apply is patch all hardware firmware for all component on my > hardware some have direct relation with spectre/meldown and as my cpu is a > xeon it is also preboot patched with kernel-firmware when the lastest Dell > bios doesn't contain all fixes. This is true, there are no such problems in plasma5, it worked for almost 2 days, everything was fine. but it's still not clear what causes problems in GNOME
I made an experiment to see if the reason for bad performance in TW is the kernel. I booted a Fedora-openSUSE hybrid, kernel kernel-5.0.0-0.rc4.git3.1.fc30 with the latest TW snapshot but I couldn't see a significant improvement in performance. I used this commandline: splash=silent pcie_aspm=force scsi_mod.use_blk_mq=1 elevator=mq-deadline spec_store_bypass_disable=off spectre_v2=retpoline l1tf=off spectre_v2_user=off workqueue.power_efficient=1 quiet loglevel=3 rd.systemd.show_status=auto rd.udev.log_priority=3 showopts The kernel has nothing to do with this issue, it seems.
Nicolas, I didn't get the idea of your experiment. What I know for sure: - the difference of desktop system responsiveness is extremely different between current Tumbleweed release and 20181015 snapshot. I've made a test with clean install 20181015, 20181018 and further snapshots. The difference appears just on 20181018 snapshot. 20181015 was really better. - the issue depends on the processor type (and GPU probably, but not sure), the affected CPUs: Intel with HT and all which has IBRS spectre_v2 mitigation. For example: Intel i5-6300U CPU is affected when HT enabled: /sys/devices/system/cpu/vulnerabilities/spectre_v2: Mitigation: Indirect Branch Restricted Speculation, IBPB: conditional, IBRS_FW, STIBP: disabled, RSB filling If HT is disabled or spectre_v2=retpoline was specified in kernel options the system is more responsive and
Earlier in this thread we thought the problem was caused by a difference in the kernel configuration between Fedora and openSUSE. We got as far as diffing the /proc/config files. I wanted to test that hypothesis.
I will share more observations, I installed openSUSE TW on another PC, with a good video card with which there should not be any problems, but with the CPU of the older model core i7 2600K, only the presence of a good video card didn’t have any effect, I also have the laptop, which is less productive than the PC under test, but does not have a discrete video card, works on the INTEL graphics, and the problem is not so noticeable there. A rather strange result is obtained, the presence of a good GPU accelerator does not help at all, it seems that instead of hardware acceleration, software acceleration works, but there is no software acceleration in GNOME, it always works on the GPU, but the worse the CPU is, the worse GNOME works. But unfortunately this does not affect the delays in animations, the delay is always about 2 seconds between the action and the execution
Now disconnected the discrete graphics, and booted from the integrated Intel HD, and there is no difference at all, that with GTX970 that with Intel HD, the behavior is absolutely the same
Created attachment 798388 [details] slow motion (4x) video comparison of Ubuntu and openSUSE Tumbleweed with and without special parameters Nothing new regarding solution, but if someone happens to have wished for better objective comparison, here's a 4-way slow motion video of Ubuntu vs openSUSE. Concentrate either on the pair of videos at the top: - Ubuntu defaults vs openSUSE with nosmt=force, spectre_v2=retpoline and energy_performance_preference set to balance_performance. or the pair of videos at the bottom: - Ubuntu defaults vs openSUSE defaults In both pairs of the videos, the overview button is pressed at the same time on both machines. That means that not only performance/smoothness is shown, but also the lag ie when there's visible action on the screen. This mainly just confirms that a) performance with defaults is very bad, quite unusable, b) performance with known tweaks is better, but still far worse than it should be.
And adding that both machines were set to use same resolution (1920x1080) and have essentially same GPU+CPU - SUSE machine has slightly faster i7-8650 while the Ubuntu machines has i7-8550.
(In reply to Timo Jyrinki from comment #164) > > This mainly just confirms that a) performance with defaults is very bad, > quite unusable, b) performance with known tweaks is better, but still far > worse than it should be. It supports, but I wouldn't say it confirms much After all, what versions of what packages are being really tested there? Is Ubuntu GNOME using identical binaries to openSUSE GNOME? If no, then those changes could be part of the factor Does Ubuntu GNOME have additional patches, or less patches than oepnSUSE GNOME? Those differences could be part of this story And the same for all of the libraries and even the kernel in use.. your testing helps, but without a full analysis of the differences at play, it doesn't help us a great deal.
With confirming I largely mean the same you do by supporting :) I mean it's confirmed there is a problem somewhere, and like I said I'm aware that this video does not bring anything new to the table regarding fixing the problem. Sometimes a video is just easier to consume than subjective observations over multiple reboots, so I thought it'd be nice to have a direct comparison video. My current best but not necessarily very good guess about the root cause for the main performance degradation is some openSUSE specific patch(es) somewhere in GNOME stack / libraries it uses. I'm also interested to see if there's anything meaningful changing once transition from libmozjs52 to libmozjs60 is complete - but I do see my TW is already using the latter for gnome-shell at least. Regarding some of the different factors you mentioned, I believe it's not something Ubuntu has added, since Fedora (reportedly, I should test at some point myself) performs as well as Ubuntu, and it's unlikely both Fedora and Ubuntu would carry a non-upstreamed critical patch for such a long time. The problem has also persisted for several months, so there has been ample time for all the development versions of the distributions to at least somewhat get "in sync", and all of them have had eg the same upstream GNOME 3.30 stack that is currently in openSUSE Tumbleweed. Some person in this thread has, in addition to the kernel parameters mentioned, tried running Fedora kernel directly in openSUSE without a big help, so it seems the kernel alone does not fix the lagginess. Surely the hyper-threading disabling and IBRS -> retpoline help, but as seen on the video not enough to get on par and no other distribution of course needs to disable hyper-threading. So here's just my summary of my current thinking. When I have extra time I always try to think of something new to try or study, but my main requirement was to get a usable desktop for doing other work with SUSE, which I now have with the workarounds. Something on my mind is to check that 20181015 <-> 20181018 snapshot delta mentioned in comment #160 and see if the difference is as big as what I have there on the video between distributions.
(In reply to Timo Jyrinki from comment #167) > Something on my mind is to check that 20181015 <-> 20181018 snapshot delta > mentioned in comment #160 and see if the difference is as big as what I have > there on the video between distributions. I think this is the first step that can be done. I unfortunately don't have a spare machine and the time to run the tests, but I think a binary search in the updates that were applied between 20181015 and 20181018 would help pinpoint the root cause. Assuming that the 20181018 snapshot has broken performance, there are N package updates between 20181015 and 20181018. One can split these N package updates in N1 and N2. It should happen that one of these contains the problematic update, so applying N1 (for instance) should reproduce the problem but not N2. And so on and so forth. If anyone has the possibility to do this I think it would be a worthwhile exercise.
(In reply to Robert Munteanu from comment #168) > (In reply to Timo Jyrinki from comment #167) > > Something on my mind is to check that 20181015 <-> 20181018 snapshot delta > > mentioned in comment #160 and see if the difference is as big as what I have > > there on the video between distributions. > > I think this is the first step that can be done. I unfortunately don't have > a spare machine and the time to run the tests, but I think a binary search > in the updates that were applied between 20181015 and 20181018 would help > pinpoint the root cause. > > Assuming that the 20181018 snapshot has broken performance, there are N > package updates between 20181015 and 20181018. One can split these N package > updates in N1 and N2. It should happen that one of these contains the > problematic update, so applying N1 (for instance) should reproduce the > problem but not N2. And so on and so forth. > > If anyone has the possibility to do this I think it would be a worthwhile > exercise. It is not necessary to have another PC, you can use the virtualbox 6.x version and choose VMSVGA as the graphic controller, there is very good performance in GNOME, there is a list of updates here > http://mirror.tspu.ru/opensuse/tumbleweed/iso/Changes.20181018.txt
Created attachment 798797 [details] after patches partially managed to solve the problem, if I remember correctly, then in mutter 3.28 there was a revert patch, after removing it, it seems that problems started, but this did not solve all the problems, delays are still observed
Anyone can check > sudo zypper ar -f obs://home:Dead_Mozay:branches:openSUSE:Factory home:Dead_Mozay:branches:openSUSE:Factory and update with vendor change > sudo zypper dup --from home:Dead_Mozay:branches:openSUSE:Factory --allow-vendor-change
(In reply to Dead Mozay from comment #171) > Anyone can check > > sudo zypper ar -f obs://home:Dead_Mozay:branches:openSUSE:Factory home:Dead_Mozay:branches:openSUSE:Factory > and update with vendor change > > sudo zypper dup --from home:Dead_Mozay:branches:openSUSE:Factory --allow-vendor-change Cool, thanks for setting this up! For the record, these are the instructions that worked for me $ sudo zypper ar -r https://download.opensuse.org/repositories/home:/Dead_Mozay:/branches:/openSUSE:/Factory/openSUSE_Factory/home:Dead_Mozay:branches:openSUSE:Factory.repo $ sudo zypper dup --from home_Dead_Mozay_branches_openSUSE_Factory --allow-vendor-change And even though the package is a branch of openSUSE_Factory, which sounds scary, it only brings in mutter with 4 additional patches: The following 4 packages are going to be upgraded: libmutter-3-0 mutter mutter-data mutter-lang In my testing the lag situation seems to be improved, although not completely smooth yet: - when hitting the 'super' key to open the applications menu, the delay is much smaller, but still noticeable - when pressings the 'show applications' button there is a definite stutter/delay when the application icons should be flying in So a definite improvement, but still not as it should be. Host info: $ inxi -G -v 1 System: Host: rombert.corp.adobe.com Kernel: 4.20.12-1-default x86_64 bits: 64 Desktop: Gnome 3.30.2 Distro: openSUSE Tumbleweed 20190301 CPU: 6-Core: Intel Core i7-8850H type: MT MCP speed: 800 MHz min/max: 800/4300 MHz Graphics: Device-1: Intel UHD Graphics 630 driver: i915 v: kernel Device-2: NVIDIA GP107GLM [Quadro P2000 Mobile] driver: nvidia v: 418.43 Display: x11 server: X.Org 1.20.4 driver: modesetting,nvidia resolution: 3840x2160~60Hz OpenGL: renderer: Quadro P2000/PCIe/SSE2 v: 4.6.0 NVIDIA 418.43 Drives: Local Storage: total: 961.19 GiB used: 97.53 GiB (10.1%) Info: Processes: 367 Uptime: 4h 16m Memory: 31.03 GiB used: 5.61 GiB (18.1%) Shell: bash inxi: 3.0.32
Created attachment 798904 [details] VIdeo of behaviour after installing patched mutter from home:Dead_Mozay:branches:openSUSE:Factory
(In reply to Robert Munteanu from comment #172) > (In reply to Dead Mozay from comment #171) > > Anyone can check > > > sudo zypper ar -f obs://home:Dead_Mozay:branches:openSUSE:Factory home:Dead_Mozay:branches:openSUSE:Factory > > and update with vendor change > > > sudo zypper dup --from home:Dead_Mozay:branches:openSUSE:Factory --allow-vendor-change > > Cool, thanks for setting this up! > > For the record, these are the instructions that worked for me > > $ sudo zypper ar -r > https://download.opensuse.org/repositories/home:/Dead_Mozay:/branches:/ > openSUSE:/Factory/openSUSE_Factory/home:Dead_Mozay:branches:openSUSE:Factory. > repo > $ sudo zypper dup --from home_Dead_Mozay_branches_openSUSE_Factory > --allow-vendor-change > > And even though the package is a branch of openSUSE_Factory, which sounds > scary, it only brings in mutter with 4 additional patches: > > The following 4 packages are going to be upgraded: > libmutter-3-0 mutter mutter-data mutter-lang > > In my testing the lag situation seems to be improved, although not > completely smooth yet: > > - when hitting the 'super' key to open the applications menu, the delay is > much smaller, but still noticeable > - when pressings the 'show applications' button there is a definite > stutter/delay when the application icons should be flying in > > So a definite improvement, but still not as it should be. > > Host info: > > $ inxi -G -v 1 > System: Host: rombert.corp.adobe.com Kernel: 4.20.12-1-default x86_64 > bits: 64 Desktop: Gnome 3.30.2 > Distro: openSUSE Tumbleweed 20190301 > CPU: 6-Core: Intel Core i7-8850H type: MT MCP speed: 800 MHz min/max: > 800/4300 MHz > Graphics: Device-1: Intel UHD Graphics 630 driver: i915 v: kernel > Device-2: NVIDIA GP107GLM [Quadro P2000 Mobile] driver: nvidia v: > 418.43 > Display: x11 server: X.Org 1.20.4 driver: modesetting,nvidia > resolution: 3840x2160~60Hz > OpenGL: renderer: Quadro P2000/PCIe/SSE2 v: 4.6.0 NVIDIA 418.43 > Drives: Local Storage: total: 961.19 GiB used: 97.53 GiB (10.1%) > Info: Processes: 367 Uptime: 4h 16m Memory: 31.03 GiB used: 5.61 GiB > (18.1%) Shell: bash inxi: 3.0.32 that's why I wrote that partially solves the problem, if hyper threading is enabled, try disabling
(In reply to Dead Mozay from comment #174) > (In reply to Robert Munteanu from comment #172) > > (In reply to Dead Mozay from comment #171) > > > Anyone can check > > > > sudo zypper ar -f obs://home:Dead_Mozay:branches:openSUSE:Factory home:Dead_Mozay:branches:openSUSE:Factory > > > and update with vendor change > > > > sudo zypper dup --from home:Dead_Mozay:branches:openSUSE:Factory --allow-vendor-change > > > > Cool, thanks for setting this up! > > > > For the record, these are the instructions that worked for me > > > > $ sudo zypper ar -r > > https://download.opensuse.org/repositories/home:/Dead_Mozay:/branches:/ > > openSUSE:/Factory/openSUSE_Factory/home:Dead_Mozay:branches:openSUSE:Factory. > > repo > > $ sudo zypper dup --from home_Dead_Mozay_branches_openSUSE_Factory > > --allow-vendor-change > > > > And even though the package is a branch of openSUSE_Factory, which sounds > > scary, it only brings in mutter with 4 additional patches: > > > > The following 4 packages are going to be upgraded: > > libmutter-3-0 mutter mutter-data mutter-lang > > > > In my testing the lag situation seems to be improved, although not > > completely smooth yet: > > > > - when hitting the 'super' key to open the applications menu, the delay is > > much smaller, but still noticeable > > - when pressings the 'show applications' button there is a definite > > stutter/delay when the application icons should be flying in > > > > So a definite improvement, but still not as it should be. > > > > Host info: > > > > $ inxi -G -v 1 > > System: Host: rombert.corp.adobe.com Kernel: 4.20.12-1-default x86_64 > > bits: 64 Desktop: Gnome 3.30.2 > > Distro: openSUSE Tumbleweed 20190301 > > CPU: 6-Core: Intel Core i7-8850H type: MT MCP speed: 800 MHz min/max: > > 800/4300 MHz > > Graphics: Device-1: Intel UHD Graphics 630 driver: i915 v: kernel > > Device-2: NVIDIA GP107GLM [Quadro P2000 Mobile] driver: nvidia v: > > 418.43 > > Display: x11 server: X.Org 1.20.4 driver: modesetting,nvidia > > resolution: 3840x2160~60Hz > > OpenGL: renderer: Quadro P2000/PCIe/SSE2 v: 4.6.0 NVIDIA 418.43 > > Drives: Local Storage: total: 961.19 GiB used: 97.53 GiB (10.1%) > > Info: Processes: 367 Uptime: 4h 16m Memory: 31.03 GiB used: 5.61 GiB > > (18.1%) Shell: bash inxi: 3.0.32 > > that's why I wrote that partially solves the problem, if hyper threading is > enabled, try disabling Yes, with: - your patches to mutter - hyper-threading disabled in BIOS - the latest TW kernel with the updated default config # CONFIG_PREEMPT_NONE is not set CONFIG_PREEMPT_VOLUNTARY=y # CONFIG_PREEMPT is not set Then responsiveness is spot on. I'll post another screencap for comparison.
Created attachment 798910 [details] VIdeo of behaviour after installing patched mutter from home:Dead_Mozay:branches:openSUSE:Factory _and_ disabling hyperthreading
Follow up comment 132 - On my T460s TW 20190310 Wayland Session (ongoing 5 hours intensive use) there were no noticable lags. Everything is smooth enough for me. There might be occasional hickups well below 300ms but no key repitition phantoms or similar disturbances. Boot param only retpoline. HT is ENABLED.
https://www.youtube.com/watch?v=wZFxpvts5t4 GNOME performance on gt 1030 w/ mutter patches from home:Dead_Mozay:branches:openSUSE:Factory. More hardware detail: https://termbin.com/gzow Hyperthreading is on.
I've been having this problem with an i5-8265U CPU and was about to install the mutter patches. It appears that there are also dbus packages there now: The following 7 packages are going to be upgraded: dbus-1 dbus-1-x11 libdbus-1-3 libmutter-3-0 mutter mutter-data mutter-lang Should I install the whole set, or just the mutter packages? Glad I found this bug. Brand new laptop, and it was having problems out of the box...
Leap 15.0 has been affected with the latest kernel update (openSUSE-SU-2019:1193-1, date 2019-04-12) on Skylake family CPUs as well. I actually have Whiskeylake CPUs. Using "spectre_v2=retpoline" now. Next Leap 15.0 kernel update will have the fix: > https://github.com/openSUSE/kernel-source/commit/6ab40f512f045ae18bba80b0ec5190e136e08bd8
Today the problem is still present. `spectre_v2=retpoline,generic` partially solves the problem, but not completely
(In reply to Dead Mozay from comment #184) > Today the problem is still present. `spectre_v2=retpoline,generic` partially > solves the problem, but not completely I am not sure what do you mean exactly. Do you still see the original performance problem? Something else maybe? In any case, do not clear needinfo that is not targeting you or which you haven't set up yourself. Thank you
This bug may be quite near unmaintainable because many different performance problems are being discussed here, but I still appreciate the knowledge about kernel advancements too. The original problem is that something regressed badly with the GNOME 3.30 update, and there hasn't been a single cause found. I hope 3.32 improves the situation. I didn't mention it here but I browsed through SUSE specific patches of all the packages that were updated on that mentioned October date, and I didn't find much suspicious. Mutter seemed the most likely candidate to improve, so I tried to use all patches from Ubuntu 18.10's 3.30.2 but it didn't help the performance by a giant leap (https://build.opensuse.org/package/show/home:tjyrinki_suse:branches:GNOME:Factory/mutter?expand=0 - now seems broken lately, used to build) Regardless of the GNOME stack, it's positive for desktop users that the CONFIG_PREEMPT_VOLUNTARY=y is now the default and IBRS is being dropped in favor of retpoline. Those will make it at least easier to know the kernel isn't involved in the GNOME performance problems.
Updated my TW to Gnome 3.32 today and the animations seems to be much smoother again. Need to see what happens if I remove my earlier workarounds: (noht spectre_v2=retpoline l1tf=off spectre_v2_user=off)
Removing all above mentioned configs brought back some visible lag (mainly "latency" in UI animations, less (but existing) stuttering in animations).
I am seeing normal performance (no stutters) as of today, with: - 'standard' kernel boot line, BOOT_IMAGE=... root=... splash=silent resume=... quiet (no spectre_v2 changes ) - only standard packages ( no patched mutter at all from comment 5 and others ) - hyper-threading enabled - all packages updated as of snapshot 20190506 ( so Gnome 3.32, Kernel 5.0.11 ) $ inxi -CG CPU: Topology: 6-Core model: Intel Core i7-8850H bits: 64 type: MT MCP L2 cache: 9216 KiB Speed: 801 MHz min/max: 800/4300 MHz Core speeds (MHz): 1: 800 2: 800 3: 800 4: 800 5: 800 6: 800 7: 800 8: 800 9: 800 10: 800 11: 800 12: 801 Graphics: Device-1: Intel UHD Graphics 630 driver: i915 v: kernel Device-2: NVIDIA GP107GLM [Quadro P2000 Mobile] driver: nvidia v: 418.56 Display: x11 server: X.Org 1.20.4 driver: modesetting,nvidia resolution: 3840x2160~60Hz OpenGL: renderer: Quadro P2000/PCIe/SSE2 v: 4.6.0 NVIDIA 418.56
Created attachment 804600 [details] slow motion (4x) comparison GNOME 3.32 Ubuntu 19.04 <-> openSUSE TW 2019-05-09 Here's an update on the direct comparison video I posted earlier. Note that screen captures at 30fps are not useful to see actual performance, one needs at least 60fps external capture. 120fps starts to be nice as the screens are 60Hz so there is some extra (temporal) room. What remains: - Some core problem remains on SUSE with GNOME 3.32, even with matching energy_performance_preference set to balance_performance. The animations are choppy, on the comparison distro they are also visibly (non slow motion) silky smooth most of the time (GNOME still has some random choppiness, seen also at the end of the video). What is improved - a lot!: - The performance is now usable with default kernel parameters and hyper-threading enabled - Lag from key press to animation seems better than before - GNOME 3.32 general optimizations probably make balance_power to be ok too (not as much difference this time as before) -- Other notes: I also tested Wayland which is the default on SUSE: - Performance problem seems roughly similar there - Wayland would seem to benefit a bit more from balance_performance (compared to balance_power), making it faster than X.Org - Key press lag is higher on Wayland regardless of the Overview key used Finally testing related note: - For comparing while pressing keys on both machines at the same time I noted it's useful to bind and use non-Super key to Overview on both machines, since the Super key adds extra delay (probably since it waits for another key)
This is an autogenerated message for OBS integration: This bug (1112824) was mentioned in https://build.opensuse.org/request/show/702938 15.0 / kernel-source https://build.opensuse.org/request/show/702940 42.3 / kernel-source
Created attachment 805136 [details] slow (4x) comparison openSUSE Leap 15 Wayland <-> openSUSE TW Wayland Here's a comparison with openSUSE Leap 15, and Wayland in use. I finally have four installations on two similar hw laptops to compare amongst. The situation seems the same despite 15's older, less optimized GNOME. The difference is obvious, with 15 having smooth animations and TW not. Now that I have good test setups to play with, I'll be happy to test anything that might improve the situation and report back. Currently I'm out of ideas to try though.
Hi folks, This is my laptop info: I use gnome desktop, when I press "win" key, activity switching animation, slow, not serious, but visible to the naked eye. the Solution: unplug AC power. When laptop using "Bat mode", The response of the system is obviously faster. "x86_energy_perf_policy" be changed between "AC mode" and "Bat mode", seems like root cause. I will upload AC mode log and Bat mode log, they come from tlp-stat report.
Created attachment 805327 [details] ac mode report from tlp-stat
Created attachment 805328 [details] bat mode report from tlp-stat
The Leap 15.0 kernel update has been released and "spectre_v2=retpoline" can be dropped again. It also comes with MDS mitigation. Thanks!
This is an autogenerated message for OBS integration: This bug (1112824) was mentioned in https://build.opensuse.org/request/show/705249 15.1 / kernel-source
Hey, > "x86_energy_perf_policy" be changed between "AC mode" and "Bat mode", seems > like root cause. > > I will upload AC mode log and Bat mode log, they come from tlp-stat report. We recently had a discussion about x86_energy_perf_policy that might be related: We dropped an upstream patch in some kernel code-streams for the x86_energy_perf_policy handling because it was broken, but it turned out, that that wasn't working either. See the discussion here: https://patchwork.kernel.org/patch/10853023/ As response to this thread, Rafael made a new patch improving the handling, see this thread: https://patchwork.kernel.org/patch/10862699/ I don't know the state of that patch, according to patchwork it is not yet merged upstream. Is this related?
(In reply to Simon Schricker from comment #209) > Hey, > > > "x86_energy_perf_policy" be changed between "AC mode" and "Bat mode", seems > > like root cause. > > > > I will upload AC mode log and Bat mode log, they come from tlp-stat report. > > We recently had a discussion about x86_energy_perf_policy that might be > related: We dropped an upstream patch in some kernel code-streams for the > x86_energy_perf_policy handling because it was broken, but it turned out, > that that wasn't working either. > > See the discussion here: > https://patchwork.kernel.org/patch/10853023/ > > As response to this thread, Rafael made a new patch improving the handling, > see this thread: > https://patchwork.kernel.org/patch/10862699/ > > I don't know the state of that patch, according to patchwork it is not yet > merged upstream. > > Is this related? almost all problems were solved by disabling debugging when building mozjs60 https://build.opensuse.org/request/show/706263 Animations have become smooth, but there are still some jerks in animations, but overall it is much better.
This is an autogenerated message for OBS integration: This bug (1112824) was mentioned in https://build.opensuse.org/request/show/714223 15.0 / kernel-source
This is an autogenerated message for OBS integration: This bug (1112824) was mentioned in https://build.opensuse.org/request/show/715440 15.1 / kernel-source
SUSE-SU-2019:1870-1: An update that solves 7 vulnerabilities and has three fixes is now available. Category: security (important) Bug References: 1102340,1112824,1130159,1133190,1134395,1135603,1136922,1137194,1138293,1139751 CVE References: CVE-2018-20836,CVE-2018-5390,CVE-2018-7191,CVE-2019-11487,CVE-2019-12456,CVE-2019-12614,CVE-2019-12818 Sources used: SUSE Linux Enterprise Server for SAP 12-SP1 (src): kernel-default-3.12.74-60.64.118.1, kernel-source-3.12.74-60.64.118.1, kernel-syms-3.12.74-60.64.118.1, kernel-xen-3.12.74-60.64.118.1, kgraft-patch-SLE12-SP1_Update_35-1-2.3.1 SUSE Linux Enterprise Server 12-SP1-LTSS (src): kernel-default-3.12.74-60.64.118.1, kernel-source-3.12.74-60.64.118.1, kernel-syms-3.12.74-60.64.118.1, kernel-xen-3.12.74-60.64.118.1, kgraft-patch-SLE12-SP1_Update_35-1-2.3.1 SUSE Linux Enterprise Module for Public Cloud 12 (src): kernel-ec2-3.12.74-60.64.118.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Created attachment 813326 [details] 20190808 openSUSE Leap 15 Wayland <-> openSUSE TW Wayland Another comparison. TW is still choppier, especially seeing it live side-by-side, while Leap 15.0 is completely smooth. Tested with the same methodology as before: 1920x1080, Wayland, balance_performance preference and the 4 roughly similar windows open. mozjs52 never had --enable-debug, and the original problem arrived with the new GNOME version that used mozjs52. The removal of mozjs60 debug option (a good catch!) improved things again, but not completely. I checked out the whole b.o.o GNOME:Factory and did not find anything else defining --enable-debug, or anything suspicious in the .spec files with a "debug" grep. Nowadays it seems finally possible to get smooth animations with this methodology on TW using the 'performance' preference, which eats battery a lot (tested) so I'm only using it for CPU_HWP_ON_AC in /etc/default/tlp. On Leap 15 there is no noticeable change between the battery preserving balance_performance and performance preferences in this test case, as it's already completely smooth on the lower setting.
SUSE-SU-2019:2450-1: An update that solves 21 vulnerabilities and has 160 fixes is now available. Category: security (important) Bug References: 1012382,1051510,1053043,1055117,1061840,1065600,1065729,1068032,1071995,1083647,1083710,1088047,1094555,1098633,1102247,1106383,1106751,1109137,1111666,11123080,1112824,1113722,1114279,1115688,1117158,1118139,1119222,1120423,1120566,1124167,1124503,1127034,1127155,1127315,1128432,1128902,1128910,1129770,1130972,1132154,1132390,1133021,1133401,1133738,1134097,1134303,1134390,1134393,1134395,1134399,1134671,1135296,1135335,1135556,1135642,1135661,1136157,1136424,1136598,1136811,1136896,1136922,1136935,1136990,1137103,1137162,1137194,1137366,1137372,1137429,1137444,1137458,1137534,1137535,1137584,1137586,1137609,1137625,1137728,1137739,1137752,1137811,1137827,1137884,1137995,1137996,1137998,1137999,1138000,1138002,1138003,1138005,1138006,1138007,1138008,1138009,1138010,1138011,1138012,1138013,1138014,1138015,1138016,1138017,1138018,1138019,1138291,1138293,1138374,1138375,1138589,1138719,1139358,1139751,1139771,1139782,1139865,1140133,1140139,1140322,1140328,1140405,1140424,1140428,1140575,1140577,1140637,1140652,1140658,1140715,1140719,1140726,1140727,1140728,1140814,1140887,1140888,1140889,1140891,1140893,1140903,1140945,1140954,1140955,1140956,1140957,1140958,1140959,1140960,1140961,1140962,1140964,1140971,1140972,1140992,1141401,1141402,1141452,1141453,1141454,1141478,1141488,1142023,1142112,1142220,1142221,1142265,1142350,1142351,1142354,1142359,1142450,1142701,1142868,1143003,1143045,1143105,1143185,1143189,1143191,1143507 CVE References: CVE-2018-16871,CVE-2018-20836,CVE-2018-20855,CVE-2019-10126,CVE-2019-10638,CVE-2019-10639,CVE-2019-1125,CVE-2019-11477,CVE-2019-11478,CVE-2019-11599,CVE-2019-11810,CVE-2019-12380,CVE-2019-12456,CVE-2019-12614,CVE-2019-12818,CVE-2019-12819,CVE-2019-13631,CVE-2019-13648,CVE-2019-14283,CVE-2019-14284,CVE-2019-3846 Sources used: SUSE Linux Enterprise Real Time Extension 12-SP4 (src): kernel-rt-4.12.14-8.3.1, kernel-rt_debug-4.12.14-8.3.1, kernel-source-rt-4.12.14-8.3.1, kernel-syms-rt-4.12.14-8.3.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
I think by now we can close this; there were fixes for mozjs, kernel and GNOME all targetting performance.