Bug 1074138 - Yast2 package manager UI very slow when listing many packages
Yast2 package manager UI very slow when listing many packages
Status: RESOLVED WONTFIX
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: YaST2
Current
Other Other
: P3 - Medium : Normal (vote)
: ---
Assigned To: YaST Team
Jiri Srain
https://trello.com/c/qDRX1OM5
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2017-12-27 13:52 UTC by Adam Majer
Modified: 2022-08-23 09:46 UTC (History)
3 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
Slow package removal log (7.03 MB, application/x-xz)
2018-01-03 09:30 UTC, Adam Majer
Details
fast package removal logs (6.92 MB, application/x-xz)
2018-01-03 09:35 UTC, Adam Majer
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Adam Majer 2017-12-27 13:52:43 UTC
I have a basic install. I would like to look through installed package list and uninstall extra things I don't really need anymore. So,

  1. yast2 -> Package Manager
  2. list all packages
  3. click on a package like Mesa-libGLESv2-devel and click uninstall
  4. wait 30+ seconds while Package Manager window is completely unresponsive and 100% CPU usage.

workaround,

  1. filter to only list few packages.
  2. click uninstall

Please fix. There is no reason Qt based UI should be slow, especially for simple tables like this.

This problems seems to exist in all versions of Yast.
Comment 2 Adam Majer 2018-01-03 09:30:06 UTC
Created attachment 754608 [details]
Slow package removal log
Comment 3 Adam Majer 2018-01-03 09:35:38 UTC
Created attachment 754609 [details]
fast package removal logs

Attached two logs. In both cases Yast2 is run with `sudo -E yast2`

Slow log is result of 

> 1. Software Management
> 2. Package Groups
> 3. All packages
> 4. Trying to remove some -devel package there. UI stuck for "few seconds"
> 5. Only 1 package removed.

Fast log is result of,

> 1. Software Management
> 2. Search
> 3. type substring of package so only few packages get listed (like 10?)
> 4. Click to remove some package there. UI is snappy.
> 5. Only 1 package removed.
Comment 4 Josef Reidinger 2018-01-09 15:46:42 UTC
thanks, for logs and way to reproduce it. I reproduce it locally. From logs I see that problem is not solver, so it is problem with too many packages in UI, which need time to render.

Reason why second case is facter is that only few packages are listed, so it operates faster, but e.g. in my case the first case shows around 53 000 packages, which slows down UI.
Comment 5 Adam Majer 2018-01-09 20:58:48 UTC
(In reply to Josef Reidinger from comment #4)
> thanks, for logs and way to reproduce it. I reproduce it locally. From logs
> I see that problem is not solver, so it is problem with too many packages in
> UI, which need time to render.
> 
> Reason why second case is facter is that only few packages are listed, so it
> operates faster, but e.g. in my case the first case shows around 53 000
> packages, which slows down UI.


And since Yast2 is using Qt, the only way this is slow is if you are using Qt wrong. Qt Model/View framework can very easily scale to millions of rows and be just as efficient as displaying 100 items. The only problem is using convenience classes that use QWidget based classes inside containers. Using 50,000+ QWidget based items will slow things down but I didn't look if this is what Yast2 is doing.
Comment 6 Stefan Hundhammer 2018-01-10 12:27:14 UTC
This does not use the model/view framework directly, it uses a QListWidget which in turn was a conversion from the old Qt 3 QListView. So it's item-based which is the reason why performance breaks down at insanely large lists. It was never meant for lists of 50,000+ items. And frankly, I have doubts how usable a list that huge is even if it responds really quickly. You have to scroll a lot to get to the item you really want.

Basically, this feeds those who always said that this "view all" filter should not be there to begin with. I defended that for so long, but maybe it's time to simply stop defending it.

For one thing, the list has become unreasonably large; back when I wrote that piece of code, we had something in the order of 1000 or maybe 2000 packages in total. But with the package split frenzy now we have so many packages that I wonder more and more how useful that is at all - both the number of packages to deal with and displaying them all at once in a single list.

Maybe we should simply drop that "all packages" filter.
Comment 7 Stefan Hundhammer 2018-01-10 12:40:12 UTC
BTW on my machine (latest Tumbleweed) I have 4399 packages installed with a total of 442552 files on the root filesystem, so it's just 100.6 files per package on average (with a median size of 2.6 kB). I think we took the package splits too far.
Comment 8 Adam Majer 2018-01-10 14:07:31 UTC
In "modern" UI, like if you look at Thunderbird or Firefox or Chrome, when you want to find something, you have search-on-type feature. A snappy UI allows for something like that to actually be implemented.

Listing all packages is useful, especially if I want to remove things I don't use and I don't know in which category things exist. In my case, it would be going down the list of all installed packages and seeing if I can uninstall things I don't use anymore.
Comment 9 Lukas Ocilka 2018-01-10 14:33:44 UTC
Let's put it this way: in rare cases, someone needs to list all installed
packages and listing all of them might be actually very slow, especially
when we use many remote repositories.

On the other hand, you are actually asking for another feature: Listing ALL
INSTALLED packages to remove some of them.

Our proposal for a workaround: Do not install those that you don't want.
Comment 10 Stefan Hundhammer 2018-01-10 14:50:26 UTC
(In reply to Adam Majer from comment #8)
> In "modern" UI, like if you look at Thunderbird or Firefox or Chrome, when
> you want to find something, you have search-on-type feature. A snappy UI
> allows for something like that to actually be implemented.

We have a search filter view. Just use that one if you want to search something. We are not trying to recreate Thunderbird here (which has this kind of search box right above the list), and both Firefox and Chrome don't deal with lists at all; they have a generic "Find on this page" search function which is something completely different.
 
> Listing all packages is useful, especially if I want to remove things I
> don't use and I don't know in which category things exist. In my case, it
> would be going down the list of all installed packages and seeing if I can
> uninstall things I don't use anymore.

Just use the "Installation Summary" filter view where you can select packages by status - e.g. "keep" in this example. 

Or use the "Repositories" filter view and select "@System".

Both will give you a considerably shorter list that will not consume as many resources and slow down performance so drastically. I see no noticable delay when I do that with my 4399 installed packages.

The only (easy) way to get all packages from all repositories is to go to the "RPM Groups" filter view and then select "ZZZ All".

Guess why it's on the bottom of that tree: Because it is expected to slow down performance.

As I wrote, I repeatedly defended that feature against requests to remove it because albeit it does have a cost in terms of performance and resource consumption, it is useful to some people in some cases. If this is now used against us, I strongly vote for removing it.

Rewriting that entire part of the YQPackageManager because of that is just insane; and that's what this comes down to. 

As I wrote not so long ago in a thread on [research] that model-view part of the Qt item widgets is a royal PITA, it is poorly documented, it is very hard to test, it is hard to maintain. And for what? For a single-digit number of users who use the application the wrong way?

Just use it the way it was meant to be used and be happy.

See also 
  https://mailman.suse.de/mlarch/SuSE/research/2017/research.2017.03/msg00035.html


> Re: [Research] Qt: How to hook into destruction of a widget?
> To: research@suse.de
> Subject: Re: [Research] Qt: How to hook into destruction of a widget?
> From: Stefan Hundhammer <shundhammer@suse.de>
> Date: Tue, 7 Mar 2017 11:47:12 +0100
> In-reply-to: <c37738d9-2c8c-919b-47d3-df70946edf4a@suse.de>
> List-archive: <https://mailman.suse.de/mailman/private/research/>
> 
> 
> 
> On 07.03.2017 11:23, Adam Majer wrote:
> 
> > I don't mean to be too picky here, just want to point out there are
> > better ways of doing these things. QListView and relevant data model
> > classes are **much** superior at this than using old-style
> > QListWidget.  Model/View separation has its many advantages.
> > 
> After having gone through this with QDirStat and after having given 7
> or 8 week-long Qt training courses each covering (almost) all aspects
> of Qt, I can only say:
> 
> No. Not only no, but HELL, NO.
> 
> Qt's model/view classes are made by theoretical people in their ivory
> tower for other theoretical people in their other ivory tower. They
> are a royal PITA, the documentation is (unlike all other Qt
> documentation) piss-poor, there are aspects of it that are completely
> (!) undocumented (talk about QPersistentModelIndex), and they are
> using private methods in their own derived classes all the time (which
> should have given them some hints that their own API is insufficient
> and too limiting), and it is completely undebuggable.
> 
> After working almost exclusively with and on Qt (we maintained the Qt
> libs for the Nokia Harmattan/MeegoTouch project on the N9 phone) I
> struggled with this stuff for QDirStat and just barely got it to work
> during the 12/2015 hackweek.
> 
> As a matter of fact, I had already officially given up QDirStat
> because of that just 2 hours prior to the 2015 SUSE Christmas party
> (already written a really interesting rant for its home page) because
> this crap just did random things with no means whatsoever to debug
> anything.
> 
> I didn't go to the party because I was so pissed off at that crap: I
> didn't want to spoil everybody else's party because of that. Only some
> hours later I got yet another idea how to track the problem down, and
> then I found what I had done wrong - without ANY of those Qt classes
> giving any hint whatsoever what they didn't like. I was just lucky.
> 
> Qt is otherwise a well-designed and great framework. But that
> model-view stuff is just a fucking piece of shit; I can't put it into
> other words.
> 
> So, for everybody just starting with Qt: By all means, forget that
> model-view stuff until you are comfortable with the easy way of doing
> things.
> 
> And when you start doing model-view stuff, try the easy
> one-dimensional list first, then work your way up to the
> two-dimensional table. Should you consider using the tree model,
> consider psychotherapy first to check what is wrong with you and what
> exactly you are trying to prove. Then talk to somebody who has done
> this before and ask him if he would really do it again.
> 
> For the purposes of average programming, use the QListWidget /
> QListWidgetItem classes rather than the QListView / QAbstractItemModel
> etc. classes. It's just too painful to use the latter with little (if
> any) gain whatsoever.
> 
> 
> Kind regards
> --
> Stefan Hundhammer  <shundhammer@suse.de>
> YaST Developer / QDirStat Author
> Qt since 1998 / Certified Qt Developer / Qt Trainer
> 
> SUSE Linux GmbH
> GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg)
> Maxfeldstr. 5, 90409 Nürnberg, Germany
Comment 11 Adam Majer 2018-01-10 14:51:16 UTC
(In reply to Lukas Ocilka from comment #9)
> Let's put it this way: in rare cases, someone needs to list all installed
> packages and listing all of them might be actually very slow, especially
> when we use many remote repositories.
> 
> On the other hand, you are actually asking for another feature: Listing ALL
> INSTALLED packages to remove some of them.

Not really. Yast package manager lists all packages, including installed ones. Sorting by install status gets me list of all installed packages.

Taking a long time to list anything is also not a problem. The problem is that changing any status of any of the item results in things stalling. So the interface is not interactive.


> Our proposal for a workaround: Do not install those that you don't want.

That's a poor workaround. Another would be to just re-install the system?

Somehow this works just fine in Aptitude on Debian with even more packages than Leap. But it doesn't work with Yast2 Qt GUI *or* console text UI.
Comment 12 Stefan Hundhammer 2018-01-11 10:58:44 UTC
(In reply to Adam Majer from comment #11)
> > Our proposal for a workaround: Do not install those that you don't want.
> 
> That's a poor workaround. Another would be to just re-install the system?

If Debian is your preference where you cherry-pick everything you like and ignore everything else, maybe...


> Somehow this works just fine in Aptitude on Debian with even more packages
> than Leap. 

Aptitude is an NCurses (text-mode) utility, so this is beyond the point you previously made about Qt's model/view.

OTOH Synaptic would be the Qt counerpart on Ubuntu (and Debian? not sure), and it doesn't have half (more exactly a quarter) of the features that our Qt package selector has. As a matter of fact, when I start that thing, it typically takes 10-20 seconds until it is finished building some index that always seems to be outdated.

So, it seems to me you are clutching at straws, relying on others just not knowing that stuff.

Also, how many packages exactly does Debian (or a recent Ubuntu) have that might possibly be displayed in any such "all packages" list? I could not find any numbers on the web.


> But it doesn't work with Yast2 Qt GUI *or* console text UI.

That should give you some hints that it's not a problem of how that list is implemented in our YQPackageSelector, but that it's rather a problem of the sheer mass of packages that accumulated with all that splitting up large packages into tiny bits and pieces.

Please remember that I do have a solid basis for comparison. When I started the data model in QDirStat, first I did it without sorting, and it was really fast, even with 400.000+ items in the tree. When I then introduced sorting with the thing that Qt recommends, the QSortFilterProxyModel, performance broke down completely because obviously it prefetched and sorted the complete tree in advance before showing *anything*. So I did that part myself (and this is when things get really hairy because the relevant Qt classes don't support you at all with this), and I only got snappy performance back because of deep internal knowledge of my data structures, lazy sorting and relying on not too many tree branches being open at the same time.

But this would not work at all with such a package list. In effect, such a data model would have to keep a duplicate list of all the packages in the current view and sort it according to the current sort column. Not only does that mean duplicate at least all the pointers, it's also that old computer science problem of doing an efficient sorting; O(n*ln(n)) is the best you can achieve (e.g. with a heap sort which does not easily degrade in the pathological case). But one way or the other, you have to sort the complete list in the model to be able to create a QModelIndex which you need if you create your own model.

And that's not much different than what the simplified QListWidget / QTableWidget (the convenience classes on top of QListView with a QAbstractListModel or QTableView with a QTableModel) do.

And if you now write that you see poor performance with the same "show all" view in the NCurses version as well this just proves the point.

So let's please stop this pointless discussion. We have too many packages now to handle them efficiently in such a "show all" list. So, if people insist that this is now too slow, the only real solution we have is not to offer that view anymore. Is that what you want?

We ALWAYS offered that only with the caveat that there would be a heavy performance penalty. This is why we don't simply show that complete list be default upon startup, why it's somewhat hidden away. If you cannot accept that limitation, don't use it.

The recommended way of all the use cases you mentioned is different for a good reason; I described it above in great detail.
Comment 13 Adam Majer 2018-01-11 11:41:05 UTC
(In reply to Stefan Hundhammer from comment #12)
> Also, how many packages exactly does Debian (or a recent Ubuntu) have that
> might possibly be displayed in any such "all packages" list? I could not
> find any numbers on the web.

Well over 60,000? When I have multi-arch enabled it lists over 120,000 just as quickly. And just logged in to my Sid VM, Aptitude with 63,000 packages uses exactly 1.6s CPU time to start and load everything.

But with all this this is *not* the issue.


> > But it doesn't work with Yast2 Qt GUI *or* console text UI.
> 
> That should give you some hints that it's not a problem of how that list is
> implemented in our YQPackageSelector, but that it's rather a problem of the
> sheer mass of packages that accumulated with all that splitting up large
> packages into tiny bits and pieces.

Frankly, I don't care *how* it is implemented. If you use widgets, it's fine. I don't care. But it shouldn't make the window non-interactive. That's the problem from where I sit, as a user, the interface is stuck.

I didn't look what is actually happening. Why the algorithm that updates a list doesn't scale.

Also, sorting is not the problem. I can resort the list in Yast without issues.

> And that's not much different than what the simplified QListWidget /
> QTableWidget (the convenience classes on top of QListView with a
> QAbstractListModel or QTableView with a QTableModel) do.
> 
> And if you now write that you see poor performance with the same "show all"
> view in the NCurses version as well this just proves the point.

No, the ncurses version has a different problem. And Aptitute ncurses version doesn't have that problem with more packages. Again, I didn't look what is actually happening, but from user standpoint, it's not good. But that's a *different* problem.

> So let's please stop this pointless discussion. We have too many packages
> now to handle them efficiently in such a "show all" list. So, if people
> insist that this is now too slow, the only real solution we have is not to
> offer that view anymore. Is that what you want?

That's a pointless discussion. Because you are just kicking the can down the road until some group will blow up with 50,000 packages. That may just happen. ;)

> We ALWAYS offered that only with the caveat that there would be a heavy
> performance penalty. This is why we don't simply show that complete list be
> default upon startup, why it's somewhat hidden away. If you cannot accept
> that limitation, don't use it.
> 
> The recommended way of all the use cases you mentioned is different for a
> good reason; I described it above in great detail.


Anyway, my point is,

  1. opening window with all packages takes time, NO PROBLEM
  2. scrolling around with all packages, fast, NO PROBLEM
  3. changing install state of any package, UI stuck, **PROBLEM**

So, not actually looking at the code (yet), by why is this happening? Why is the UI stuck when only few elements should be updated and one redrawn? Is this a Qt problem or Yast problem?
Comment 14 Stefan Hundhammer 2020-01-13 16:36:49 UTC
Some of this may be related to the performance fix in https://github.com/libyui/libyui-qt-pkg/pull/82 .

But some other things will remain unchanged by that PR.
Comment 15 Stefan Hundhammer 2022-08-23 09:46:49 UTC
No activity in this bug for 4 years, and the PR from comment #14 changed at least one performance bottleneck.

For the rest, it's unreasonable to expect we will do a major rewrite of this using Qt's model/view classes.

Closing as WONTFIX.