Bug 1199924 - [Build 20220524] qemu: buffer overflow detected
[Build 20220524] qemu: buffer overflow detected
Status: RESOLVED FIXED
: 1199967 (view as bug list)
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Virtualization:Other
Current
Other Other
: P5 - None : Normal (vote)
: ---
Assigned To: Dario Faggioli
E-mail List
https://openqa.opensuse.org/tests/237...
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2022-05-25 16:37 UTC by Dominique Leuenberger
Modified: 2022-07-04 19:17 UTC (History)
14 users (show)

See Also:
Found By: openQA
Services Priority:
Business Priority:
Blocker: Yes
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dominique Leuenberger 2022-05-25 16:37:40 UTC
## Observation


error: Failed to start domain 'vm-swtpm-legacy'
error: internal error: qemu unexpectedly closed the monitor: 2022-05-25T16:30:05.738186Z qemu-system-x86_64: -accel kvm: warning: Number of SMP cpus requested (2) exceeds the recommended cpus supported by KVM (1)
2022-05-25T16:30:05.738259Z qemu-system-x86_64: -accel kvm: warning: Number of hotpluggable cpus requested (2) exceeds the recommended cpus supported by KVM (1)
2022-05-25T16:30:05.742354Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2022-05-25T16:30:05.742369Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2022-05-25T16:30:05.743989Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2022-05-25T16:30:05.744050Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
*** buffer overflow detected ***: terminated

openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-security_swtpm@64bit fails in
[swtpm_verify](https://openqa.opensuse.org/tests/2375131/modules/swtpm_verify/steps/15)

## Test suite description
Maintainer - Richard Fan


## Reproducible

Fails since (at least) Build [20220524](https://openqa.opensuse.org/tests/2374114)


## Expected result

Last good: [20220523](https://openqa.opensuse.org/tests/2369000) (or more recent)


## Further details

Always latest result in this scenario: [latest](https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=security_swtpm&version=Tumbleweed)
Comment 1 Dominique Leuenberger 2022-05-25 18:47:56 UTC
Same error shows up in this case:
https://openqa.opensuse.org/tests/2375666#step/usr_sbin_dnsmasq/1

This test should be simpler, no swtpm involved
Comment 2 Mathias Homann 2022-05-26 11:12:35 UTC
I'm seeing the same errors. Had to roll back to 20220523.
Comment 3 Dominique Leuenberger 2022-05-26 11:15:05 UTC
(In reply to Mathias Homann from comment #2)
> I'm seeing the same errors. Had to roll back to 20220523.

Can you verify if it is 'just' the qemu packages that need a rollback? The change there in that snapshot looks quite harmless, but if we can confirm that, i will put a fever into snapshot 0527 and possibly into the update channel
Comment 4 Mathias Homann 2022-05-26 12:14:43 UTC
(In reply to Dominique Leuenberger from comment #3)
> (In reply to Mathias Homann from comment #2)
> > I'm seeing the same errors. Had to roll back to 20220523.
> 
> Can you verify if it is 'just' the qemu packages that need a rollback? The
> change there in that snapshot looks quite harmless, but if we can confirm
> that, i will put a fever into snapshot 0527 and possibly into the update
> channel

I went back up to 20220524, rebooted, verified that vms dont start.
Then I switched back to 20220523 with tumbleweed-cli and rolled back only all installed qemu packages, restarted libvird, and vms start again.
Comment 5 Dominique Leuenberger 2022-05-26 12:56:33 UTC
Thanks.

I just added qemu rev 227 (the one from snapshot 0523 into the Update channel.

It should appear there shortly (build in Progress)
This should then allow you to be fully up-to-date again without locking to older snapshots.

If confirmed, snapshot 0526 should get the guy intree instead of the update channel again
Comment 6 Mathias Homann 2022-05-26 15:19:38 UTC
(In reply to Dominique Leuenberger from comment #5)
> Thanks.
> 
> I just added qemu rev 227 (the one from snapshot 0523 into the Update
> channel.
> 
> It should appear there shortly (build in Progress)
> This should then allow you to be fully up-to-date again without locking to
> older snapshots.
> 
> If confirmed, snapshot 0526 should get the guy intree instead of the update
> channel again

I don't think that worked - switched back up to 0524, and did "zypper dup" - and I got the bad build of qemu again.
Comment 7 Dominique Leuenberger 2022-05-26 15:22:25 UTC
You did get the qemu packages from the update repo?

https://download.opensuse.org/repositories/openSUSE:/Factory:/Update/standard/
Comment 8 Mathias Homann 2022-05-26 15:25:02 UTC
(In reply to Dominique Leuenberger from comment #7)
> You did get the qemu packages from the update repo?
> 
> https://download.opensuse.org/repositories/openSUSE:/Factory:/Update/
> standard/

I did, they're the same packages as from the snapshot 20220524 itself.
Comment 9 Diego Ercolani 2022-05-26 15:36:10 UTC
It would be useful to rollback the qemu version
Comment 10 Paul Fee 2022-05-26 16:11:46 UTC
Version that breaks: qemu-x86-6.2.0-48.1.x86_64.rpm

$ sudo virsh start VM1
error: Failed to start domain 'VM1'
error: internal error: qemu unexpectedly closed the monitor: qxl_send_events: spice-server bug: guest stopped, ignoring
*** buffer overflow detected ***: terminated

Downgrading just the qemu-x86 package to the previous version works as temporary fix:

http://download.opensuse.org/tumbleweed/repo/oss/x86_64/qemu-x86-6.2.0-47.2.x86_64.rpm

$ sudo zypper in --oldpackage ./qemu-x86-6.2.0-47.2.x86_64.rpm
$ sudo virsh start VM1
Domain 'VM1' started
Comment 11 Nik Kai 2022-05-26 18:57:45 UTC
If it helps, I also have an error that prevents me from launching my VM after the latest update.

Fehler beim Starten der Domain: Interner Fehler: qemu unexpectedly closed the monitor: *** buffer overflow detected ***: terminated

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 72, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 108, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1407, in startup
    self._backend.create()
  File "/usr/lib64/python3.8/site-packages/libvirt.py", line 1352, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: Interner Fehler: qemu unexpectedly closed the monitor: *** buffer overflow detected ***: terminated


A lot of things went wrong this snapshot and the last one. Spectacle doesnt launch, fonts in KDE are blurry because of wrong hinting with Noto Sans and now this. Whats going on..? :)
Comment 12 Dominique Leuenberger 2022-05-26 19:08:28 UTC
Can you guys try qemu-6.2.0-48.2 from the update channel please?
Comment 13 Dominique Leuenberger 2022-05-26 19:09:23 UTC
(In reply to Dominique Leuenberger from comment #12)
> Can you guys try qemu-6.2.0-48.2 from the update channel please?

Together with the other qemu packages from the update repo, all same version of course
Comment 14 Mathias Homann 2022-05-26 19:16:05 UTC
(In reply to Dominique Leuenberger from comment #13)
> (In reply to Dominique Leuenberger from comment #12)
> > Can you guys try qemu-6.2.0-48.2 from the update channel please?
> 
> Together with the other qemu packages from the update repo, all same version
> of course

still the same error.
Comment 15 Dominique Leuenberger 2022-05-26 19:29:34 UTC
(In reply to Mathias Homann from comment #14)
> (In reply to Dominique Leuenberger from comment #13)
> > (In reply to Dominique Leuenberger from comment #12)
> > > Can you guys try qemu-6.2.0-48.2 from the update channel please?
> > 
> > Together with the other qemu packages from the update repo, all same version
> > of course
> 
> still the same error.

That's bad - it means a revert of qemu does not fix the issue and it's rather something that came in through the backdoor and with a rebuild ...

So anything that changed between may 17 (last build of qemu that worked) and may 24, when a new qemu patch was added. But the old sources would have caused the same issue in rebuild only

Based on an apparent unrelated bug, my best guess is gnutls.

A testbuild to confirm this theory should appear soon in
https://download.opensuse.org/repositories/home:/dimstar:/boo1199924/standard
Comment 16 Dominique Leuenberger 2022-05-26 19:33:18 UTC
Build can be monitored at https://build.opensuse.org/project/show/home:dimstar:boo1199924
Comment 17 Dario Faggioli 2022-05-27 06:37:06 UTC
(In reply to Dominique Leuenberger from comment #15)
> (In reply to Mathias Homann from comment #14)
> > (In reply to Dominique Leuenberger from comment #13)
> > > (In reply to Dominique Leuenberger from comment #12)
> > > > Can you guys try qemu-6.2.0-48.2 from the update channel please?
> > > 
> > > Together with the other qemu packages from the update repo, all same version
> > > of course
> > 
> > still the same error.
> 
> That's bad - it means a revert of qemu does not fix the issue and it's
> rather something that came in through the backdoor and with a rebuild ...
> 
Yeah, I have the same feeling.

> So anything that changed between may 17 (last build of qemu that worked) and
> may 24, when a new qemu patch was added. But the old sources would have
> caused the same issue in rebuild only
>
I think GCC 12 happened within that window, didn't it?

> Based on an apparent unrelated bug, my best guess is gnutls.
> 
> A testbuild to confirm this theory should appear soon in
> https://download.opensuse.org/repositories/home:/dimstar:/boo1199924/standard
>
Right, good guess (FWIW), it could very well be it...
Comment 18 Dario Faggioli 2022-05-27 06:41:29 UTC
(In reply to Dominique Leuenberger from comment #15)
> That's bad - it means a revert of qemu does not fix the issue and it's
> rather something that came in through the backdoor and with a rebuild ...
> 
In fact...

(In reply to Dominique Leuenberger from comment #0)
> 2022-05-25T16:30:05.742354Z qemu-system-x86_64: warning: host doesn't
> support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
> 2022-05-25T16:30:05.742369Z qemu-system-x86_64: warning: host doesn't
> support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
> 2022-05-25T16:30:05.743989Z qemu-system-x86_64: warning: host doesn't
> support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
> 2022-05-25T16:30:05.744050Z qemu-system-x86_64: warning: host doesn't
> support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
> *** buffer overflow detected ***: terminated
> 
I've seen reports of various packages starting to fail to start with this exact same error (most recent one I've heard about is GNOME Calculator)
Comment 19 Dominique Leuenberger 2022-05-27 06:41:57 UTC
(In reply to Dario Faggioli from comment #17)
> (In reply to Dominique Leuenberger from comment #15)
> > So anything that changed between may 17 (last build of qemu that worked) and
> > may 24, when a new qemu patch was added. But the old sources would have
> > caused the same issue in rebuild only
> >
> I think GCC 12 happened within that window, didn't it?

GCC 12 happened may 10th, qemu worked for two weeks post that
Comment 20 Dominique Leuenberger 2022-05-27 06:43:35 UTC
(In reply to Dario Faggioli from comment #18)
> (In reply to Dominique Leuenberger from comment #15)
> > That's bad - it means a revert of qemu does not fix the issue and it's
> > rather something that came in through the backdoor and with a rebuild ...
> > 
> In fact...
> 
> I've seen reports of various packages starting to fail to start with this
> exact same error (most recent one I've heard about is GNOME Calculator)

Yep, that's http://bugzilla.opensuse.org/show_bug.cgi?id=1199929 which also points at gnutls.
Comment 21 Dario Faggioli 2022-05-27 06:45:03 UTC
(In reply to Dominique Leuenberger from comment #19)
> (In reply to Dario Faggioli from comment #17)
> > (In reply to Dominique Leuenberger from comment #15)
> > > So anything that changed between may 17 (last build of qemu that worked) and
> > > may 24, when a new qemu patch was added. But the old sources would have
> > > caused the same issue in rebuild only
> > >
> > I think GCC 12 happened within that window, didn't it?
> 
> GCC 12 happened may 10th, qemu worked for two weeks post that
>
Ok, I had the dates mixed up, my bad.

In fact, it does seem that it really could be gnutls, as it's causing similar issues in quite a few other places, e.g.: bug 199929
Comment 22 Dario Faggioli 2022-05-27 06:52:43 UTC
(In reply to Dario Faggioli from comment #21)
> In fact, it does seem that it really could be gnutls, as it's causing
> similar issues in quite a few other places, e.g.: bug 199929
>
Err.. https://bugzilla.opensuse.org/show_bug.cgi?id=1199929 (not sure why it was converted to that url... :-O)
Comment 23 Dario Faggioli 2022-05-27 06:53:35 UTC
(In reply to Dario Faggioli from comment #22)
> Err.. https://bugzilla.opensuse.org/show_bug.cgi?id=1199929 (not sure why it
> was converted to that url... :-O)
>
Ah, ok, see it now. Missing a "1". Well, sorry for the noise (and quite a bit of it :-( )
Comment 24 Dominique Leuenberger 2022-05-27 07:28:11 UTC
*** Bug 1199967 has been marked as a duplicate of this bug. ***
Comment 25 Felix Niederwanger 2022-05-27 08:03:40 UTC
I just tested https://download.opensuse.org/repositories/home:/dimstar:/boo1199924/standard/ and on my laptop the issue is still present. All packages from this repo are installed (qemu-* and gnutls).

> phoenix@racetrack-7290:~> virsh start tumbleweed
> error: Failed to start domain 'tumbleweed'
> error: internal error: qemu unexpectedly closed the monitor: qxl_send_events: spice-server bug: guest stopped, ignoring
> *** buffer overflow detected ***: terminated

AFAICS this build does not resolve it.
Comment 26 Oliver Schwabedissen 2022-05-27 08:28:23 UTC
Sure it's (only) gnutls? Reverting back to snapshot 20220523 using tumbleweed-cli didn't work for me, vms still didn't start.
So I rolled back to a snapshot recorded 05/23/2022 using snapper and got snapshot 20220521. I locked all gnutls packages (gnutls, libgnutls30 and libgnutls-dane0, all at version 3.7.4-2.1), then called "zypper dup" and updated to snapshot 20220525. But again vms didn't start, although gnutls was still at 3.7.4.

So I used snapper again to roll back to snapshot 20220521. vms start and also Spectacle works. Guess I will stay at this snapshot until most problems are sorted out.
Comment 27 Dario Faggioli 2022-05-27 08:34:27 UTC
(In reply to Oliver Schwabedissen from comment #26)
> So I used snapper again to roll back to snapshot 20220521. vms start and
> also Spectacle works. Guess I will stay at this snapshot until most problems
> are sorted out.
>
Mmm... yes, Spectacle was also broken, apparently, but that was, AFAIUI, a different issue (see bug 1199931).

Can you, by any chance try GNOME Calculator? (or another app that was suffering from the "*** buffer overflow detected ***" issue... GNOME Calculator is just one that I'm sure it does, and that might be fairly quick to install and test, I guess)
Comment 28 Mathias Homann 2022-05-27 08:42:42 UTC
(In reply to Oliver Schwabedissen from comment #26)
> Sure it's (only) gnutls? Reverting back to snapshot 20220523 using
> tumbleweed-cli didn't work for me, vms still didn't start.


ok now that is ...weird. For me, rolling back to 20220523 is what makes VMs work again - and replacing the qemu-x86 package with the one from 20220523 is actually the only thing that matters...

Right now I am fully on 20220525 except for that one package which i got from 20220523 and locked.
VMs work fine.
Comment 29 Dario Faggioli 2022-05-27 09:02:31 UTC
(In reply to Mathias Homann from comment #28)
> (In reply to Oliver Schwabedissen from comment #26)
> > Sure it's (only) gnutls? Reverting back to snapshot 20220523 using
> > tumbleweed-cli didn't work for me, vms still didn't start.
> 
> 
> ok now that is ...weird. For me, rolling back to 20220523 is what makes VMs
> work again - and replacing the qemu-x86 package with the one from 20220523
> is actually the only thing that matters...
> 
> Right now I am fully on 20220525 except for that one package which i got
> from 20220523 and locked.
> VMs work fine.
>
Mmm... can I see a:

> rpm -q --changelog qemu|head -20

?
Comment 30 Dominique Leuenberger 2022-05-27 09:16:42 UTC
So, the problem we see is qemu, same source, that built on may 17 worked, built on may 24 fails.

FORTIFY_SOURCE=3 has happened in this timeframe, which might be a source here
Comment 31 Dario Faggioli 2022-05-27 09:20:54 UTC
(In reply to Dario Faggioli from comment #29)
> (In reply to Mathias Homann from comment #28)
> > (In reply to Oliver Schwabedissen from comment #26)
> > > Sure it's (only) gnutls? Reverting back to snapshot 20220523 using
> > > tumbleweed-cli didn't work for me, vms still didn't start.
> > 
> > 
> > ok now that is ...weird. For me, rolling back to 20220523 is what makes VMs
> > work again - and replacing the qemu-x86 package with the one from 20220523
> > is actually the only thing that matters...
> > 
> > Right now I am fully on 20220525 except for that one package which i got
> > from 20220523 and locked.
> > VMs work fine.
> >
> Mmm... can I see a:
> 
> > rpm -q --changelog qemu|head -20
> 
> ?
Ok, I checked myself.

20220525, which you say does not work, has qemu-6.2.0-47.2.x86_64.rpm. Last change was:

> * Wed May 11 2022 Martin Liška <mliska@suse.cz>
> - Filter out rpmlint error that is valid for qemu, but will
>   have its badness increased in the future.

Now, 20220523, which apparently is working, has, as far as I can tell, qemu-6.2.0-47.2.x86_64.rpm, i.e., _the_same_ package. Last change was:

> * Wed May 11 2022 Martin Liška <mliska@suse.cz>
> - Filter out rpmlint error that is valid for qemu, but will
>   have its badness increased in the future.

So, the same change.

Can you confirm the versions?
Comment 32 Dario Faggioli 2022-05-27 09:22:18 UTC
(In reply to Dominique Leuenberger from comment #30)
> So, the problem we see is qemu, same source, that built on may 17 worked,
> built on may 24 fails.
> 
> FORTIFY_SOURCE=3 has happened in this timeframe, which might be a source here
>
If someone tells me how to turn this off (or back to something else), I can try a build
Comment 33 Dominique Leuenberger 2022-05-27 09:24:19 UTC
I set my branch to fortify source= 2, qemu is rebuilding - rel 50.3, once ready, should be worth a test again
Comment 34 Paul Fee 2022-05-27 09:28:29 UTC
> Can you confirm the versions?

Works: 6.2.0-47.2
Fails: 6.2.0-48.1
Fails: 6.2.0-48.2

Tumbleweed fully updated (zypper dup) to openSUSE-release-20220525-1634.1.x86_64.

Only the qemu-x86 needs downgraded to get VMs working.
Comment 35 Dario Faggioli 2022-05-27 09:30:32 UTC
(In reply to Dominique Leuenberger from comment #33)
> I set my branch to fortify source= 2, qemu is rebuilding - rel 50.3, once
> ready, should be worth a test again
>
Ok, I'm doing the same (but with the new QEMU version) here:

https://build.opensuse.org/project/show/home:dfaggioli:boo1199924

(I've copied the project config from here: https://build.opensuse.org/projects/home:dimstar:boo1199924/prjconf )
Comment 36 Dario Faggioli 2022-05-27 11:00:49 UTC
Ok, packages from here work for me:

https://download.opensuse.org/repositories/home:/dfaggioli:/boo1199924/openSUSE_Tumbleweed/home:dfaggioli:boo1199924.repo

BEWARE that, in there, you'd find QEMU 7.0, not QEMU 6.2. And the reason is I've already submitted that (i.e., QEMU 7.0) to Factory, and, considering that Dominique was already building 6.2 himself anyway, I wanted to check the new one.

So:
- QEMU 7.0, built with FORTIFY_SOURCE=2
- GNUtls 3.7.4

Seems fine to me.
Comment 37 Felix Niederwanger 2022-05-27 11:41:50 UTC
Can confirm, Dario's repo with qemu 7 works also fine here on my side.
Comment 38 Dario Faggioli 2022-05-27 11:51:10 UTC
@Dominique, in your project, I think I still see -D_FORTIFY_SOURCE=3, in the latest available build:

https://build.opensuse.org/package/live_build_log/home:dimstar:boo1199924/qemu:testsuite/standard/x86_64
https://build.opensuse.org/build/home:dimstar:boo1199924/standard/x86_64/qemu:testsuite/_log

> [  171s] + /home/abuild/rpmbuild/BUILD/qemu-6.2.0/configure --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --docdir=/usr/share/doc/packages --firmwarepath=/usr/share/qemu-testsuite --python=/usr/bin/python3 '--extra-cflags=-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables [...]

For mine, I re-triggered the build manually, after changing the prjconf. Do you maybe need to do the same?
Comment 39 Dario Faggioli 2022-05-27 12:20:33 UTC
In any case, I think we're coming to the conclusion that FORTIFY=3 has problems, and that FORTIFY=2 could be a solution.

So, what's the way forward here? Will Factory be reverted to 2 entirely? Will this happen only for the QEMU package?

Specifically, do I need to do anything, in the package and/or in the devel project?
Comment 40 Dominique Leuenberger 2022-05-27 12:26:22 UTC
osc rbl home:dimstar:boo1199924 qemu standard x86_64 | grep FORTIFY looks good

That build resulted in 6.2.0-50.3 which should be working. Any Testers?

As we seem to be on track there we will need this as part of the qemu.spec instead of the prjconf:

The fortify_source=2 should be injected into the extra cflags passed to configure
Comment 41 Paul Fee 2022-05-27 12:45:31 UTC
> That build resulted in 6.2.0-50.3 which should be working. Any Testers?

Downloaded https://download.opensuse.org/repositories/home:/dimstar:/boo1199924/standard/x86_64/qemu-x86-6.2.0-50.3.x86_64.rpm

Works for me.

$ sudo virsh start VM1
Domain 'VM1' started

Works: 6.2.0-47.2
Fails: 6.2.0-48.1
Fails: 6.2.0-48.2
Works: 6.2.0-50.3 (from home:dimstar)
Comment 42 Diego Ercolani 2022-05-27 12:51:25 UTC
From the qemu download page:
Version numbering

Since version 3.0.0, QEMU uses a time based version numbering scheme:

major
    incremented by 1 for the first release of the year
minor
    reset to 0 with every major increment, otherwise incremented by 1 for each release from git master
micro
    always 0 for releases from git master, incremented by 1 for each stable branch release 

So I think should be enough to go to version 7 as we are in the middle of 2022 (version 7.x.y branch)
Comment 43 Dominique Leuenberger 2022-05-27 12:54:42 UTC
(In reply to Paul Fee from comment #41
> Works: 6.2.0-47.2
> Fails: 6.2.0-48.1
> Fails: 6.2.0-48.2
> Works: 6.2.0-50.3 (from home:dimstar)

Ok, thanks! With this we an say conclusively that fortify 3 is the culprit for qemu.

I injected fortify 2 into openSUSE:Factory:Update - qemu packages in there should fall out in 1-2hours and work too, without home repos 

The gnutls error from gnome-calculator only put us on the wrong path as it seems:

Dario: can you please prepare a 6.2 submission with only the cflags adjusted, do i can take that into snapshot 0527 without staging blocks?
Comment 44 Dario Faggioli 2022-05-27 12:57:14 UTC
(In reply to Dominique Leuenberger from comment #43)
> Dario: can you please prepare a 6.2 submission with only the cflags
> adjusted, do i can take that into snapshot 0527 without staging blocks?>
>
Working on it!
Comment 45 Dario Faggioli 2022-05-27 14:14:27 UTC
(In reply to Dario Faggioli from comment #44)
> (In reply to Dominique Leuenberger from comment #43)
> > Dario: can you please prepare a 6.2 submission with only the cflags
> > adjusted, do i can take that into snapshot 0527 without staging blocks?>
> >
> Working on it!
>
@Dominiqu, does this make sense?

https://build.opensuse.org/request/show/979490
Comment 46 Dario Faggioli 2022-05-27 14:19:44 UTC
(In reply to Dominique Leuenberger from comment #43) 
> The gnutls error from gnome-calculator only put us on the wrong path as it
> seems:
> 
Right. Symptoms were the same there too (in GNOME Calculator, I mean), though... Are we sure it's gnutls and not FORTIFY_SOURCES in that as well?

Well, anyway, that's off-topic here...

> Dario: can you please prepare a 6.2 submission with only the cflags
> adjusted, do i can take that into snapshot 0527 without staging blocks?
>
Done. Actually, I reinstated the ARM build fix as well, the one that you reverted,  when thinking it was the problem.

This was in released snapshot and working well already, so I think it's not a problem.
Comment 47 Nik Kai 2022-05-27 14:31:31 UTC
Just wanted to confirm that 6.2.0-48.3 from today's update solved the issue. No more buffer overflow.
Thank you all! 

virsh start win10
Domain 'win10' started
Comment 48 Dominique Leuenberger 2022-05-27 14:38:45 UTC
(In reply to Dario Faggioli from comment #45)
> (In reply to Dario Faggioli from comment #44)
> > (In reply to Dominique Leuenberger from comment #43)
> > > Dario: can you please prepare a 6.2 submission with only the cflags
> > > adjusted, do i can take that into snapshot 0527 without staging blocks?>
> > >
> > Working on it!
> >
> @Dominiqu, does this make sense?
> 
> https://build.opensuse.org/request/show/979490

To me that looks fine. And yes, the arm fix can go back in. It just seemed most obvious that the PKG change might be responsible. We have proven this to be wrong by now
Comment 49 Mathias Homann 2022-05-27 14:42:54 UTC
(In reply to Nik Kai from comment #47)
> Just wanted to confirm that 6.2.0-48.3 from today's update solved the issue.
> No more buffer overflow.
> Thank you all! 
> 
> virsh start win10
> Domain 'win10' started

same here, all is well with 6.2.0-48.3
Comment 50 Dario Faggioli 2022-05-27 14:51:41 UTC
(In reply to Dominique Leuenberger from comment #48)
> (In reply to Dario Faggioli from comment #45)
> > 
> > https://build.opensuse.org/request/show/979490
> 
> It just seemed
> most obvious that the PKG change might be responsible. We have proven this
> to be wrong by now
>
Sure! I wasn't blaming or complaining (sorry in case it sounded like that!).

However strange it looked, the decision of reverting it was the one that made the most sense (given the data coming from this bug). :-)

Happy that it they can go in at once, so we don't have to transition toward the "aarch64 is broken" again :-D

For everyone, once this is released, please, let us know here if things still work, so we can close the bug.
Comment 51 Paul Fee 2022-05-27 15:14:22 UTC
(In reply to Dario Faggioli from comment #50)
> 
> For everyone, once this is released, please, let us know here if things
> still work, so we can close the bug.

$ sudo zypper dup
$ rpm -q qemu-x86
qemu-x86-6.2.0-48.3.x86_64
$ sudo virsh start VM1
Domain 'VM1' started

Working as expected using latest Tumbleweed official package.
Comment 52 Dario Faggioli 2022-05-27 17:00:06 UTC
(In reply to Paul Fee from comment #51)
> (In reply to Dario Faggioli from comment #50)
> > 
> > For everyone, once this is released, please, let us know here if things
> > still work, so we can close the bug.
> 
> $ sudo zypper dup
> $ rpm -q qemu-x86
> qemu-x86-6.2.0-48.3.x86_64
> $ sudo virsh start VM1
> Domain 'VM1' started
> 
> Working as expected using latest Tumbleweed official package.
>
Wow... Already?

Ok, perfect, closing it them.

Thanks everyone for all the information and all the testing!
Comment 53 Paul Fee 2022-05-30 08:56:04 UTC
Final testing now that extra updates have arrived...

Full update to openSUSE-release-20220527-1638.1 via "sudo zypper dup".

Now running qemu-x86-6.2.0-50.1.

$ rpm -q qemu-x86 --changelog | head
* Fri May 27 2022 Dario Faggioli <dfaggioli@suse.com>
- It has been observed that building QEMU with _FORTIFY_SOURCE=3
  causes problem (see bsc#1199924). Force it to =2 for now, while
  we investigate the issue.

VMs start up as expected without issue.
Comment 54 Martin Liška 2022-05-30 09:17:00 UTC
(In reply to Dario Faggioli from comment #52)
> (In reply to Paul Fee from comment #51)
> > (In reply to Dario Faggioli from comment #50)
> > > 
> > > For everyone, once this is released, please, let us know here if things
> > > still work, so we can close the bug.
> > 
> > $ sudo zypper dup
> > $ rpm -q qemu-x86
> > qemu-x86-6.2.0-48.3.x86_64
> > $ sudo virsh start VM1
> > Domain 'VM1' started
> > 
> > Working as expected using latest Tumbleweed official package.
> >
> Wow... Already?
> 
> Ok, perfect, closing it them.
> 
> Thanks everyone for all the information and all the testing!

May I please ask where will you track the debugging attempt as -D_FORTIFY_SOURCE=3 very likely exposes a real issue in qemu?
Comment 55 Dario Faggioli 2022-05-30 10:09:54 UTC
(In reply to Martin Liška from comment #54)
> May I please ask where will you track the debugging attempt as
> -D_FORTIFY_SOURCE=3 very likely exposes a real issue in qemu?
>
Right. We've added it to our team backlog, but it's not something we have the time/manpower to deal with right away.

In any case, it would be best if this is addressed upstream. In fact, I've QEMU's community about it:
https://lore.kernel.org/qemu-devel/6a6dbfb53f2ea5a9740249c2fdf480be183e6ee8.camel@suse.com/

Let's see if anyone picks it up/has time for helping us with that.
Comment 56 Martin Liška 2022-05-30 10:20:22 UTC
(In reply to Dario Faggioli from comment #55)
> (In reply to Martin Liška from comment #54)
> > May I please ask where will you track the debugging attempt as
> > -D_FORTIFY_SOURCE=3 very likely exposes a real issue in qemu?
> >
> Right. We've added it to our team backlog, but it's not something we have
> the time/manpower to deal with right away.

I see.

> 
> In any case, it would be best if this is addressed upstream. In fact, I've
> QEMU's community about it:
> https://lore.kernel.org/qemu-devel/6a6dbfb53f2ea5a9740249c2fdf480be183e6ee8.
> camel@suse.com/

You did well. One question: would it be possible to get a backtrace from the abort. Note the 'buffer overflow detected' write to stderr is following by call of abort and a backtrace would really help us?

> 
> Let's see if anyone picks it up/has time for helping us with that.
Comment 57 Martin Liška 2022-05-30 12:26:42 UTC
Btw. the issue happens here:

Starting program: /usr/bin/qemu-system-x86_64 -name guest=opensuse-factory,debug-threads=on -S -machine pc-q35-6.2,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram -accel kvm -cpu host,migratable=on -m 2048 -object \{\"qom-type\":\"memory-backend-ram\",\"id\":\"pc.ram\",\"size\":2147483648\} -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid b1375f79-0900-4ee9-b033-c7fd4ba6a748 -no-user-config -nodefaults -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=19,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=20,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=21,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=22,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 -device pcie-root-port,port=23,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 -device pcie-root-port,port=24,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x3 -device pcie-root-port,port=25,chassis=10,id=pci.10,bus=pcie.0,addr=0x3.0x1 -device pcie-root-port,port=26,chassis=11,id=pci.11,bus=pcie.0,addr=0x3.0x2 -device pcie-root-port,port=27,chassis=12,id=pci.12,bus=pcie.0,addr=0x3.0x3 -device pcie-root-port,port=28,chassis=13,id=pci.13,bus=pcie.0,addr=0x3.0x4 -device pcie-root-port,port=29,chassis=14,id=pci.14,bus=pcie.0,addr=0x3.0x5 -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 -device virtio-scsi-pci,id=scsi0,bus=pci.6,addr=0x0 -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 -blockdev \{\"driver\":\"file\",\"filename\":\"/var/lib/libvirt/images/opensuse-factory.qcow2\",\"node-name\":\"libvirt-1-storage\",\"auto-read-only\":true,\"discard\":\"unmap\"\} -blockdev \{\"node-name\":\"libvirt-1-format\",\"read-only\":false,\"discard\":\"unmap\",\"driver\":\"qcow2\",\"file\":\"libvirt-1-storage\",\"backing\":null\} -device ide-hd,bus=ide.0,drive=libvirt-1-format,id=sata0-0-0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0,index=0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -audiodev \{\"id\":\"audio1\",\"driver\":\"spice\"\} -spice port=5901,addr=127.0.0.1,disable-ticketing=on,seamless-migration=on -device virtio-vga,id=video0,max_outputs=1,bus=pcie.0,addr=0x1 -device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0,audiodev=audio1 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0 -object \{\"qom-type\":\"rng-random\",\"id\":\"objrng0\",\"filename\":\"/dev/urandom\"\} -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.5,addr=0x0 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
...

*** buffer overflow detected ***: terminated

Thread 1 "qemu-system-x86" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff642c380 (LWP 121307)]
0x00007ffff71ff55c in __pthread_kill_implementation () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff71ff55c in __pthread_kill_implementation () at /lib64/libc.so.6
#1  0x00007ffff71ac6f6 in raise () at /lib64/libc.so.6
#2  0x00007ffff7195814 in abort () at /lib64/libc.so.6
#3  0x00007ffff71f279e in __libc_message () at /lib64/libc.so.6
#4  0x00007ffff729767a in __fortify_fail () at /lib64/libc.so.6
#5  0x00007ffff7295c36 in  () at /lib64/libc.so.6
#6  0x00007ffff72957f5 in __snprintf_chk () at /lib64/libc.so.6
#7  0x0000555555b1c1fd in pcibus_get_fw_dev_path ()
#8  0x0000555555f2bde4 in qdev_get_fw_dev_path_helper.constprop ()
#9  0x0000555555f2bd86 in qdev_get_fw_dev_path_helper.constprop ()
#10 0x00005555559a6e5d in get_boot_device_path ()
#11 0x00005555559a712c in get_boot_devices_list ()
#12 0x0000555555b1a3d0 in fw_cfg_machine_reset ()
#13 0x0000555555bf4c2d in pc_machine_reset ()
#14 0x0000555555c66988 in qemu_system_reset ()
#15 0x0000555555a6dff6 in qdev_machine_creation_done ()
#16 0x0000555555c79186 in qmp_x_exit_preconfig.part ()
#17 0x0000555555c7b459 in qemu_init ()
#18 0x0000555555960a29 in main ()

I have a patch for that.
Comment 58 Martin Liška 2022-05-30 12:34:41 UTC
The abort comes from the following line:

https://gitlab.com/qemu-project/qemu/-/blob/master/hw/pci/pci.c#L2650

        snprintf(path + off, sizeof(path) + off, ",%x", PCI_FUNC(d->devfn));

where the second argument should be sizeof(path) - off
Comment 59 Martin Liška 2022-05-30 12:48:27 UTC
Let's come up with a proper fix, reopenning.
Comment 60 Claudio Fontana 2022-05-31 14:28:19 UTC
I cooked up an upstream patch, queued now by the maintainer,

https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg06183.html

Thanks Martin, Dario for the find!
Comment 62 Martin Liška 2022-06-15 08:12:42 UTC
(In reply to Claudio Fontana from comment #60)
> I cooked up an upstream patch, queued now by the maintainer,
> 
> https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg06183.html
> 
> Thanks Martin, Dario for the find!

Great, can you please cherry-pick that to Virtualization/qemu package and push it to Factory?
Comment 63 Martin Liška 2022-06-21 20:10:46 UTC
Fixed in devel project.
Comment 64 Swamp Workflow Management 2022-07-04 13:20:01 UTC
SUSE-SU-2022:2254-1: An update that solves three vulnerabilities and has three fixes is now available.

Category: security (important)
Bug References: 1197084,1198035,1198037,1198712,1199018,1199924
CVE References: CVE-2021-4206,CVE-2021-4207,CVE-2022-26354
JIRA References: 
Sources used:
openSUSE Leap 15.3 (src):    qemu-5.2.0-150300.115.2, qemu-linux-user-5.2.0-150300.115.2, qemu-testsuite-5.2.0-150300.115.4
SUSE Linux Enterprise Module for Server Applications 15-SP3 (src):    qemu-5.2.0-150300.115.2
SUSE Linux Enterprise Module for Basesystem 15-SP3 (src):    qemu-5.2.0-150300.115.2
SUSE Linux Enterprise Micro 5.2 (src):    qemu-5.2.0-150300.115.2
SUSE Linux Enterprise Micro 5.1 (src):    qemu-5.2.0-150300.115.2

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 65 Swamp Workflow Management 2022-07-04 19:17:37 UTC
SUSE-SU-2022:2260-1: An update that solves four vulnerabilities and has 5 fixes is now available.

Category: security (important)
Bug References: 1197084,1198035,1198037,1198711,1198712,1199015,1199018,1199625,1199924
CVE References: CVE-2021-4206,CVE-2021-4207,CVE-2022-26353,CVE-2022-26354
JIRA References: 
Sources used:
openSUSE Leap 15.4 (src):    qemu-6.2.0-150400.37.5.3, qemu-linux-user-6.2.0-150400.37.5.1, qemu-testsuite-6.2.0-150400.37.5.5
SUSE Linux Enterprise Module for Server Applications 15-SP4 (src):    qemu-6.2.0-150400.37.5.3
SUSE Linux Enterprise Module for Basesystem 15-SP4 (src):    qemu-6.2.0-150400.37.5.3

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.