Bug 1075622 - Libvirt cannot control Xen installed on the same machine
Libvirt cannot control Xen installed on the same machine
Status: RESOLVED FIXED
: 1076754 (view as bug list)
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Virtualization:Other
Current
x86-64 Other
: P2 - High : Major (vote)
: ---
Assigned To: Charles Arnold
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-01-11 16:03 UTC by Gabriel Canhete
Modified: 2019-07-12 14:19 UTC (History)
6 users (show)

See Also:
Found By: Customer
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
tentative fix (1.03 KB, patch)
2018-01-11 16:47 UTC, Cédric Bosdonnat
Details | Diff
Logs from coredumpctl (31.65 KB, text/plain)
2018-01-12 14:22 UTC, Gabriel Canhete
Details
Logs from journalctl (2.05 KB, text/plain)
2018-01-12 14:22 UTC, Gabriel Canhete
Details
output in stdout from coredumpctl (757 bytes, text/plain)
2018-01-12 14:26 UTC, Gabriel Canhete
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Gabriel Canhete 2018-01-11 16:03:20 UTC
When you boot to Xen, any libvirt operations involving the hypervisor on the same machine will fail with the following error (as seen on virt-manager):

Cannot connect to libvirt

local variable 'exc' referenced before assignment

Check this:
- A Xen kernel has been initialized
- The Xen service has been initialized

Libvirt URI is: xen:///

Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/connection.py", line 1116, in _open_thread
is_active, connectError = self._do_open()
File "/usr/share/virt-manager/virtManager/connection.py", line 1064, in _do_open
connectError = (str(exc), tb, warnconsole)
UnboundLocalError: local variable 'exc' referenced before assignment

I know the hypervisor is working because i can run this:
# xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  3875     2     r-----     521.1
Xenstore                                     1    31     1     -b----       0.0

I can also control Xen on a Leap machine through virt-manager+ssh, and everything works as expected.
Comment 1 Cédric Bosdonnat 2018-01-11 16:47:18 UTC
Created attachment 755705 [details]
tentative fix

Could you please check that this patch fixes the issue for you. I wonder if there is only that python3-related error.
Comment 2 Gabriel Canhete 2018-01-11 16:53:58 UTC
Sorry for the ignorance, but how do I apply the patch?
Comment 3 Cédric Bosdonnat 2018-01-11 20:55:41 UTC
(In reply to Gabriel Canhete from comment #2)
> Sorry for the ignorance, but how do I apply the patch?

zypper in patch
cd /usr/share/virt-manager
patch -p1 <path/to/path

You can run patch with --dry-run first to see if it applies cleanly.

Then you can restart you virt-manager to see if it works.
Comment 4 Gabriel Canhete 2018-01-11 21:58:57 UTC
Cannot connect to libvirt.

Fail to connect to socket in '/var/run/libvirt/libvirt-sock': Connection refused

Verify this
 - A Xen host kernel has been initialized
 - The Xen service has been initialized

Libvirt URI is: xen:///

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/connection.py", line 1036, in _do_open
    self._backend.open(self._do_creds_password)
  File "/usr/share/virt-manager/virtinst/connection.py", line 144, in open
    open_flags)
  File "/usr/lib64/python3.6/site-packages/libvirt.py", line 105, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirt.libvirtError: fail to connect to socket in '/var/run/libvirt/libvirt-sock': Connection refused
Comment 5 Gabriel Canhete 2018-01-11 23:27:55 UTC
After applying the patch, it looks like libvirt is crashing instantly. I've tried respawning it with systemctl, but in the end, this keeps happening:

# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: disabled
   Active: failed (Result: signal) since Thu 2018-01-11 21:23:48 -02; 2s ago
     Docs: man:libvirtd(8)
           https://libvirt.org
  Process: 7459 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=killed, signal=ABRT)
 Main PID: 7459 (code=killed, signal=ABRT)

If I'm fast enough, virt-manager shows this error:

Cannot connect to libvirt
cannot receive data: connection closed by other end

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/connection.py", line 1036, in _do_open
    self._backend.open(self._do_creds_password)
  File "/usr/share/virt-manager/virtinst/connection.py", line 144, in open
    open_flags)
  File "/usr/lib64/python3.6/site-packages/libvirt.py", line 105, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirt.libvirtError: cannot receive data: connection closed by other end
Comment 6 Cédric Bosdonnat 2018-01-12 10:04:32 UTC
Could you provide the libvirtd log?

> journalctl -b -u libvirtd

Could you also check if there is anything concerning a libvirtd crash in coredumpctl?

> coredumpctl list

If there is something concerning libvirt, I'ld love to have the backtrace:

> coredumpctl dump /usr/sbin/libvirtd
Comment 7 Gabriel Canhete 2018-01-12 14:22:16 UTC
Created attachment 755890 [details]
Logs from coredumpctl
Comment 8 Gabriel Canhete 2018-01-12 14:22:59 UTC
Created attachment 755891 [details]
Logs from journalctl
Comment 9 Gabriel Canhete 2018-01-12 14:26:52 UTC
Created attachment 755893 [details]
output in stdout from coredumpctl
Comment 10 Gabriel Canhete 2018-01-12 14:28:03 UTC
The main coredump is about 275MiB, so I cant upload it here. I'll put it in a Mega link instead.
Comment 12 jean-paul POZZI 2018-01-15 15:55:22 UTC
Hello,

I get the same problem after a system update.

Regards

JP P
Comment 13 Cédric Bosdonnat 2018-01-15 16:55:09 UTC
Here is the corresponding backtrace:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f68dbd526b1 in __GI_abort () at abort.c:79
#2  0x00007f68dbd496fa in __assert_fail_base (fmt=0x7f68dbe9c458 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x7f68b2894390 "((void)\"application must negotiate with libxl about SIGCHLD\", !(sigchld_saved_action.sa_flags & SA_SIGINFO) && (sigchld_saved_action.sa_handler == SIG_DFL || sigchld_saved_action.sa_handler == SIG_IGN"..., file=file@entry=0x7f68b2894196 "libxl_fork.c", line=line@entry=353, function=function@entry=0x7f68b28945b0 <__PRETTY_FUNCTION__.21175> "sigchld_installhandler_core")
    at assert.c:92
#3  0x00007f68dbd49772 in __GI___assert_fail (
    assertion=assertion@entry=0x7f68b2894390 "((void)\"application must negotiate with libxl about SIGCHLD\", !(sigchld_saved_action.sa_flags & SA_SIGINFO) && (sigchld_saved_action.sa_handler == SIG_DFL || sigchld_saved_action.sa_handler == SIG_IGN"..., file=file@entry=0x7f68b2894196 "libxl_fork.c", line=line@entry=353, function=function@entry=0x7f68b28945b0 <__PRETTY_FUNCTION__.21175> "sigchld_installhandler_core")
    at assert.c:101
#4  0x00007f68b284cd80 in sigchld_installhandler_core () at libxl_fork.c:350
#5  libxl__sigchld_needed (gc=0x7f68a0e43780) at libxl_fork.c:402
#6  0x00007f68b284d18e in libxl_childproc_setmode (ctx=<optimized out>, hooks=hooks@entry=0x7f68b2b12200 <libxl_child_hooks>, user=0x7f68800f8d70) at libxl_fork.c:653
#7  0x00007f68b2ae2895 in libxlStateInitialize (privileged=<optimized out>, callback=<optimized out>, opaque=<optimized out>) at libxl/libxl_driver.c:693
#8  0x00007f68dcb38e95 in virStateInitialize (privileged=true, callback=0x5650020b1480, opaque=0x5650039e49a0) at libvirt.c:770
#9  0x00005650020b14db in ?? ()
#10 0x00007f68dca8f562 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#11 0x00007f68dc0dc558 in start_thread (arg=0x7f68a0e44700) at pthread_create.c:465
#12 0x00007f68dbe136df in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) fra 3
#3  0x00007f68dbd49772 in __GI___assert_fail (
    assertion=assertion@entry=0x7f68b2894390 "((void)\"application must negotiate with libxl about SIGCHLD\", !(sigchld_saved_action.sa_flags & SA_SIGINFO) && (sigchld_saved_action.sa_handler == SIG_DFL || sigchld_saved_action.sa_handler == SIG_IGN"..., file=file@entry=0x7f68b2894196 "libxl_fork.c", line=line@entry=353, function=function@entry=0x7f68b28945b0 <__PRETTY_FUNCTION__.21175> "sigchld_installhandler_core")
    at assert.c:101
101	assert.c: No such file or directory.
Comment 14 Cédric Bosdonnat 2018-01-16 10:05:06 UTC
The libvirt crash is very similar to bug 1074132. I'll let you guys reopen that bug for the xen/libvirtd part. I won't close this one as duplicate though for the virt-manager part of the initial bug.
Comment 15 James Fehlig 2018-01-16 21:26:42 UTC
(In reply to jean-paul POZZI from comment #12)
> I get the same problem after a system update.

By update, do you mean updating your existing 42.3 system, or upgrading a 42.2 system to 42.3? Either way, what versions of xen and libvirt are installed after the update?

And if you upgraded 42.2 to 42.3, please explain the steps used. E.g.

- Applied all updates to 42.2 and rebooted
- Changed all repos to point to 42.3
- zypper dup to upgrade to 42.3
- rebooted to Xen and observed the problem
Comment 16 Charles Arnold 2018-01-19 13:14:49 UTC
*** Bug 1076754 has been marked as a duplicate of this bug. ***
Comment 17 Cédric Bosdonnat 2019-07-12 09:00:31 UTC
Charles, I'll let you decide what to do with that bug we never managed to reproduce (for the virt-manager part of it)
Comment 18 Charles Arnold 2019-07-12 14:19:02 UTC
This bug was written against Tumbleweed (a year and a half ago) when the
version of virt-manager was 1.4.3.
virt-manager is now at version 2.1.0 which has had major rewrites including
the switch to requiring python3 only.

I am closing it as fixed but if there are still problems running on the
current Tumbleweed version feel free to reopen with details included.