Bug 1125841 - AppArmor security driver not enabled for QEMU in libvirt
AppArmor security driver not enabled for QEMU in libvirt
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Virtualization:Tools
Current
x86-64 openSUSE Factory
: P5 - None : Normal (vote)
: ---
Assigned To: James Fehlig
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2019-02-19 01:34 UTC by Martin Kalivoda
Modified: 2019-04-29 02:35 UTC (History)
3 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
Screenshot of important part of strace log (48.95 KB, image/png)
2019-02-20 22:34 UTC, Martin Kalivoda
Details
Audit log from clean VM where libvirtd failed due to this bug (23.95 KB, text/plain)
2019-02-27 01:23 UTC, Martin Kalivoda
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Kalivoda 2019-02-19 01:34:57 UTC
User-Agent:       Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36
Build Identifier: 

Setting
security_driver = "apparmor"
in /etc/libvirt/qemu.conf or enabling apparmor individually per domain leads to error with libvirt 5:
Security driver apparmor not enabled


Reproducible: Always

Steps to Reproduce:
1. Set apparmor as security driver in /etc/libvirt/qemu.conf .
2. Use libvirt to start QEMU domain.
Actual Results:  
Error:
Security driver apparmor not enabled
Failed to initialize security drivers
Initialization of QEMU state driver failed: internal error: Failed to initialize security drivers

Expected Results:  
Domain would start and be confined by AppArmor as it used to be before libvirt 5.
Comment 1 James Fehlig 2019-02-19 22:51:44 UTC
Is apparmor running? E.g. what is the output of 'systemctl status apparmor.service' and 'aa-status'?
Comment 2 Martin Kalivoda 2019-02-19 23:27:02 UTC
(In reply to James Fehlig from comment #1)
> Is apparmor running? E.g. what is the output of 'systemctl status
> apparmor.service' and 'aa-status'?

● apparmor.service - Load AppArmor profiles
   Loaded: loaded (/usr/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
   Active: active (exited) since Tue 2019-02-19 01:07:46 CET; 23h ago
 Main PID: 696 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/apparmor.service

Feb 19 01:07:38 vhost4 systemd[1]: Starting Load AppArmor profiles...
Feb 19 01:07:53 vhost4 apparmor.systemd[696]: Restarting AppArmor
Feb 19 01:07:53 vhost4 apparmor.systemd[696]: Reloading AppArmor profiles
Feb 19 01:07:53 vhost4 apparmor.systemd[696]: Skipped: /etc/apparmor.d/libvirt
Feb 19 01:07:46 vhost4 systemd[1]: Started Load AppArmor profiles.

-------------

apparmor module is loaded.
51 profiles are loaded.
50 profiles are in enforce mode.
(omitted)
1 profiles are in complain mode.
   /usr/lib/systemd/system-generators/lvm2-activation-generator
8 processes have profiles defined.
8 processes are in enforce mode.
(omitted)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

No mentions of libvirt in any of the redacted parts
Comment 3 Martin Kalivoda 2019-02-19 23:34:14 UTC
Well, there are of course libvirtd profiles (libvirtd, libvirtd//qemu_bridge_helper, virt-aa-helper), but not domain profiles (I have previously working profiles in /etc/apparmor.d/libvirt).
Comment 4 Martin Kalivoda 2019-02-19 23:44:49 UTC
It should be noted, that this is quite worrying me for another reason:
comments in /etc/libvirt/qemu.conf say that
# SUSE Note:
# Currently, Apparmor is the default security framework in SUSE
# distros.  If Apparmor is enabled on the host, libvirtd is
# generously confined but users must opt-in to confine qemu
# instances.  Change this to a non-zero value to enable default
# Apparmor confinement of qemu instances.
However, changing the value of security_default_confined to 1 will NOT trigger the error and it will leave everything unconfined. Only reason why I got hit by this is because I am cautious about default and changed the security_driver explicitly.
Comment 5 James Fehlig 2019-02-20 02:04:01 UTC
(In reply to Martin Kalivoda from comment #2)
> (In reply to James Fehlig from comment #1)
> > Is apparmor running? E.g. what is the output of 'systemctl status
> > apparmor.service' and 'aa-status'?
> 
> ● apparmor.service - Load AppArmor profiles
>    Loaded: loaded (/usr/lib/systemd/system/apparmor.service; enabled; vendor
> preset: enabled)
>    Active: active (exited) since Tue 2019-02-19 01:07:46 CET; 23h ago
>  Main PID: 696 (code=exited, status=0/SUCCESS)
>     Tasks: 0 (limit: 4915)
>    Memory: 0B
>    CGroup: /system.slice/apparmor.service
> 
> Feb 19 01:07:38 vhost4 systemd[1]: Starting Load AppArmor profiles...
> Feb 19 01:07:53 vhost4 apparmor.systemd[696]: Restarting AppArmor
> Feb 19 01:07:53 vhost4 apparmor.systemd[696]: Reloading AppArmor profiles
> Feb 19 01:07:53 vhost4 apparmor.systemd[696]: Skipped:
> /etc/apparmor.d/libvirt

Why was this skipped? Do you get any errors parsing the libvirtd profile? E.g. try 'apparmor_parser -r /etc/apparmor.d/usr.sbin.libvirtd' and see if any parsing errors are reported.

> Feb 19 01:07:46 vhost4 systemd[1]: Started Load AppArmor profiles.
> 
> -------------
> 
> apparmor module is loaded.
> 51 profiles are loaded.
> 50 profiles are in enforce mode.
> (omitted)
> 1 profiles are in complain mode.
>    /usr/lib/systemd/system-generators/lvm2-activation-generator
> 8 processes have profiles defined.
> 8 processes are in enforce mode.
> (omitted)
> 0 processes are in complain mode.
> 0 processes are unconfined but have a profile defined.
> 
> No mentions of libvirt in any of the redacted parts

/usr/sbin/libvirtd is not mentioned in the output of 'aa-status'?
Comment 6 James Fehlig 2019-02-20 02:06:38 UTC
(In reply to Martin Kalivoda from comment #3)
> Well, there are of course libvirtd profiles (libvirtd,
> libvirtd//qemu_bridge_helper, virt-aa-helper), but not domain profiles (I
> have previously working profiles in /etc/apparmor.d/libvirt).

Domain profiles in /etc/apparmor.d/libvirt/ are generated when starting a VM and removed when shutting it down.
Comment 7 James Fehlig 2019-02-20 02:09:11 UTC
(In reply to Martin Kalivoda from comment #4)
> It should be noted, that this is quite worrying me for another reason:
> comments in /etc/libvirt/qemu.conf say that
> # SUSE Note:
> # Currently, Apparmor is the default security framework in SUSE
> # distros.  If Apparmor is enabled on the host, libvirtd is
> # generously confined but users must opt-in to confine qemu
> # instances.  Change this to a non-zero value to enable default
> # Apparmor confinement of qemu instances.
> However, changing the value of security_default_confined to 1 will NOT
> trigger the error and it will leave everything unconfined. Only reason why I
> got hit by this is because I am cautious about default and changed the
> security_driver explicitly.

Since the libvirt apparmor driver is not loaded, the default security model is 'none'. Setting security_default_confined has no effect when the active model is a no-op.
Comment 8 Martin Kalivoda 2019-02-20 08:28:54 UTC
(In reply to James Fehlig from comment #6)
> Domain profiles in /etc/apparmor.d/libvirt/ are generated when starting a VM
> and removed when shutting it down.
This does not seems right, because
https://gitlab.com/apparmor/apparmor/wikis/Libvirt#advanced-usage
says, that
> If you need to adjust access controls for a single guest, adjust
/etc/apparmor.d/libvirt-, where  is the UUID of
the guest
Which would be totally useless if it would get removed at shutdown.
It is also mentioned here
https://doc.opensuse.org/documentation/leap/virtualization/html/book.virt/cha.lxc.html#sec.lxc.config.apparmor
Now, checking the source code and when this removal you mention was introduced:
https://github.com/libvirt/libvirt/commit/eba2225bc52624e748cb875e10962bc4c46a0516#diff-8852eb1be9ce9ea8c64fb23af57a0e88
and that it is included also in libvirt 4.0.0 which is in Leap 15 it seem that both documentations are wrong.

> Since the libvirt apparmor driver is not loaded, the default security model
> is 'none'. Setting security_default_confined has no effect when the active
> model is a no-op.
This is unfortuates, because comment makes it sound that this "if apparmor is available" applies only to libvirtd, but qemu instances "will be confined by default" when config is set to nonzero (as the condition does not seem to apply to this part as it is different sentence).

I just installed new Tumbleweed with KVM Host and Virtualization Tools pattern. I changed only security_driver, started libvirt and I get the same error. No other changes were made after installation.
Comment 9 James Fehlig 2019-02-20 16:58:54 UTC
(In reply to Martin Kalivoda from comment #8)
> it seem that both documentations are wrong.

Agreed, but let's stay focused on this bug.

> I just installed new Tumbleweed with KVM Host and Virtualization Tools
> pattern. I changed only security_driver, started libvirt and I get the same
> error. No other changes were made after installation.

I don't see any problems on my Tumbleweed KVM host, with or without apparmor confinement of VMs. Please answer my questions in #5. We need to figure out why the libvirtd profile was not parsed.
Comment 10 Martin Kalivoda 2019-02-20 19:49:07 UTC
(In reply to James Fehlig from comment #9)
> why the libvirtd profile was not parsed.
Libvirtd itself is confined and it's supporting utilities (libvirtd//qemu_bridge_helper, virt-aa-helper) are confined too, as stated in my comment #3 (which corrects #2 - it was poorly worded, because it was unclear whether libvirt means both libvirt-* profiled or also libvirtd - I meant just libvirt-*).
To make it clear:
- libvirtd is confined
- apparmor works fine otherwise
- failure in clean install can be triggered by explicitly setting security_driver to apparmor, nothing more is necessary.

> I don't see any problems on my Tumbleweed KVM host
Do you have updated Tumbleweed with libvirt 5 and did you explicitly set security_driver to apparmor in /etc/libvirt/qemu.conf ?
Comment 11 James Fehlig 2019-02-20 20:23:45 UTC
(In reply to Martin Kalivoda from comment #10)
> Do you have updated Tumbleweed with libvirt 5 and did you explicitly set
> security_driver to apparmor in /etc/libvirt/qemu.conf ?

Yes. I also have 'security_default_confined = 1'. After starting a VM I see its auto-generated profile has been parsed and loaded (along with other libvirt-based profiles)

# aa-status | grep libvirt
   /usr/sbin/dnsmasq//libvirt_leaseshelper
   /usr/sbin/libvirtd
   /usr/sbin/libvirtd//qemu_bridge_helper
   libvirt-66154842-e926-4f92-92f0-1c1bf61dd1ff
   /usr/sbin/libvirtd (21446) 
   /usr/bin/qemu-system-x86_64 (21753) libvirt-66154842-e926-4f92-92f0-1c1bf61dd1ff
Comment 12 Martin Kalivoda 2019-02-20 21:18:09 UTC
(In reply to James Fehlig from comment #11)
> (In reply to Martin Kalivoda from comment #10)
Old profiles in /etc/apparmor.d/libvirt were incorrect (unexpected tokens). I deleted them, but it didn't solve anything... which makes sense, since even clean install does not work for me. Apparently, just setting security_driver to apparmor is not enough to make it work but it used to be enough in near past.
Should it still work? I just did anither install of Tumbleweed, this time into VM and I still get the error after setting security_driver to apparmor.
Comment 13 James Fehlig 2019-02-20 21:49:23 UTC
Enable debug in /etc/libvirt/libvirtd.conf, restart libvirtd, and see if there is any info indicating why the apparmor driver is not loading. E.g. in /etc/libvirt/libvirtd.conf

log_level = 1
log_outputs="1:file:/tmp/libvirtd.log"

restart libvird then look for apparmor hints in /tmp/libvirtd.log
Comment 14 Martin Kalivoda 2019-02-20 22:31:38 UTC
Unfortunately it fails between two lines, not meaningful info is there. In the meantime I have examined the source code.

My problem comes from here
https://github.com/libvirt/libvirt/blob/600462834f4ec1955a9a48a1b6b4a390b9c31553/src/security/security_driver.c#L69
It calls probe on each item of array of externs: security_drivers.

Probe of apparmor driver is called AppArmorSecurityManagerProbe:
https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997dc7c1/src/security/security_apparmor.c#L347
It checks if TEMPLATE.qemu and TEMPLATE.lxc exist (both do as it does not print error). Before that it checks for use_apparmor(), since later checks were not even hit (they log errors and if they passed, then I wouldn't get my error), this must be the culprit.

Now use_apparmor():
https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997dc7c1/src/security/security_apparmor.c#L241
First check (virResolveLink) is ok, because it does not print error. Next I dont' use lxc so no prolem here. Next it can probably access APPARMOR_PROFILES_PATH. Then it checks if libvirtd profile exists and is in enforcing mode by calling profile_status(libvirt_daemon, 1).

Now profile_status(libvirt_daemon, 1):
https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997dc7c1/src/security/security_apparmor.c#L71
Manually evaluating the code seems to be ok. String building should not fail, access to APPARMOR_PROFILES_PATH is ok, because it does not print error. The, libvirtd line is present and "(enforce)" is present next to it. So no problem here either.

So I have started almighty strace and found that it passed readlink test, access test just to fail after
https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997dc7c1/src/security/security_apparmor.c#L90
It must fail after that point, because it didn't log any error.
Unfortunately, this would mean that strstr failed for some reason... I'll try to intercept c-library calls to check it out.
Also, I don't see any way how it could possibly work for you :/
Comment 15 Martin Kalivoda 2019-02-20 22:34:07 UTC
Created attachment 797345 [details]
Screenshot of important part of strace log

Captured on clean Tumbleweed VM
Comment 16 Martin Kalivoda 2019-02-20 23:35:36 UTC
C library functions were inlined by compiler... ltrace is of no use here. But at least I am sure it fails where I though because I have checked that it reads whole /sys/kernel/security/apparmor/profiles and fails right after that.
Comment 17 James Fehlig 2019-02-21 00:44:20 UTC
(In reply to Martin Kalivoda from comment #14)
> So I have started almighty strace and found that it passed readlink test,
> access test just to fail after
> https://github.com/libvirt/libvirt/blob/
> d56afb8e3997ae19fd7449f773065a2b997dc7c1/src/security/security_apparmor.c#L90
> It must fail after that point, because it didn't log any error.
> Unfortunately, this would mean that strstr failed for some reason... I'll
> try to intercept c-library calls to check it out.

I suspect this is a regression caused by my commit to change the libvirtd profile to a named profile

https://libvirt.org/git/?p=libvirt.git;a=commit;h=a3ab6d42d825499af44b8f19f9299e150d9687bc

> Also, I don't see any way how it could possibly work for you :/

I had modified /etc/apparmor.d/usr.sbin/libvirtd so it was not overwritten when I updated to libvirt 5.0.0, so in the end I wasn't using the new named profile :-(.
Comment 18 Martin Kalivoda 2019-02-21 01:07:59 UTC
(In reply to James Fehlig from comment #17)
That must be it! It takes full path to libvirtd here
https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997dc7c1/src/security/security_apparmor.c#L246
and then passes it to profile_status where profile list gets matched against it and matching fails, because profile list contains just profile name.

One posiblity should be to remove this
https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997dc7c1/src/security/security_apparmor.c#L246-L250
and replace this
https://github.com/libvirt/libvirt/blob/d56afb8e3997ae19fd7449f773065a2b997dc7c1/src/security/security_apparmor.c#L260
with
rc = profile_status("libvirtd", 1);
Comment 19 Christian Boltz 2019-02-26 20:58:35 UTC
To make sure we get this right, can you please check /var/log/audit/audit.log for AppArmor denials (or simply attach that file)?
Comment 20 Martin Kalivoda 2019-02-27 01:23:13 UTC
Created attachment 798191 [details]
Audit log from clean VM where libvirtd failed due to this bug

(In reply to Christian Boltz from comment #19)
Comment 21 Christian Boltz 2019-02-27 19:50:14 UTC
(In reply to Martin Kalivoda from comment #20)
> Created attachment 798191 [details]
> Audit log from clean VM where libvirtd failed due to this bug

The log shows that several profiles were loaded, but doesn't show any denials caused by AppArmor. This could mean a) there were no denials ;-) or b) something gets denied by a "deny" rule, which also silences logging.
Comment 22 Martin Kalivoda 2019-02-28 01:22:15 UTC
(In reply to Christian Boltz from comment #21)
This has nothing to do with apparmor itself and I see no reason to expect denials in audit log. As described in #18: This bug is caused by wrong expectations from libvirt - it expects /usr/sbin/libvirtd in profile lists but profile list only contains "libvirtd" because profile was changed to named profile with name "libvirtd" by commit linked in #17. Code linked in #18 then thinks that libvirtd is not confined and thinks that apparmor is not activated which then results in error from original post.
Comment 23 James Fehlig 2019-02-28 04:31:09 UTC
(In reply to Martin Kalivoda from comment #22)
> (In reply to Christian Boltz from comment #21)
> This has nothing to do with apparmor itself and I see no reason to expect
> denials in audit log. As described in #18: This bug is caused by wrong
> expectations from libvirt - it expects /usr/sbin/libvirtd in profile lists
> but profile list only contains "libvirtd" because profile was changed to
> named profile with name "libvirtd" by commit linked in #17. Code linked in
> #18 then thinks that libvirtd is not confined and thinks that apparmor is
> not activated which then results in error from original post.

Yep, it's a libvirt problem. Sorry I haven't had time to create a proper fix. Hopefully before the week is out...
Comment 24 James Fehlig 2019-03-01 23:02:51 UTC
I sent a small patch series upstream to fix this issue and another one I encountered while testing VM confinement with the named libvirtd profile

https://www.redhat.com/archives/libvir-list/2019-March/msg00053.html
Comment 25 James Fehlig 2019-03-06 20:27:58 UTC
(In reply to James Fehlig from comment #24)
> I sent a small patch series upstream to fix this issue and another one I
> encountered while testing VM confinement with the named libvirtd profile
> 
> https://www.redhat.com/archives/libvir-list/2019-March/msg00053.html

I've added these patches to the libvirt 5.1.0 package and submitted to Factory.
Comment 27 Swamp Workflow Management 2019-03-07 16:10:06 UTC
This is an autogenerated message for OBS integration:
This bug (1125841) was mentioned in
https://build.opensuse.org/request/show/682527 Factory / libvirt