Bug 1195463 - smb not starting after patch openSUSE-SLE-15.3-2022-283=1
smb not starting after patch openSUSE-SLE-15.3-2022-283=1
Status: NEW
Classification: openSUSE
Product: openSUSE Distribution
Classification: openSUSE
Component: AppArmor
Leap 15.3
Other Other
: P5 - None : Normal (vote)
: ---
Assigned To: Christian Boltz
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2022-02-02 19:03 UTC by Ralf Kölmel
Modified: 2022-08-02 16:17 UTC (History)
3 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
smb systemd service output and apparmor extract (2.27 KB, text/plain)
2022-02-02 19:03 UTC, Ralf Kölmel
Details
log extracts from audit.log, zypper history and listing of apparmor files v1 (8.54 KB, text/x-log)
2022-02-07 09:35 UTC, Ralf Kölmel
Details
audit log of smb/nmb restart after apparmor cache deletions (1.88 KB, text/plain)
2022-02-07 09:38 UTC, Ralf Kölmel
Details
extract of the audit.log during smb and nmb service restart (1.88 KB, text/x-log)
2022-03-10 12:28 UTC, Ralf Kölmel
Details
tar of apparmor/samba relevant files (642.81 KB, application/octet-stream)
2022-03-24 11:57 UTC, Noel Power
Details
tar of apparmor log (and zypper history) (12.80 KB, application/x-bzip)
2022-03-29 13:41 UTC, Noel Power
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ralf Kölmel 2022-02-02 19:03:57 UTC
Created attachment 855811 [details]
smb systemd service output and apparmor extract

smb service is not starting after applying recent samba patch openSUSE-SLE-15.3-2022-283=1 .
It seems that the changes causes the apparmor "permission" problems.
Comment 1 Christian Boltz 2022-02-02 20:34:20 UTC
This looks like a duplicate of bug 1195412, but the strange thing is - you should have got updated AppArmor packages together with the new Samba packages.

Can you please show the result of   rpm -qa 'apparmor*'   ?
(Expected result: All AppArmor packages should have version 2.13.6.)
Comment 2 Ralf Kölmel 2022-02-02 20:44:19 UTC
the most recent apparmor packages are installed:
rpm -qa 'apparmor*'
apparmor-abstractions-2.13.6-150300.3.11.2.noarch
apparmor-utils-2.13.6-150300.3.11.2.noarch
apparmor-parser-lang-2.13.6-150300.3.11.2.noarch
apparmor-utils-lang-2.13.6-150300.3.11.2.noarch
apparmor-parser-2.13.6-150300.3.11.2.x86_64
apparmor-docs-2.13.6-150300.3.11.2.noarch
apparmor-profiles-2.13.6-150300.3.11.2.noarch
Comment 3 Ralf Kölmel 2022-02-02 20:46:54 UTC
if i stop apparmor (aa-teardown), the smb service is starting.
Comment 4 Christian Boltz 2022-02-04 20:08:16 UTC
Hmm, that's interesting[tm] - in theory it should work with these packages.

The log in your initial bugreport looks like you didn't have the updated profiles yet, therefore a few questions:
- do you still see the same DENIED lines in /var/log/audit/audit.log ?
- can you show the lines for the samba and apparmor packages from
  /var/log/zypp/history ? (The timestamp is relevant for the next question.)
- when the profiles were reloaded during installing the update, does audit.log
  say   operation="profile_replace" info="same as current profile, skipping" 
  or just   operation="profile_replace"   for the   smbd   profile?
  (If it has "same as current profile", there might be something interesting
  with your cache.)
  See [1] below regarding timestamps in audit.log.

Speaking about the cache - I'd also be interested in the result of

ls -l /etc/apparmor.d/usr.sbin.smbd /etc/apparmor.d/local/usr.sbin.smbd* /var/cache/apparmor/*/usr.sbin.smbd

to see the timestamps of your profile, included local files, and the cache files.


With some luck, the output of above ls already explains the problem - my wild guess is that your /etc/apparmor.d/local/usr.sbin.smbd-shares could be newer than the packaged profile, and therefore the cache looks "new enough".



[1]
If you want to search for the audit.log entries at the time of installing the update yourself, you might wonder where the timestamp in audit.log hides - it's at the beginning of the line at msg=audit(1644003177 (as unix timestamp) and   date -d @1644003177   will translate it  to a human-readable date. 

For the other way round (translating the timestamp from zypper.log to a unix timestamp) you can use   date +%s -d '2022-02-04 20:05'

If you don't want to search for the relevant audit.log timestamp, just provide the zypp/history sniplet and attach your full audit.log.
Comment 5 Ralf Kölmel 2022-02-07 09:35:38 UTC
Created attachment 855932 [details]
log extracts from audit.log, zypper history and listing of apparmor files v1

I've provided the requested infos in the attached file.
It seems so that your "guess" is true and the profiles were not replaced.
For now i have deleted the apparmor cache directories which solved the smb start problem, but the DENIED- messages are still present.
Comment 6 Ralf Kölmel 2022-02-07 09:38:13 UTC
Created attachment 855934 [details]
audit log of smb/nmb restart after apparmor cache deletions

also after apparmor cache deletions the same apparmor DENIED messages are still present
Comment 7 Ralf Kölmel 2022-02-07 09:55:20 UTC
(In reply to Ralf Kölmel from comment #6)
> Created attachment 855934 [details]
> audit log of smb/nmb restart after apparmor cache deletions
> 
> also after apparmor cache deletions the same apparmor DENIED messages are
> still present
one correction:
the apparmor DENIED messages are not totally the same, the messages for "samba-bgqd" are different and the message in the smb log "  exit_daemon: daemon failed to start: Samba failed to init printing subsystem, error code 13" on affected systems
Comment 8 Christian Boltz 2022-03-09 22:08:18 UTC
From the log in the original report:
> type=AVC msg=audit(1643828237.305:807): apparmor="DENIED" operation="open" profile="smbd" name="/etc/ssl/openssl.cnf" pid=6144 comm="smbd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

IIRC I've never seen Samba trying to read the openssl.cnf (but I have to admit that I use it only rarely). Do you have a special/unusual samba config that could explain it?

@Noel: if you have an idea, feel free to answer as well ;-)

> type=AVC msg=audit(1643828237.385:809): apparmor="DENIED" operation="exec" profile="smbd" name="/usr/lib64/samba/samba-bgqd" pid=6148 comm="smbd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0

I assume that was fixed by deleting the cache?

Also, if you see other DENIED events, please tell me.


Unfortunately I'm still not sure why you needed to delete the cache.

The "same as current profile, skipping" log entries from comment 5 happened at the same time as the updated apparmor-profiles package was installed. This means AppArmor thought the cache was new enough. While this happens on the kernel side, it could also be caused by apparmor_parser - if it thinks the cache is up to date, it just passes the cache to the kernel. (The cache is checked based on the timestamp of the profile and all included files, unfortunately not based on the content of those files)

My guess was that you might have a /etc/apparmor.d/local/usr.sbin.smbd-shares (generated from your smb.conf) that is newer than the new packaged smbd profile - but your usr.sbin.smbd-shares is a year old, so my guess doesn't fit your case.
Would have been too easy ;-) and I'm somewhat afraid that in your case it might stay a mystery what exactly happened.
Comment 9 Noel Power 2022-03-10 11:05:27 UTC
(In reply to Christian Boltz from comment #8)
> From the log in the original report:
> > type=AVC msg=audit(1643828237.305:807): apparmor="DENIED" operation="open" profile="smbd" name="/etc/ssl/openssl.cnf" pid=6144 comm="smbd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
> 
> IIRC I've never seen Samba trying to read the openssl.cnf (but I have to
> admit that I use it only rarely). Do you have a special/unusual samba config
> that could explain it?
> 
> @Noel: if you have an idea, feel free to answer as well ;-)
no idea, sorry, I'm guessing this must be pulled in by some external library used by samba. I've cc'ed samba-maintainers so maybe someone else here might have an idea 
> 
> > type=AVC msg=audit(1643828237.385:809): apparmor="DENIED" operation="exec" profile="smbd" name="/usr/lib64/samba/samba-bgqd" pid=6148 comm="smbd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
> 
> I assume that was fixed by deleting the cache?
> 
> Also, if you see other DENIED events, please tell me.
> 
> 
> Unfortunately I'm still not sure why you needed to delete the cache.
> 
> The "same as current profile, skipping" log entries from comment 5 happened
> at the same time as the updated apparmor-profiles package was installed.
> This means AppArmor thought the cache was new enough. While this happens on
> the kernel side, it could also be caused by apparmor_parser - if it thinks
> the cache is up to date, it just passes the cache to the kernel. (The cache
> is checked based on the timestamp of the profile and all included files,
> unfortunately not based on the content of those files)
> 
> My guess was that you might have a
> /etc/apparmor.d/local/usr.sbin.smbd-shares (generated from your smb.conf)
> that is newer than the new packaged smbd profile - but your
> usr.sbin.smbd-shares is a year old, so my guess doesn't fit your case.
> Would have been too easy ;-) and I'm somewhat afraid that in your case it
> might stay a mystery what exactly happened.

I have experienced cache related problems a couple of times recently, however every time I try to pin it down and reproduce it I have failed :/
Comment 10 Ralf Kölmel 2022-03-10 12:28:07 UTC
Created attachment 856907 [details]
extract of the audit.log during smb and nmb service restart

I'm still having "denied" messages during smb/nmb startup(s. attached file) , but they don't break the smb service start.

The question is if automatic cleaning of the apparmor cache (through configuration in the rpm spec) after an update of the apparmor packages would be a pragmatic way to workaround the problem.
For now i'm afraid to do automatic updates (of apparmor packages) on affected systems because it could break the samba daemon restart.
But besides a workaround the best would be to find the problem.
Comment 11 Christian Boltz 2022-03-13 11:31:49 UTC
(In reply to Ralf Kölmel from comment #10)
> I'm still having "denied" messages during smb/nmb startup(s. attached file)
> , but they don't break the smb service start.

That log basically shows 3 [groups of] denials:
- capability net_admin -> bug 1196922 aka bug 1196850 comment #3
- /proc/*/fd/ -> https://gitlab.com/apparmor/apparmor/-/merge_requests/860
- smbd and samba-bgqd reading /etc/ssl/openssl.cnf -> this bugzilla comment

Allowing to read the openssl config is quite harmless. Also, samba.spec contains BuildRequires: libopenssl-devel so reading openssl.cnf isn't too surprising.

Long story short: I just submitted 
https://gitlab.com/apparmor/apparmor/-/merge_requests/862

> The question is if automatic cleaning of the apparmor cache (through
> configuration in the rpm spec) after an update of the apparmor packages
> would be a pragmatic way to workaround the problem.

That workaround would be   rm /var/cache/apparmor/*/usr.sbin.smbd   but I'm not sure if I like it in a %post script ;-)  (would need to be done before restarting samba)

> For now i'm afraid to do automatic updates (of apparmor packages) on
> affected systems because it could break the samba daemon restart.

Understandable. If it helps:
- you can run the above workaround before updating
- this is the first time I've seen such an issue, therefore I'd say the risk
  for similar problems in future updates is quite low. (Yes, famous last 
  words ;-)

> But besides a workaround the best would be to find the problem.

Agreed.

(In reply to Noel Power from comment #9)
> I have experienced cache related problems a couple of times recently,
> however every time I try to pin it down and reproduce it I have failed :/

Timestamps of the profiles and indirectly also your samba config (via the autogenerated profile sniplet) are relevant, which makes reproducing harder.

If that happens again, please save the following files and directories in a tarball (timestamps are most important, but in worst case it's also possible to investigate the content of the cache files):
- /etc/samba/smb.conf (or at least its timestamp)
- /etc/apparmor.d/
- /var/cache/apparmor/
- /usr/share/apparmor/cache/
- /var/log/audit/audit.log
Comment 12 Noel Power 2022-03-14 09:09:27 UTC
(In reply to Christian Boltz from comment #11)

> That log basically shows 3 [groups of] denials:
> - capability net_admin -> bug 1196922 aka bug 1196850 comment #3
> - /proc/*/fd/ -> https://gitlab.com/apparmor/apparmor/-/merge_requests/860
> - smbd and samba-bgqd reading /etc/ssl/openssl.cnf -> this bugzilla comment
> 
> Allowing to read the openssl config is quite harmless. Also, samba.spec
> contains BuildRequires: libopenssl-devel so reading openssl.cnf isn't too
> surprising.
yep, correct

> (In reply to Noel Power from comment #9)
> > I have experienced cache related problems a couple of times recently,
> > however every time I try to pin it down and reproduce it I have failed :/
> 
> Timestamps of the profiles and indirectly also your samba config (via the
> autogenerated profile sniplet) are relevant, which makes reproducing harder.
> 
> If that happens again, please save the following files and directories in a
> tarball (timestamps are most important, but in worst case it's also possible
> to investigate the content of the cache files):
> - /etc/samba/smb.conf (or at least its timestamp)
> - /etc/apparmor.d/
> - /var/cache/apparmor/
> - /usr/share/apparmor/cache/
> - /var/log/audit/audit.log

Thanks for pointing out the important information to save, I will keep an eye out, hopefully I will experience this again soon so I remember this bug and/or the instructions :-)
Comment 13 Noel Power 2022-03-24 11:56:30 UTC
(In reply to Noel Power from comment #12)
> (In reply to Christian Boltz from comment #11)
[...]
> 
> Thanks for pointing out the important information to save, I will keep an
> eye out, hopefully I will experience this again soon so I remember this bug
> and/or the instructions :-)

Hi Christian, I reproduced this, better still I think I can keep on reproducing it thanks to a vm (snapshotted before/after)

this is Leap 15.4 (not probably fully updated) but I attempted to update with https://build.opensuse.org/package/show/home:npower:branches:security:apparmor/apparmor (adjusts apparmor with /proc/{pipd}/fd rule)

What is happening is the cached file /etc/apparmor.d/cache.d/*/samba-bgqd  doesn't appear to be updated correctly with the install

on system

localhost:~ # ls -lrt /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd
-rw------- 1 root root 37305 Mar 23 20:43 /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd
localhost:~ # ls -lrt /usr/share/apparmor/cache/d29c4283.0/samba-bgqd
-rw-r--r-- 1 root root 37345 Mar 23 15:50 /usr/share/apparmor/cache/d29c4283.0/samba-bgqd


but it *did* update it (see date)

if I stop apparmor, remove /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd and restart the cache file size matches /usr/share/apparmor/cache/d29c4283.0/samba-bgqd and I no longer get the DENIED for /proc/{pipd}/fd

I will attach a tarball with the data you requested. Let me know if you need any other info and/or debugging I can do to help
Comment 14 Noel Power 2022-03-24 11:57:40 UTC
Created attachment 857334 [details]
tar of apparmor/samba relevant files
Comment 15 Christian Boltz 2022-03-24 22:45:19 UTC
(In reply to Noel Power from comment #13)
> Hi Christian, I reproduced this, better still I think I can keep on
> reproducing it thanks to a vm (snapshotted before/after)

Sounds promising :-) and even if I don't yet know exactly what's going on, that should help with debugging.

> this is Leap 15.4 (not probably fully updated) but I attempted to update
> with
> https://build.opensuse.org/package/show/home:npower:branches:security:
> apparmor/apparmor (adjusts apparmor with /proc/{pipd}/fd rule)

Looks good, feel free to send a SR ;-)

> What is happening is the cached file /etc/apparmor.d/cache.d/*/samba-bgqd 
> doesn't appear to be updated correctly with the install
> on system

Just for the records: on openSUSE, /etc/apparmor.d/cache.d/ is a symlink to /var/cache/apparmor/ (IIRC some other distros have the cache directly in /etc/apparmor.d/cache.d/)

I'll paste the relevant files/timestamps from the tarball (ignoring abstractions for now) in a slightly more readable format:

Mar 23 16:51 ./etc/apparmor.d/samba-bgqd
Mar 23 16:51 ./etc/apparmor.d/local/samba-bgqd
Mar 23 16:50 ./usr/share/apparmor/cache/d29c4283.0/samba-bgqd
Mar 23 21:43 ./var/cache/apparmor/d29c4283.0/samba-bgqd

This means:
- the packaged cache in /usr/ has a too-old timestamp (that's a packaging 
  bug, and I have a solution mostly ready - just waiting for an upstream 
  answer for a detail question)
- the locally generated cache in /var/ looks new enough (clearly newer
  than the profile in /etc/)

> but it *did* update it (see date)

Indeed, and that's the strange part.

> if I stop apparmor, remove /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd and
> restart the cache file size matches
> /usr/share/apparmor/cache/d29c4283.0/samba-bgqd and I no longer get the
> DENIED for /proc/{pipd}/fd

Just deleting the cache file in /var/ + rcapparmor reload should be enough - no need to stop apparmor.

> I will attach a tarball with the data you requested. Let me know if you need
> any other info and/or debugging I can do to help

Just to be sure - does your tarball contain the "broken" or the "working" cache?


I'd also be interested in an audit.log that includes reloading the samba-bgqd profile multiple times and in multiple cases? Please first read the "Crazy idea" at the end, then run

sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
rcapparmor reload
sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
echo "zypper up" >> /var/log/audit/audit.log
zypper up  # or whatever you need to install your updated AppArmor packages
sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
rcapparmor reload
sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
echo "samba-bgqd cache deleted" >> /var/log/audit/audit.log
rm -v /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
rcapparmor reload
sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log

With this log spamming ;-) audit.log should show where things break.
/var/log/zypp/history (the section with installing your updated AppArmor packages is enough) might also be helpful.

Crazy idea, since you already have your own branch packages: can you please add 
    sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
    sleep 5
in %post of apparmor-parser, apparmor-profiles and apparmor-abstractions before doing the above test? That sha256sum+sleep should be the last thing %post does.

(wild guess: both -profiles and -abstractions reload the profiles, and I wonder if that could be part of the problem. The first one triggers cache generation, and second one changes a file in /etc - with same timestamp as all packaged profiles, so that the cache doesn't get regenerated.)
Comment 16 Noel Power 2022-03-25 11:30:38 UTC
(In reply to Christian Boltz from comment #15)
> (In reply to Noel Power from comment #13)
> > Hi Christian, I reproduced this, better still I think I can keep on
> > reproducing it thanks to a vm (snapshotted before/after)
> 
> Sounds promising :-) and even if I don't yet know exactly what's going on,
> that should help with debugging.
> 
> > this is Leap 15.4 (not probably fully updated) but I attempted to update
> > with
> > https://build.opensuse.org/package/show/home:npower:branches:security:
> > apparmor/apparmor (adjusts apparmor with /proc/{pipd}/fd rule)
> 
> Looks good, feel free to send a SR ;-)

right, hehe I was in the middle of doing that and testing when I got the lucky break of (hopefully) being able to consistently have this fail

see https://build.opensuse.org/request/show/964827

> 
> > What is happening is the cached file /etc/apparmor.d/cache.d/*/samba-bgqd 
> > doesn't appear to be updated correctly with the install
> > on system
> 
> Just for the records: on openSUSE, /etc/apparmor.d/cache.d/ is a symlink to
> /var/cache/apparmor/ (IIRC some other distros have the cache directly in
> /etc/apparmor.d/cache.d/)

oh yeah, I already realised this when I expanded the tar ball on another machine to look at it and the cache.d contents were completely different to what I was expecting :-)

> 
> I'll paste the relevant files/timestamps from the tarball (ignoring
> abstractions for now) in a slightly more readable format:
> 
> Mar 23 16:51 ./etc/apparmor.d/samba-bgqd
> Mar 23 16:51 ./etc/apparmor.d/local/samba-bgqd
> Mar 23 16:50 ./usr/share/apparmor/cache/d29c4283.0/samba-bgqd
> Mar 23 21:43 ./var/cache/apparmor/d29c4283.0/samba-bgqd
> 
> This means:
> - the packaged cache in /usr/ has a too-old timestamp (that's a packaging 
>   bug, and I have a solution mostly ready - just waiting for an upstream 
>   answer for a detail question)
> - the locally generated cache in /var/ looks new enough (clearly newer
>   than the profile in /etc/)
> 
> > but it *did* update it (see date)
> 
> Indeed, and that's the strange part.
> 
> > if I stop apparmor, remove /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd and
> > restart the cache file size matches
> > /usr/share/apparmor/cache/d29c4283.0/samba-bgqd and I no longer get the
> > DENIED for /proc/{pipd}/fd
> 
> Just deleting the cache file in /var/ + rcapparmor reload should be enough -
> no need to stop apparmor.
good to know, hehe I am a hammer for everything kinda guy
> 
> > I will attach a tarball with the data you requested. Let me know if you need
> > any other info and/or debugging I can do to help
> 
> Just to be sure - does your tarball contain the "broken" or the "working"
> cache?

tar ball contains detail of broken cache file (note though the version of my package is slightly older (doesn't include the openssl bit)

> 
> 
> I'd also be interested in an audit.log that includes reloading the
> samba-bgqd profile multiple times and in multiple cases? Please first read
> the "Crazy idea" at the end, then run

I'll look at this on Tue when I am back from vacation :->

> 
> sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
> rcapparmor reload
> sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
> echo "zypper up" >> /var/log/audit/audit.log
> zypper up  # or whatever you need to install your updated AppArmor packages
> sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
> rcapparmor reload
> sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
> echo "samba-bgqd cache deleted" >> /var/log/audit/audit.log
> rm -v /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
> rcapparmor reload
> sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
> 
> With this log spamming ;-) audit.log should show where things break.
> /var/log/zypp/history (the section with installing your updated AppArmor
> packages is enough) might also be helpful.
> 
> Crazy idea, since you already have your own branch packages: can you please
> add 
>     sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
>     sleep 5
> in %post of apparmor-parser, apparmor-profiles and apparmor-abstractions
> before doing the above test? That sha256sum+sleep should be the last thing
> %post does.
> 
> (wild guess: both -profiles and -abstractions reload the profiles, and I
> wonder if that could be part of the problem. The first one triggers cache
> generation, and second one changes a file in /etc - with same timestamp as
> all packaged profiles, so that the cache doesn't get regenerated.)
Comment 17 OBSbugzilla Bot 2022-03-25 21:10:03 UTC
This is an autogenerated message for OBS integration:
This bug (1195463) was mentioned in
https://build.opensuse.org/request/show/964948 Factory / apparmor
Comment 18 Noel Power 2022-03-29 13:40:37 UTC
(In reply to Christian Boltz from comment #15)

> I'd also be interested in an audit.log that includes reloading the
> samba-bgqd profile multiple times and in multiple cases? Please first read
> the "Crazy idea" at the end, then run
> 
> sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
> rcapparmor reload
> sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
> echo "zypper up" >> /var/log/audit/audit.log
> zypper up  # or whatever you need to install your updated AppArmor packages
> sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
> rcapparmor reload
> sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
> echo "samba-bgqd cache deleted" >> /var/log/audit/audit.log
> rm -v /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
> rcapparmor reload
> sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
> 
> With this log spamming ;-) audit.log should show where things break.
> /var/log/zypp/history (the section with installing your updated AppArmor
> packages is enough) might also be helpful.
> 
> Crazy idea, since you already have your own branch packages: can you please
> add 
>     sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
>     sleep 5
> in %post of apparmor-parser, apparmor-profiles and apparmor-abstractions
> before doing the above test? That sha256sum+sleep should be the last thing
> %post does.
> 
> (wild guess: both -profiles and -abstractions reload the profiles, and I
> wonder if that could be part of the problem. The first one triggers cache
> generation, and second one changes a file in /etc - with same timestamp as
> all packaged profiles, so that the cache doesn't get regenerated.)

gonna attach files for this, I followed the instructions above but additionally started and stopped rcsmb  e.g.

  zypper up  # or whatever you need to install your updated AppArmor packages

=> echo "##### starting smbd" >> /var/log/audit/audit.log
=> rcsmb start
=> echo "##### stopping smbd" >> /var/log/audit/audit.log
=> rcsmb stop

sha256sum /var/cache/apparmor/*/samba-bgqd >> /var/log/audit/audit.log
Comment 19 Noel Power 2022-03-29 13:41:13 UTC
Created attachment 857476 [details]
tar of apparmor log (and zypper history)
Comment 20 Christian Boltz 2022-03-29 20:54:32 UTC
Hmm, in this case only deleting the cache file helped :-/
I know it's hard to find out in hindsight, but do you remember if the cache file you deleted had a newer timestamp than the profile in the apparmor-profiles rpm installed during zypper up?


Anyway, I think there are two things I need to fix:

1. The timestamp of the precompiled cache (/usr/share/apparmor/cache/) is older than the profiles.
cp -r instead of cp -a for copying the cache in %install will fix that, and should prevent generation of cache files in /var/cache/apparmor (you might need to delete the content of /var/cache/apparmor to see this effect - nevertheless I'm not sure if I should do that in %post. Note that this is just about getting rid of some files in /var/cache/apparmor, but together with 2 the cache file shouldn't hurt.)

2. Reloading the profiles (which also means updating the cache) is done in %post of apparmor-profiles _and_ apparmor-abstractions.
Unfortunately the first reload happens when only one of the packages got updated, therefore the cache is based on a half-updated state. When the second package gets updated, the cache is new enough (newer than the packaged profiles) and won't get updated again.
The change for 1 makes it less likely that this happens, but to be sure, I'll move reloading the profiles from %post to %posttrans.


I just submitted a package with these fixes to home:cboltz/apparmor.
Can you please test if you still can reproduce the issue with these packages?
(Please do NOT delete any cache files manually before installing the packages to get a real-world result. You should get updated cache files in /var/cache/apparmor/, probably with the same content/checksum as in /usr/share/apparmor/cache/.)
Comment 21 Noel Power 2022-03-30 08:30:46 UTC
(In reply to Christian Boltz from comment #20)
> Hmm, in this case only deleting the cache file helped :-/
> I know it's hard to find out in hindsight, but do you remember if the cache
> file you deleted had a newer timestamp than the profile in the
> apparmor-profiles rpm installed during zypper up?
> 
no but I should be able to reproduce this and tell you

> 
> Anyway, I think there are two things I need to fix:
> 
> 1. The timestamp of the precompiled cache (/usr/share/apparmor/cache/) is
> older than the profiles.
> cp -r instead of cp -a for copying the cache in %install will fix that, and
> should prevent generation of cache files in /var/cache/apparmor (you might
> need to delete the content of /var/cache/apparmor to see this effect -
> nevertheless I'm not sure if I should do that in %post. Note that this is
> just about getting rid of some files in /var/cache/apparmor, but together
> with 2 the cache file shouldn't hurt.)
> 
> 2. Reloading the profiles (which also means updating the cache) is done in
> %post of apparmor-profiles _and_ apparmor-abstractions.
> Unfortunately the first reload happens when only one of the packages got
> updated, therefore the cache is based on a half-updated state. When the
> second package gets updated, the cache is new enough (newer than the
> packaged profiles) and won't get updated again.
> The change for 1 makes it less likely that this happens, but to be sure,
> I'll move reloading the profiles from %post to %posttrans.
> 
> 
> I just submitted a package with these fixes to home:cboltz/apparmor.
> Can you please test if you still can reproduce the issue with these packages?
> (Please do NOT delete any cache files manually before installing the
> packages to get a real-world result. You should get updated cache files in
> /var/cache/apparmor/, probably with the same content/checksum as in
> /usr/share/apparmor/cache/.)

 will try after (re)trying the debug package to get your answer above
Comment 22 Noel Power 2022-03-30 08:53:31 UTC
(In reply to Noel Power from comment #21)
> (In reply to Christian Boltz from comment #20)
> > Hmm, in this case only deleting the cache file helped :-/
> > I know it's hard to find out in hindsight, but do you remember if the cache
> > file you deleted had a newer timestamp than the profile in the
> > apparmor-profiles rpm installed during zypper up?
> > 
> no but I should be able to reproduce this and tell you

#before install

localhost:~ # ls -lrtR /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd
-rw------- 1 root root 37305 Mar 23 16:43 /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd
localhost:~ # ls -lrt /usr/share/apparmor/cache/d29c4283.0/samba-bgqd
-rw-r--r-- 1 root root 37305 Feb 16 21:25 /usr/share/apparmor/cache/d29c4283.0/samba-bgqd


#after install

localhost:~ #  ls -lrtR /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd
-rw------- 1 root root 37305 Mar 30 09:45 /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd
localhost:~ #  ls -lrt /usr/share/apparmor/cache/d29c4283.0/samba-bgqd
-rw-r--r-- 1 root root 38145 Mar 29 12:30 /usr/share/apparmor/cache/d29c4283.0/samba-bgqd

#after cache delete

localhost:~ # ls -lrt /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd
-rw------- 1 root root 38145 Mar 30 09:45 /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd
localhost:~ #  ls -lrt /usr/share/apparmor/cache/d29c4283.0/samba-bgqd
-rw-r--r-- 1 root root 38145 Mar 29 12:30 /usr/share/apparmor/cache/d29c4283.0/samba-bgqd
Comment 23 Noel Power 2022-03-30 10:34:49 UTC
(In reply to Noel Power from comment #21)
u please test if you still can reproduce the issue with these packages?
> > (Please do NOT delete any cache files manually before installing the
> > packages to get a real-world result. You should get updated cache files in
> > /var/cache/apparmor/, probably with the same content/checksum as in
> > /usr/share/apparmor/cache/.)
> 
>  will try after (re)trying the debug package to get your answer above

#before install of your latest

localhost:~ #  ls -lrt /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd
-rw------- 1 root root 37305 Mar 23 16:43 /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd
localhost:~ # ls -lrt /usr/share/apparmor/cache/d29c4283.0/samba-bgqd
-rw-r--r-- 1 root root 37305 Feb 16 21:25 /usr/share/apparmor/cache/d29c4283.0/samba-bgqd

#after install

localhost:~ # ls -lrt /usr/share/apparmor/cache/d29c4283.0/samba-bgqd
-rw-r--r-- 1 root root 38145 Mar 30 10:12 /usr/share/apparmor/cache/d29c4283.0/samba-bgqd
localhost:~ # ls -lrt /etc/apparmor.d/cache.d/d29c4283.0/samba-bgqd
-rw------- 1 root root 38145 Mar 30 11:30 /etc/apparmor.d/cache.d/d29c4283.0

so.... looking good!!!
Comment 26 OBSbugzilla Bot 2022-04-03 15:20:04 UTC
This is an autogenerated message for OBS integration:
This bug (1195463) was mentioned in
https://build.opensuse.org/request/show/966667 Factory / apparmor
Comment 30 Swamp Workflow Management 2022-08-02 16:16:11 UTC
SUSE-RU-2022:2628-1: An update that has two recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1195463,1196850
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.3 (src):    apparmor-2.13.6-150300.3.15.1, libapparmor-2.13.6-150300.3.15.1
SUSE Linux Enterprise Module for Server Applications 15-SP3 (src):    apparmor-2.13.6-150300.3.15.1
SUSE Linux Enterprise Module for Basesystem 15-SP3 (src):    apparmor-2.13.6-150300.3.15.1, libapparmor-2.13.6-150300.3.15.1
SUSE Linux Enterprise Micro 5.2 (src):    apparmor-2.13.6-150300.3.15.1, libapparmor-2.13.6-150300.3.15.1
SUSE Linux Enterprise Micro 5.1 (src):    apparmor-2.13.6-150300.3.15.1, libapparmor-2.13.6-150300.3.15.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 31 Swamp Workflow Management 2022-08-02 16:17:26 UTC
SUSE-RU-2022:2627-1: An update that has two recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1195463,1196850
CVE References: 
JIRA References: 
Sources used:
SUSE Linux Enterprise Software Development Kit 12-SP5 (src):    apparmor-2.8.2-56.9.1
SUSE Linux Enterprise Server 12-SP5 (src):    apparmor-2.8.2-56.9.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.