Bug 1154871 - p11-kit-nss-trust breaks Firefox addon update check
p11-kit-nss-trust breaks Firefox addon update check
Status: IN_PROGRESS
: 1172010 (view as bug list)
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Firefox
Current
Other Other
: P2 - High : Normal (vote)
: ---
Assigned To: Ludwig Nussel
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2019-10-23 14:23 UTC by Marcus Rückert
Modified: 2020-08-26 22:13 UTC (History)
5 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Marcus Rückert 2019-10-23 14:23:27 UTC
When running with p11-kit-nss-trust:

```
1571839945089 addons.update-checker WARN Request failed: 
https://versioncheck.addons.mozilla.org/update/VersionCheck.php?reqVersion=2&id=jid1-MnnxcxisBPnSXQ@jetpack&version=2018.12.17&maxAppVersion=null&status=userEnabled&appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=69.0.3&appOS=Linux&appABI=x86_64-gcc3&locale=en-US&currentAppVersion=69.0.3&updateType=97&compatMode=normal - [Exception "Certificate issuer is not built-in."  nsresult: "0x80004004 (NS_ERROR_ABORT)"  location: "JS frame :: resource://gre/modules/CertUtils.jsm :: checkCert :: line 182"  data: no]
```

Just replacing p11-kit-nss-trust with mozilla-nss-certs again fixes the issue.
But this makes using our internal SUSE CA more annoying for our users.

Can we adapt the p11-kit-nss-trust API to the new expected API from Firefox?
Comment 1 Marcus Meissner 2019-10-23 14:36:04 UTC
-> bugowner of p11-kit-nss-trust
Comment 2 Ludwig Nussel 2019-10-23 15:38:07 UTC
what is the new api from firefox? In any case, would be a job for upstream
Comment 3 Marcus Rückert 2019-10-23 17:43:20 UTC
well something tells firefox "hey this cert is from your internal DB and not an user imported cert" and for its version checking host it wants that flag.
Comment 4 Ludwig Nussel 2019-10-24 12:01:45 UTC
also happens on 15.1 so something must have changed in FF. Question is what does it use to determine built in or not?
Comment 5 Marcus Rückert 2019-10-24 13:19:45 UTC
I would assume a field in the struct returned from the mozilla nss cert db API
Comment 7 Ludwig Nussel 2019-11-15 12:06:30 UTC
Can you give https://build.opensuse.org/package/show/home:lnussel:ca-certificates a try?
Comment 13 Swamp Workflow Management 2019-12-10 14:13:35 UTC
SUSE-RU-2019:3240-1: An update that has one recommended fix can now be installed.

Category: recommended (moderate)
Bug References: 1154871
CVE References: 
Sources used:
SUSE Linux Enterprise Module for Open Buildservice Development Tools 15-SP1 (src):    p11-kit-0.23.2-4.5.2
SUSE Linux Enterprise Module for Desktop Applications 15 (src):    p11-kit-0.23.2-4.5.2
SUSE Linux Enterprise Module for Basesystem 15-SP1 (src):    ca-certificates-mozilla-2.34-4.15.2, p11-kit-0.23.2-4.5.2
SUSE Linux Enterprise Module for Basesystem 15 (src):    ca-certificates-mozilla-2.34-4.15.2, p11-kit-0.23.2-4.5.2

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 14 Swamp Workflow Management 2019-12-15 20:10:35 UTC
openSUSE-RU-2019:2686-1: An update that has one recommended fix can now be installed.

Category: recommended (moderate)
Bug References: 1154871
CVE References: 
Sources used:
openSUSE Leap 15.1 (src):    ca-certificates-mozilla-2.34-lp151.2.6.1, p11-kit-0.23.2-lp151.4.3.1
Comment 15 Rolf Krahl 2019-12-18 10:19:26 UTC
Patch openSUSE-2019-2686 seem to break curl (and probably other things).

For the sake of reproducibility, let's look at the things in docker containers:

If a take the current opensuse/leap:15.1 image from Docker Hub, zypper addrepo works:

====================

$ docker images opensuse/leap:15.1           
REPOSITORY       TAG              IMAGE ID         CREATED          SIZE
opensuse/leap    15.1             fef5ad254f63     6 weeks ago      103MB
$ docker run -ti --rm opensuse/leap:15.1 bash
371cbf74a98c:/ # zypper --non-interactive addrepo https://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_Leap_15.1/devel:languages:python.repo
Adding repository 'Python Modules (openSUSE_Leap_15.1)' .................[done]
Repository 'Python Modules (openSUSE_Leap_15.1)' successfully added

URI         : http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_Leap_15.1/
Enabled     : Yes                                                                                   
GPG Check   : Yes                                                                                   
Autorefresh : No                                                                                    
Priority    : 99 (default priority)                                                                 

Repository priorities are without effect. All enabled repositories share the same priority.
371cbf74a98c:/ # exit

====================

Now, if I install Patch openSUSE-2019-2686 first, zypper addrepo is broken:

====================

$ docker run -ti --rm opensuse/leap:15.1 bash
fea96c249a96:/ # zypper --non-interactive install patch:openSUSE-2019-2686
Building repository 'Non-OSS Repository' cache ........................[done]
Building repository 'Main Repository' cache ...........................[done]
Building repository 'Main Update Repository' cache ....................[done]
Building repository 'Update Repository (Non-Oss)' cache ...............[done]
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following NEW patch is going to be installed:
  openSUSE-2019-2686

The following 4 packages are going to be upgraded:
  ca-certificates-mozilla libp11-kit0 p11-kit p11-kit-tools

4 packages to upgrade.
Overall download size: 642.6 KiB. Already cached: 0 B. After the operation, additional 674.4 KiB will be used.
Continue? [y/n/v/...? shows all options] (y): y
Retrieving package ca-certificates-mozilla-2.34-lp151.2.6.1.noarch                                                                                      (1/4), 357.2 KiB (961.6 KiB unpacked)
Retrieving: ca-certificates-mozilla-2.34-lp151.2.6.1.noarch.rpm .......[done]
Retrieving package libp11-kit0-0.23.2-lp151.4.3.1.x86_64                                                                                                (2/4), 116.0 KiB (408.5 KiB unpacked)
Retrieving: libp11-kit0-0.23.2-lp151.4.3.1.x86_64.rpm .................[done]
Retrieving package p11-kit-tools-0.23.2-lp151.4.3.1.x86_64                                                                                              (3/4),  79.4 KiB (238.1 KiB unpacked)
Retrieving: p11-kit-tools-0.23.2-lp151.4.3.1.x86_64.rpm ...............[done]
Retrieving package p11-kit-0.23.2-lp151.4.3.1.x86_64                                                                                                    (4/4),  90.2 KiB (263.0 KiB unpacked)
Retrieving: p11-kit-0.23.2-lp151.4.3.1.x86_64.rpm .....................[done]

Checking for file conflicts: ..........................................[done]
(1/4) Installing: ca-certificates-mozilla-2.34-lp151.2.6.1.noarch .....[done]
Additional rpm output:
p11-kit: ca-certificates-mozila.trust.p11-kit: nss-mozilla-ca-policy: invalid or unsupported attribute                                                                                       
p11-kit: ca-certificates-mozila.trust.p11-kit: nss-mozilla-ca-policy: invalid or unsupported attribute                                                                                       
p11-kit: ca-certificates-mozila.trust.p11-kit: nss-mozilla-ca-policy: invalid or unsupported attribute                                                                                       
p11-kit: ca-certificates-mozila.trust.p11-kit: nss-mozilla-ca-policy: invalid or unsupported attribute                                                                                       
p11-kit: ca-certificates-mozila.trust.p11-kit: nss-mozilla-ca-policy: invalid or unsupported attribute                                                                                       
p11-kit: ca-certificates-mozila.trust.p11-kit: nss-mozilla-ca-policy: invalid or unsupported attribute                                                                                       
p11-kit: ca-certificates-mozila.trust.p11-kit: nss-mozilla-ca-policy: invalid or unsupported attribute                                                                                       
p11-kit: ca-certificates-mozila.trust.p11-kit: nss-mozilla-ca-policy: invalid or unsupported attribute                                                                                       


(2/4) Installing: libp11-kit0-0.23.2-lp151.4.3.1.x86_64 ...............[done]
(3/4) Installing: p11-kit-tools-0.23.2-lp151.4.3.1.x86_64 .............[done]
(4/4) Installing: p11-kit-0.23.2-lp151.4.3.1.x86_64 ...................[done]
 
fea96c249a96:/ # zypper --non-interactive addrepo https://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_Leap_15.1/devel:languages:python.repo
Download (curl) error for 'https://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_Leap_15.1/devel:languages:python.repo':
Error code: Curl error 60                                                                                                                                                                    
Error message: SSL certificate problem: unable to get local issuer certificate                                                                                                               

Abort, retry, ignore? [a/r/i/...? shows all options] (a): a
Problem encountered while trying to read the file at the specified URI:
ABORT request: Aborting requested by user                                                                                                                                                    
fea96c249a96:/ # exit

====================

Note the error messages in the additional rpm output. It seem to be a problem with the order of the updates. Indeed, if I update p11-kit first, things work again:

====================

$ docker run -ti --rm opensuse/leap:15.1 bash
446c8dc8c7a5:/ # zypper --non-interactive update p11-kit
Building repository 'Non-OSS Repository' cache ........................[done]
Building repository 'Main Repository' cache ...........................[done]
Building repository 'Main Update Repository' cache ....................[done]
Building repository 'Update Repository (Non-Oss)' cache ...............[done]
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following package is going to be upgraded:
  p11-kit

1 package to upgrade.
Overall download size: 90.2 KiB. Already cached: 0 B. After the operation, additional 8.0 B will be used.
Continue? [y/n/v/...? shows all options] (y): y
Retrieving package p11-kit-0.23.2-lp151.4.3.1.x86_64                                                                                                    (1/1),  90.2 KiB (263.0 KiB unpacked)
Retrieving: p11-kit-0.23.2-lp151.4.3.1.x86_64.rpm .....................[done]

Checking for file conflicts: ..........................................[done]
(1/1) Installing: p11-kit-0.23.2-lp151.4.3.1.x86_64 ...................[done]
 
446c8dc8c7a5:/ # zypper --non-interactive install patch:openSUSE-2019-2686
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following NEW patch is going to be installed:
  openSUSE-2019-2686

The following 3 packages are going to be upgraded:
  ca-certificates-mozilla libp11-kit0 p11-kit-tools

3 packages to upgrade.
Overall download size: 552.5 KiB. Already cached: 0 B. After the operation, additional 674.4 KiB will be used.
Continue? [y/n/v/...? shows all options] (y): y
Retrieving package ca-certificates-mozilla-2.34-lp151.2.6.1.noarch                                                                                      (1/3), 357.2 KiB (961.6 KiB unpacked)
Retrieving: ca-certificates-mozilla-2.34-lp151.2.6.1.noarch.rpm .......[done]
Retrieving package libp11-kit0-0.23.2-lp151.4.3.1.x86_64                                                                                                (2/3), 116.0 KiB (408.5 KiB unpacked)
Retrieving: libp11-kit0-0.23.2-lp151.4.3.1.x86_64.rpm .................[done]
Retrieving package p11-kit-tools-0.23.2-lp151.4.3.1.x86_64                                                                                              (3/3),  79.4 KiB (238.1 KiB unpacked)
Retrieving: p11-kit-tools-0.23.2-lp151.4.3.1.x86_64.rpm ...............[done]

Checking for file conflicts: ..........................................[done]
(1/3) Installing: ca-certificates-mozilla-2.34-lp151.2.6.1.noarch .....[done]
(2/3) Installing: libp11-kit0-0.23.2-lp151.4.3.1.x86_64 ...............[done]
(3/3) Installing: p11-kit-tools-0.23.2-lp151.4.3.1.x86_64 .............[done]
 
446c8dc8c7a5:/ # zypper --non-interactive addrepo https://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_Leap_15.1/devel:languages:python.repo
Adding repository 'Python Modules (openSUSE_Leap_15.1)' ...............[done]
Repository 'Python Modules (openSUSE_Leap_15.1)' successfully added

URI         : http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_Leap_15.1/
Enabled     : Yes                                                                                   
GPG Check   : Yes                                                                                   
Autorefresh : No                                                                                    
Priority    : 99 (default priority)                                                                 

Repository priorities are without effect. All enabled repositories share the same priority.
446c8dc8c7a5:/ # exit
Comment 16 Marcus Meissner 2019-12-18 10:26:13 UTC
this will be fixed by running once:

update-ca-certificates

problem is that p11-kit was updated after ca-certificates-mozilla, which needsth newer p11-kit :(
Comment 17 Rolf Krahl 2019-12-18 11:39:04 UTC
(In reply to Marcus Meissner from comment #16)
> this will be fixed by running once:
> 
> update-ca-certificates
> 
> problem is that p11-kit was updated after ca-certificates-mozilla, which
> needsth newer p11-kit :(

Yes, that is what I suggested in the last example. The problem with running update-ca-certificates is that this is not foreseen in automatic workflows. I noticed the issue because my docker builds failed.
Comment 18 Swamp Workflow Management 2019-12-18 15:10:06 UTC
This is an autogenerated message for OBS integration:
This bug (1154871) was mentioned in
https://build.opensuse.org/request/show/757879 Factory / ca-certificates-mozilla
Comment 20 Swamp Workflow Management 2020-02-19 11:14:35 UTC
SUSE-RU-2020:0404-1: An update that has one recommended fix can now be installed.

Category: recommended (moderate)
Bug References: 1154871
CVE References: 
Sources used:
SUSE Linux Enterprise Software Development Kit 12-SP5 (src):    p11-kit-0.20.7-3.3.4
SUSE Linux Enterprise Software Development Kit 12-SP4 (src):    p11-kit-0.20.7-3.3.4
SUSE Linux Enterprise Server 12-SP5 (src):    p11-kit-0.20.7-3.3.4
SUSE Linux Enterprise Server 12-SP4 (src):    p11-kit-0.20.7-3.3.4
SUSE Linux Enterprise Desktop 12-SP4 (src):    p11-kit-0.20.7-3.3.4
SUSE CaaS Platform 3.0 (src):    p11-kit-0.20.7-3.3.4

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 21 Swamp Workflow Management 2020-03-05 17:29:23 UTC
SUSE-RU-2020:0596-1: An update that has 7 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 1010996,1071152,1071390,1082318,1100415,1154871,1160160
CVE References: 
Sources used:
SUSE Linux Enterprise Server 12-SP5 (src):    ca-certificates-mozilla-2.40-12.20.1
SUSE Linux Enterprise Server 12-SP4 (src):    ca-certificates-mozilla-2.40-12.20.1
SUSE Linux Enterprise Desktop 12-SP4 (src):    ca-certificates-mozilla-2.40-12.20.1
SUSE CaaS Platform 3.0 (src):    ca-certificates-mozilla-2.40-12.20.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 22 Marcus Meissner 2020-03-11 06:38:33 UTC
I will revert this for SLE12, as it breaks various non-zypper upadte scenarios.
Comment 24 Jan Zerebecki 2020-03-23 19:08:14 UTC
Noticed the problem with ca-certificates-mozilla (version 2.40-lp151.2.6.1) on Leap 15.1. This made the OpenStack devstack CI fail. Should be fixed by https://build.opensuse.org/request/show/787571 .
Comment 26 Ludwig Nussel 2020-04-20 14:54:47 UTC
What is supposed to happen here now? Looks like I made change to Base:System/ca-certificates-mozilla on December 18 to fix this issue. Not sure why it didn't reach SLE12. Now there's a deviation of packages again. So just a matter of adjusting the version numbers in Base:Systems's ca-certificates-mozilla and submit back to SLE12?
Comment 27 Marcus Meissner 2020-04-20 15:17:55 UTC
I have been having fallout with this feature and it is still broken as-is.

For SLE12 I have already reverted it. (SLE12 was even worse, we have a too old p11-kit in older SPs, and only the newer one was updated.)

The problem is I think that the ordering can not be ensured the way you envisioned, regardless of how much prerequires you add.

When installing fresh after GA, zypper will just use a random (well alphabetical to some degree) order, and install p11-kit-tool updates after ca-certificates-mozilla ... 

But then zypper will have lost any SSL root CAs in the middle and the system is broken.

So I think the transition of databases you have done is impossible to deploy via maintenance update unless ca-certificates(-mozilla) keeps both its databases in parallel.

Otherwise it actively breaks customers (and not just our openqa runs).

I tried some more things on SLE12 , but failed miserable every time, and currently would just revert it for all versions where p11-kit is not yet fixed at GA time.
Comment 28 Ludwig Nussel 2020-04-20 15:44:28 UTC
The conflict does not work? If the conflict is present I can't see how zypper would install ca-certificates mozilla before p11-kit. rpm would complain.
Comment 29 Marcus Meissner 2020-04-20 15:54:30 UTC
no, sadly not.

zypper does not use rpm dependency handling ... for a "complete transaction", so one "zypper patch" run, it applies all RPMs with these internal settings, which do what you fear they do:

zypp/target/TargetImpl.cc
            // Why force and nodeps?
            //
            // Because zypp builds the transaction and the resolver asserts that
            // everything is fine.
            // We use rpm just to unpack and register the package in the database.
            // We do this step by step, so rpm is not aware of the bigger context.
            // So we turn off rpms internal checks, because we do it inside zypp.

            flags |= rpm::RPMINST_NODEPS;
            flags |= rpm::RPMINST_FORCE;


I am not sure it does re-ordering to solve out conflicts, i expect it does not do that. A cursory libzypp check only sees reordering to avoid too many MEDIA changes, not by any solver guidance.
Comment 30 Josef Cejka 2020-05-28 11:23:41 UTC
*** Bug 1172010 has been marked as a duplicate of this bug. ***
Comment 33 Swamp Workflow Management 2020-08-20 19:13:31 UTC
SUSE-RU-2020:2284-1: An update that has 6 recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1010996,1071152,1071390,1154871,1174673,973042
CVE References: 
JIRA References: 
Sources used:
SUSE Linux Enterprise Server for SAP 15 (src):    ca-certificates-mozilla-2.42-4.26.1
SUSE Linux Enterprise Server 15-LTSS (src):    ca-certificates-mozilla-2.42-4.26.1
SUSE Linux Enterprise Module for Basesystem 15-SP1 (src):    ca-certificates-mozilla-2.42-4.26.1
SUSE Linux Enterprise High Performance Computing 15-LTSS (src):    ca-certificates-mozilla-2.42-4.26.1
SUSE Linux Enterprise High Performance Computing 15-ESPOS (src):    ca-certificates-mozilla-2.42-4.26.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 34 Swamp Workflow Management 2020-08-23 22:14:57 UTC
openSUSE-RU-2020:1253-1: An update that has 6 recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1010996,1071152,1071390,1154871,1174673,973042
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.1 (src):    ca-certificates-mozilla-2.42-lp151.2.9.1
Comment 35 Swamp Workflow Management 2020-08-26 22:13:53 UTC
openSUSE-RU-2020:1264-1: An update that has 6 recommended fixes can now be installed.

Category: recommended (important)
Bug References: 1010996,1071152,1071390,1154871,1174673,973042
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.2 (src):    ca-certificates-mozilla-2.42-lp152.2.4.2