Bug 1143215 - mariadb 10.3.16 after any update Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed
mariadb 10.3.16 after any update Plugin 'ROCKSDB' registration as a STORAGE E...
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Other
Current
All openSUSE Factory
: P5 - None : Normal (vote)
: ---
Assigned To: Kristyna Streitova
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2019-07-29 10:01 UTC by Илья Индиго
Modified: 2019-12-11 15:48 UTC (History)
1 user (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
mariadb.log (6.64 KB, text/x-log)
2019-07-29 10:01 UTC, Илья Индиго
Details
mariadb-10.3.17.log (9.45 KB, text/x-log)
2019-08-02 22:16 UTC, Илья Индиго
Details
system-helper patch (1.16 KB, patch)
2019-08-06 12:04 UTC, Kristyna Streitova
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Илья Индиго 2019-07-29 10:01:07 UTC
Created attachment 811922 [details]
mariadb.log

After upgrading mariadb to 10.3.16 (currently not yet in Factory), even after manually starting "sudo mysql_upgrade -u my_admin_user -p", it always leads to an initialization error of the RocksDB plugin.
Manually running "sudo systemctl restart mariadb" after upgrading or reinstalling the mariadb package solves the problem.

[ERROR] RocksDB: Error opening instance, Status Code: 5, Status: IO error: While lock file: ./#rocksdb/LOCK: Resource temporarily unavailable
[ERROR] Plugin 'ROCKSDB' init function returned error.
[ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.
[ERROR] mysqld: Can't lock aria control file '/var/lib/mysql/aria_log_control' for exclusive use, error: 11. Will retry for 30 seconds

From important notes.
1 Python2 is not installed on the system.
2 The root @ localhost user has been deleted on the server.
3 In /etc/my.cnf
     * skip-network in my.conf (no network, only socket)
     * sqlmode default (STRICT_TRANS_TABLES, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER, NO_ENGINE_SUBSTITUTION).
Comment 1 Kristyna Streitova 2019-08-02 17:08:43 UTC
Hello Ilya,

thanks for your bug report. I've tried to reproduce it with the new version I've just submitted [1], but unsuccessfully. Can you please install this updated (10.3.17) package and check if it's still affected? If so, I would need precise steps to reproduce.

Thanks!


[1] https://build.opensuse.org/request/show/720706
Comment 2 Илья Индиго 2019-08-02 22:16:50 UTC
Created attachment 812667 [details]
mariadb-10.3.17.log
Comment 3 Илья Индиго 2019-08-02 22:17:15 UTC
Hello, Kristyna!

The fault is reproduced on mariadb 10.3.17, moreover, on a clean installation and default configuration, with the exception of connecting ha_rocksdb.so plugin.

0 If you already have mariadb installed and running, stop it, save or delete the data and configuration, and delete it.

sudo systemctl stop mariadb
sudo rm -rf /var/log/mysql /var/lib/mysql /etc/my.cnf.d /etc/my.cnf
sudo zypper rm mariadb

1 Install mariadb and run.

sudo zypper in mariadb
sudo systemctl start mariadb

2 Create file /etc/my.cnf.d/rocksdb.cnf and restart mariadb.

sudo echo -e '[server]\nplugin-load-add=ha_rocksdb.so' > /etc/my.cnf.d/rocksdb.cnf
sudo systemctl restart mariadb

3 Reinstall and restart mariadb.

sudo zypper in -f mariadb
sudo systemctl restart mariadb

4 Check the mariadb log for errors.

sudo cat /var/log/mysql/mysqld.log | grep ERROR

2019-08-03  0:01:55 0 [ERROR] RocksDB: Error opening instance, Status Code: 5, Status: IO error: While lock file: ./#rocksdb/LOCK: Resource temporarily unavailable
2019-08-03  0:01:55 0 [ERROR] Plugin 'ROCKSDB' init function returned error.
2019-08-03  0:01:55 0 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.
2019-08-03  0:01:55 0 [ERROR] mysqld: Can't lock aria control file '/var/lib/mysql/aria_log_control' for exclusive use, error: 11. Will retry for 30 seconds
Comment 4 Илья Индиго 2019-08-03 09:38:08 UTC
-sudo echo -e '[server]\nplugin-load-add=ha_rocksdb.so' > /etc/my.cnf.d/rocksdb.cnf

+sudo echo -e '[server]\nplugin-load-add=ha_rocksdb.so' | sudo tee /etc/my.cnf.d/rocksdb.cnf
Comment 5 Kristyna Streitova 2019-08-06 12:04:56 UTC
Created attachment 812985 [details]
system-helper patch

The problem is that rocksdb is locked by another mysqld process (protected MySQL) that was started for upgrading tables after MariaDB installation.

From mysql-systemd-helper:

# Shut down MySQL
echo "Shuting down protected MySQL"
kill "$(cat "$protected/mysqld.pid")"
for i in {1..30}; do
	/usr/bin/mysqladmin --socket="$protected/mysql.sock" ping > /dev/null 2>&1 || break
done
/usr/bin/mysqladmin --socket="$protected/mysql.sock" ping > /dev/null 2>&1 && kill -9 "$(cat "$protected/mysqld.pid")"

As you can see, there is no "sleep" call so MariaDB doesn't have enough time to terminate the mysqld process before the regular mysqld process is started. I also found out that even though ping reports that mysqld is dead, the process is still present in the process list for the next few moments. This can be seen in the journalctl as well:

"mariadb.service: Found left-over process 22468 (mysqld) in control group while starting unit. Ignoring."

I've adjusted mysql-systemd-helper so now it waits 30 seconds and checks whether the PID is still there or not. You can test the patch in the attachment. If you want to force running the upgrade without reinstall, just touch .run-mysql_upgrade file in your datadir and restart mariadb.
Comment 6 Илья Индиго 2019-08-06 15:32:54 UTC
Many thanks to you, Kristyna! :-)

I checked, the patch works!
There are no more problems updating. :-)
Comment 8 Kristyna Streitova 2019-08-08 12:47:50 UTC
The fix was submitted.

|   Codestream     | Request |
|------------------|---------|
| openSUSE:Factory | #721764 |
| SLE12SP1         | #198445 |
| SLE12SP4         | #198429 |
| SLE15            | #198428 |

Closing as fixed.
Comment 10 Swamp Workflow Management 2019-08-23 19:17:00 UTC
openSUSE-RU-2019:1991-1: An update that has one recommended fix can now be installed.

Category: recommended (moderate)
Bug References: 1143215
CVE References: 
Sources used:
openSUSE Leap 15.1 (src):    mariadb-10.2.25-lp151.2.6.1
openSUSE Leap 15.0 (src):    mariadb-10.2.25-lp150.2.16.1
Comment 12 Swamp Workflow Management 2019-09-06 19:13:39 UTC
SUSE-SU-2019:2330-1: An update that solves three vulnerabilities and has one errata is now available.

Category: security (important)
Bug References: 1126088,1132666,1136035,1143215
CVE References: CVE-2019-2614,CVE-2019-2627,CVE-2019-2628
Sources used:
SUSE OpenStack Cloud Crowbar 9 (src):    mariadb-10.2.25-3.19.2
SUSE OpenStack Cloud 9 (src):    mariadb-10.2.25-3.19.2
SUSE Linux Enterprise Server 12-SP4 (src):    mariadb-10.2.25-3.19.2, mariadb-connector-c-3.1.2-2.6.6
SUSE Linux Enterprise Desktop 12-SP4 (src):    mariadb-10.2.25-3.19.2, mariadb-connector-c-3.1.2-2.6.6

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 13 Swamp Workflow Management 2019-09-25 19:11:57 UTC
SUSE-SU-2019:2461-1: An update that fixes 6 vulnerabilities is now available.

Category: security (moderate)
Bug References: 1127027,1132826,1141798,1142058,1143215
CVE References: CVE-2019-2614,CVE-2019-2627,CVE-2019-2737,CVE-2019-2739,CVE-2019-2740,CVE-2019-2805
Sources used:
SUSE OpenStack Cloud Crowbar 8 (src):    mariadb-10.0.40.1-29.32.1
SUSE OpenStack Cloud 8 (src):    mariadb-10.0.40.1-29.32.1
HPE Helion Openstack 8 (src):    mariadb-10.0.40.1-29.32.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 16 Swamp Workflow Management 2019-10-30 20:18:07 UTC
SUSE-SU-2019:2867-1: An update that solves 11 vulnerabilities and has 10 fixes is now available.

Category: security (moderate)
Bug References: 1019074,1096985,1106515,1115960,1116846,1118900,1120657,1125893,1126088,1132593,1132666,1136035,1141121,1141676,1143215,1145796,1146578,1148158,1148383,1150895,917802
CVE References: CVE-2015-3448,CVE-2016-10127,CVE-2018-15727,CVE-2018-19039,CVE-2018-558213,CVE-2019-13611,CVE-2019-15043,CVE-2019-2614,CVE-2019-2627,CVE-2019-2628,CVE-2019-5477
Sources used:
SUSE OpenStack Cloud Crowbar 8 (src):    crowbar-core-5.0+git.1569597589.1f025c557-3.32.2, crowbar-ha-5.0+git.1567673535.607aada-3.26.2, crowbar-openstack-5.0+git.1570141351.058c8bd44-4.31.2, crowbar-ui-1.2.0+git.1568396400.0344a727-3.12.3, galera-3-25.3.25-4.6.3, grafana-4.6.5-4.6.3, mariadb-10.2.25-4.14.2, mariadb-connector-c-3.1.2-3.12.3, novnc-1.0.0-3.6.3, openstack-cinder-11.2.3~dev16-3.21.4, openstack-cinder-doc-11.2.3~dev16-3.21.3, openstack-glance-15.0.3~dev3-3.12.4, openstack-glance-doc-15.0.3~dev3-3.12.3, openstack-heat-9.0.8~dev13-3.24.4, openstack-heat-doc-9.0.8~dev13-3.24.3, openstack-horizon-plugin-neutron-vpnaas-ui-1.0.1~dev3-3.6.4, openstack-keystone-12.0.4~dev4-5.27.4, openstack-keystone-doc-12.0.4~dev4-5.27.3, openstack-monasca-installer-20190923_16.32-3.9.3, openstack-neutron-11.0.9~dev51-3.24.5, openstack-neutron-doc-11.0.9~dev51-3.24.4, openstack-neutron-gbp-7.3.1~dev56-3.9.4, openstack-neutron-lbaas-11.0.4~dev6-3.15.4, openstack-neutron-lbaas-doc-11.0.4~dev6-3.15.4, openstack-nova-16.1.9~dev7-3.29.3, openstack-nova-doc-16.1.9~dev7-3.29.3, python-amqp-2.2.2-3.6.3, python-ovs-2.7.2-3.6.1, python-pysaml2-4.0.2-5.3.3, python-urllib3-1.22-5.9.3, release-notes-suse-openstack-cloud-8.20190911-3.20.3, rubygem-easy_diff-1.0.0-3.4.2
SUSE OpenStack Cloud 8 (src):    ardana-ansible-8.0+git.1566374355.c509923-3.67.3, ardana-glance-8.0+git.1566376789.be0fe01-3.17.3, ardana-horizon-8.0+git.1565816064.5d4f73f-3.18.3, ardana-input-model-8.0+git.1566517401.98450e6-3.33.3, ardana-manila-8.0+git.1568835837.2452e7a-1.21.3, ardana-neutron-8.0+git.1568220097.74ee4b4-3.33.3, ardana-nova-8.0+git.1566902754.c58ff69-3.35.3, ardana-octavia-8.0+git.1568373448.bcaee7e-3.20.3, ardana-tempest-8.0+git.1566471887.fd2fec7-3.27.3, galera-3-25.3.25-4.6.3, grafana-4.6.5-4.6.3, mariadb-10.2.25-4.14.2, mariadb-connector-c-3.1.2-3.12.3, novnc-1.0.0-3.6.3, openstack-cinder-11.2.3~dev16-3.21.4, openstack-cinder-doc-11.2.3~dev16-3.21.3, openstack-glance-15.0.3~dev3-3.12.4, openstack-glance-doc-15.0.3~dev3-3.12.3, openstack-heat-9.0.8~dev13-3.24.4, openstack-heat-doc-9.0.8~dev13-3.24.3, openstack-horizon-plugin-neutron-vpnaas-ui-1.0.1~dev3-3.6.4, openstack-keystone-12.0.4~dev4-5.27.4, openstack-keystone-doc-12.0.4~dev4-5.27.3, openstack-monasca-installer-20190923_16.32-3.9.3, openstack-neutron-11.0.9~dev51-3.24.5, openstack-neutron-doc-11.0.9~dev51-3.24.4, openstack-neutron-gbp-7.3.1~dev56-3.9.4, openstack-neutron-lbaas-11.0.4~dev6-3.15.4, openstack-neutron-lbaas-doc-11.0.4~dev6-3.15.4, openstack-nova-16.1.9~dev7-3.29.3, openstack-nova-doc-16.1.9~dev7-3.29.3, python-amqp-2.2.2-3.6.3, python-ovs-2.7.2-3.6.1, python-pysaml2-4.0.2-5.3.3, python-python-engineio-2.0.2-3.3.3, python-urllib3-1.22-5.9.3, release-notes-suse-openstack-cloud-8.20190911-3.20.3, venv-openstack-aodh-5.1.1~dev7-12.20.2, venv-openstack-barbican-5.0.2~dev3-12.21.2, venv-openstack-ceilometer-9.0.8~dev7-12.18.2, venv-openstack-cinder-11.2.3~dev16-14.21.2, venv-openstack-designate-5.0.3~dev7-12.19.2, venv-openstack-freezer-5.0.0.0~xrc2~dev2-10.16.2, venv-openstack-glance-15.0.3~dev3-12.19.2, venv-openstack-heat-9.0.8~dev13-12.21.2, venv-openstack-horizon-12.0.4~dev6-14.26.2, venv-openstack-ironic-9.1.8~dev7-12.21.2, venv-openstack-keystone-12.0.4~dev4-11.22.3, venv-openstack-magnum-5.0.2_5.0.2_5.0.2~dev31-11.20.2, venv-openstack-manila-5.1.1~dev2-12.23.2, venv-openstack-monasca-2.2.2~dev1-11.18.2, venv-openstack-monasca-ceilometer-1.5.1_1.5.1_1.5.1~dev3-8.16.2, venv-openstack-murano-4.0.2~dev2-12.16.2, venv-openstack-neutron-11.0.9~dev51-13.24.3, venv-openstack-nova-16.1.9~dev7-11.22.3, venv-openstack-octavia-1.0.6~dev2-12.21.2, venv-openstack-sahara-7.0.4~dev1-11.20.2, venv-openstack-swift-2.15.2-11.13.3, venv-openstack-trove-8.0.1~dev13-11.20.2
HPE Helion Openstack 8 (src):    ardana-ansible-8.0+git.1566374355.c509923-3.67.3, ardana-glance-8.0+git.1566376789.be0fe01-3.17.3, ardana-horizon-8.0+git.1565816064.5d4f73f-3.18.3, ardana-input-model-8.0+git.1566517401.98450e6-3.33.3, ardana-manila-8.0+git.1568835837.2452e7a-1.21.3, ardana-neutron-8.0+git.1568220097.74ee4b4-3.33.3, ardana-nova-8.0+git.1566902754.c58ff69-3.35.3, ardana-octavia-8.0+git.1568373448.bcaee7e-3.20.3, ardana-tempest-8.0+git.1566471887.fd2fec7-3.27.3, galera-3-25.3.25-4.6.3, grafana-4.6.5-4.6.3, mariadb-10.2.25-4.14.2, mariadb-connector-c-3.1.2-3.12.3, novnc-1.0.0-3.6.3, openstack-cinder-11.2.3~dev16-3.21.4, openstack-cinder-doc-11.2.3~dev16-3.21.3, openstack-glance-15.0.3~dev3-3.12.4, openstack-glance-doc-15.0.3~dev3-3.12.3, openstack-heat-9.0.8~dev13-3.24.4, openstack-heat-doc-9.0.8~dev13-3.24.3, openstack-horizon-plugin-neutron-vpnaas-ui-1.0.1~dev3-3.6.4, openstack-keystone-12.0.4~dev4-5.27.4, openstack-keystone-doc-12.0.4~dev4-5.27.3, openstack-monasca-installer-20190923_16.32-3.9.3, openstack-neutron-11.0.9~dev51-3.24.5, openstack-neutron-doc-11.0.9~dev51-3.24.4, openstack-neutron-gbp-7.3.1~dev56-3.9.4, openstack-neutron-lbaas-11.0.4~dev6-3.15.4, openstack-neutron-lbaas-doc-11.0.4~dev6-3.15.4, openstack-nova-16.1.9~dev7-3.29.3, openstack-nova-doc-16.1.9~dev7-3.29.3, python-amqp-2.2.2-3.6.3, python-pysaml2-4.0.2-5.3.3, python-python-engineio-2.0.2-3.3.3, python-urllib3-1.22-5.9.3, release-notes-hpe-helion-openstack-8.20190911-3.20.3, venv-openstack-aodh-5.1.1~dev7-12.20.2, venv-openstack-barbican-5.0.2~dev3-12.21.2, venv-openstack-ceilometer-9.0.8~dev7-12.18.2, venv-openstack-cinder-11.2.3~dev16-14.21.2, venv-openstack-designate-5.0.3~dev7-12.19.2, venv-openstack-freezer-5.0.0.0~xrc2~dev2-10.16.2, venv-openstack-glance-15.0.3~dev3-12.19.2, venv-openstack-heat-9.0.8~dev13-12.21.2, venv-openstack-horizon-hpe-12.0.4~dev6-14.26.2, venv-openstack-ironic-9.1.8~dev7-12.21.2, venv-openstack-keystone-12.0.4~dev4-11.22.3, venv-openstack-magnum-5.0.2_5.0.2_5.0.2~dev31-11.20.2, venv-openstack-manila-5.1.1~dev2-12.23.2, venv-openstack-monasca-2.2.2~dev1-11.18.2, venv-openstack-monasca-ceilometer-1.5.1_1.5.1_1.5.1~dev3-8.16.2, venv-openstack-murano-4.0.2~dev2-12.16.2, venv-openstack-neutron-11.0.9~dev51-13.24.3, venv-openstack-nova-16.1.9~dev7-11.22.3, venv-openstack-octavia-1.0.6~dev2-12.21.2, venv-openstack-sahara-7.0.4~dev1-11.20.2, venv-openstack-swift-2.15.2-11.13.3, venv-openstack-trove-8.0.1~dev13-11.20.2

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 18 Swamp Workflow Management 2019-12-11 14:35:18 UTC
SUSE-SU-2019:3270-1: An update that solves four vulnerabilities and has 6 fixes is now available.

Category: security (moderate)
Bug References: 1075812,1123053,1126088,1126428,1129729,1132666,1136035,1143215,1152916,1155089
CVE References: CVE-2017-1002201,CVE-2019-2614,CVE-2019-2627,CVE-2019-2628
Sources used:
SUSE OpenStack Cloud 7 (src):    caasp-openstack-heat-templates-1.0+git.1560518045.ad7dc6d-1.9.1, crowbar-core-4.0+git.1573109906.0f62e9503-9.57.2, crowbar-openstack-4.0+git.1573038068.1e32b3205-9.62.2, crowbar-ui-1.1.0+git.1547500033.d0fb2bf2-4.12.1, galera-3-25.3.25-11.1, mariadb-10.2.25-13.1, mariadb-connector-c-3.1.2-1.9.1, openstack-dashboard-theme-SUSE-2016.2-5.9.2, openstack-heat-templates-0.0.0+git.1515995585.81ed236-12.1, openstack-neutron-9.4.2~dev21-7.35.3, openstack-neutron-doc-9.4.2~dev21-7.35.1, openstack-nova-14.0.11~dev13-4.37.3, openstack-nova-doc-14.0.11~dev13-4.37.2, patterns-cloud-20170124-4.6.1, python-oslo.messaging-5.10.2-3.12.1, python-oslo.utils-3.16.1-3.6.1, python-pysaml2-4.0.2-3.14.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.