neutron-ovn-tempest-ovs-master-fedora failing 100% of the time since 12/13/2021

[Miguel Lavalle]

Job neutron-ovn-tempest-ovs-master-fedora is failing because devstack fails to build. Specifically, set the root user password in mariadb fails:

2021-12-31 03:01:18.052 | + lib/databases/mysql:configure_database_mysql:97 : sudo mysqladmin -u root password secretdatabase
2021-12-31 03:01:18.118 | mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2021-12-31 03:01:18.118 | Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety.
2021-12-31 03:01:18.118 | mysqladmin: unable to change password; error: 'You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '() IDENTIFIED BY 'secretdatabase'' at line 1'

which in turn leads to the following failure:

2021-12-31 03:01:18.140 | + lib/databases/mysql:configure_database_mysql:115 : sudo mysql -uroot -psecretdatabase -h127.0.0.1 -e 'CREATE USER IF NOT EXISTS '\''root'\''@'\''%'\'' identified by '\''secretdatabase'\'';'
2021-12-31 03:01:18.200 | mysql: [Warning] Using a password on the command line interface can be insecure.
2021-12-31 03:01:18.204 | ERROR 1698 (28000): Access denied for user 'root'@'localhost'
2021-12-31 03:01:18.209 | + lib/databases/mysql:configure_database_mysql:1 : exit_trap

This terminates the execution of devstack/stack.sh

This job currently runs on Fedora 34

[Miguel Lavalle (minsel)]

I am able to reproduce this failure in a local vm. After further investigation, I determined that for command:

sudo mysqladmin -u root password secretdatabase

to execute successfully, the following link must exist:

$ ls -l /usr/bin/mysqladmin
lrwxrwxrwx. 1 root root 13 Dec 2 13:44 /usr/bin/mysqladmin -> mariadb-admin

However, community-mysql is a dependency of mariadb-devel/server:

dnf deplist mariadb-server | grep mysql

.....

  dependency: (mysql-selinux if selinux-policy-targeted)
   provider: mysql-selinux-1.0.4-2.fc34.noarch
  dependency: mysql(x86-64)
   provider: community-mysql-8.0.27-1.fc34.x86_64
  dependency: mysql-compat-client(x86-64)
   provider: community-mysql-8.0.27-1.fc34.x86_64
  dependency: (mysql-selinux if selinux-policy-targeted)
   provider: mysql-selinux-1.0.4-2.fc34.noarch
  dependency: mysql(x86-64)
   provider: community-mysql-8.0.27-1.fc34.x86_64
  dependency: mysql-compat-client(x86-64)
   provider: community-mysql-8.0.27-1.fc34.x86_64
......

This causes a clash that prevents mariadb-admin from being installed and mysqladmin is installed from mysql community as follows:

$ ls /usr/bin/mysqladmin -l
-rwxr-xr-x. 1 root root 6738168 Nov 1 01:52 /usr/bin/mysqladmin

I initiated a thread in the mariadb mailing list: https://lists.launchpad.net/maria-discuss/msg06179.html. There is a bug that describes this situation: https://bugzilla.redhat.com/show_bug.cgi?id=2026933

The recommended work around from the mariadb mailing list is to explicitly install the "mariadb" package

[Miguel Lavalle (minsel)]

Proposed fix: https://review.opendev.org/c/openstack/devstack/+/823218

[Miguel Lavalle (minsel)]

Fix was merged and job is green again: https://zuul.opendev.org/t/openstack/build/a063fb7cd33e4f858d42e5613b246f64