Running MariaDB bootstrap container: Container exited with non-zero return code

[olr]

I'm trying to deploy an all-in-one Openstack (neutron, horizon,glance,keystone,nova) with Kolla-ansible an a single desktop.

I closely followed this procedure: https://docs.openstack.org/kolla-ansible/latest/user/quickstart.html

I got this error:

TASK [mariadb : Running MariaDB bootstrap container] ***************************************************************************************
task path: /usr/share/kolla-ansible/ansible/roles/mariadb/tasks/bootstrap_cluster.yml:2
Using module file /usr/share/kolla-ansible/ansible/library/kolla_docker.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: olr
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1520662078.92-32759724646146 `" && echo ansible-tmp-1520662078.92-32759724646146="` echo /root/.ansible/tmp/ansible-tmp-1520662078.92-32759724646146 `" ) && sleep 0'
<localhost> PUT /tmp/tmpyCb5HR TO /root/.ansible/tmp/ansible-tmp-1520662078.92-32759724646146/kolla_docker.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1520662078.92-32759724646146/ /root/.ansible/tmp/ansible-tmp-1520662078.92-32759724646146/kolla_docker.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1520662078.92-32759724646146/kolla_docker.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1520662078.92-32759724646146/" > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": true,
    "invocation": {
        "module_args": {
            "action": "start_container",
            "api_version": "auto",
            "auth_email": null,
            "auth_password": null,
            "auth_registry": null,
            "auth_username": null,
            "cap_add": [],
            "detach": false,
            "environment": {
                "DB_MAX_TIMEOUT": 60,
                "DB_ROOT_PASSWORD": "r0af3BWbDySQuam9VDC8u2AEIE14YGJvozxDUIOr",
                "KOLLA_BOOTSTRAP": null,
                "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS",
                "KOLLA_SERVICE_NAME": "bootstrap-mariadb"
            },
            "graceful_timeout": 10,
            "image": "kolla/centos-binary-mariadb:pike",
            "labels": {
                "BOOTSTRAP": null
            },
            "name": "bootstrap_mariadb",
            "privileged": false,
            "remove_on_exit": true,
            "restart_policy": "never",
            "restart_retries": 10,
            "security_opt": [],
            "tls_cacert": null,
            "tls_cert": null,
            "tls_key": null,
            "tls_verify": false,
            "volumes": [
                "/etc/kolla//mariadb/:/var/lib/kolla/config_files/:ro",
                "/etc/localtime:/etc/localtime:ro",
                "kolla_logs:/var/log/kolla/",
                "mariadb:/var/lib/mysql"
            ],
            "volumes_from": null
        }
    },
    "msg": "Container exited with non-zero return code"
}

RUNNING HANDLER [mariadb : restart slave mariadb] ******************************************************************************************

RUNNING HANDLER [mariadb : restart master mariadb] *****************************************************************************************
 to retry, use: --limit @/usr/share/kolla-ansible/ansible/site.retry

PLAY RECAP *********************************************************************************************************************************
localhost : ok=41 changed=23 unreachable=0 failed=1

Command failed ansible-playbook -i all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy /usr/share/kolla-ansible/ansible/site.yml --verbose --verbose --verbose

-----------------------------------------------------------------------------------------

I checked logs via: docker logs bootstrap_mariadb

INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc//my.cnf
INFO:__main__:Copying /var/lib/kolla/config_files/galera.cnf to /etc//my.cnf
INFO:__main__:Setting permission for /etc//my.cnf
INFO:__main__:Copying /var/lib/kolla/config_files/wsrep-notify.sh to /usr/local/bin/wsrep-notify.sh
INFO:__main__:Setting permission for /usr/local/bin/wsrep-notify.sh
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/lib/mysql
Installing MariaDB/MySQL system tables in '/var/lib/mysql/' ...
180310 1:08:00 [Note] /usr/sbin/mysqld (mysqld 10.0.34-MariaDB-wsrep) starting as process 27 ...
OK
Filling help tables...
180310 1:08:04 [Note] /usr/sbin/mysqld (mysqld 10.0.34-MariaDB-wsrep) starting as process 57 ...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
To do so, start the server, then issue the following commands:

'/usr/bin/mysqladmin' -u root password 'new-password'
'/usr/bin/mysqladmin' -u root -h localhost password 'new-password'

Alternatively you can run:
'/usr/bin/mysql_secure_installation'

which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.

See the MariaDB Knowledgebase at http://mariadb.com/kb or the
MySQL manual for more instructions.

You can start the MariaDB daemon with:
cd '/usr' ; /usr/bin/mysqld_safe --datadir='/var/lib/mysql/'

You can test the MariaDB daemon with mysql-test-run.pl
cd '/usr/mysql-test' ; perl mysql-test-run.pl

Please report any problems at http://mariadb.org/jira

The latest information about MariaDB is available at http://mariadb.org/.
You can find additional information about the MySQL part at:
http://dev.mysql.com
Consider joining MariaDB's strong and vibrant community:
https://mariadb.org/get-involved/

180310 01:08:08 mysqld_safe Logging to '/var/log/kolla/mariadb/mariadb.log'.
180310 01:08:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
spawn mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!

By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y
 ... Success!

Cleaning up...

All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

-----------------------------------------------------------

Any help please?

[Chris L (onyx4)]

This is most likely the same issue as https://bugs.launchpad.net/kolla-ansible/+bug/1748194

Just keep re-running the kolla-ansible -i all-in-one reconfigure

and it will fail at the next container until it's done with all the containers.

This is a very annoying bug and I'm hitting this on both Ubuntu and Centos deployments with the latest pike release

[Chris L (onyx4)]

Running the command below fixes the issue: It's due to an incompatibility with the new 3.0 python docker package.

# pip install docker==2.7.0