Devstack depends on running mysql on ubuntu 16.04

Bug #1735097 reported by Tobias Brox on 2017-11-29
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
devstack
Low
Unassigned

Bug Description

I followed the instructions at https://docs.openstack.org/devstack/latest/guides/single-vm.html to set up devstack, copied the cloud init to a local file, replaced the passwords (in case some service would barf that "'password' is a too insecure password") did a ...

    openstack server create --os-cloud=foo --image ubuntu16.04 --flavor large --key-name someprivatekey --nic net-id=bar --user-data=devstack-userdata.yaml devstackubuntu --availability-zone zoo

and eventually, not much happened.

I ran stack.sh manually, and eventually saw it carping at mysql initialization, connection refused. I ran "sudo systemctl mysql start" and tried running stack.sh again ... success ... except some password troubles.

Here is the error log from mysql after first run of stack.sh:

    2017-11-28T13:26:39.622798Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated.
Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
    2017-11-28T13:26:41.197198Z 0 [Warning] InnoDB: New log files created, LSN=45790
    2017-11-28T13:26:41.387967Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
    2017-11-28T13:26:41.479207Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: c5365d25-d43f-11e7-a51e-fa163ef80495.
    2017-11-28T13:26:41.488984Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
    2017-11-28T13:26:41.489656Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.

Here are the last output from stack.sh:

    +lib/databases/mysql:configure_database_mysql:90 sudo mysql -uroot -pHorseStaple76 -h127.0.0.1 -e 'GRANT ALL PRIVILEGES ON *.* TOoot'\''@'\''%'\'' identified by '\''HorseStaple76'\'';'
    mysql: [Warning] Using a password on the command line interface can be insecure.
    ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
    +lib/databases/mysql:configure_database_mysql:1 exit_trap
    +./stack.sh:exit_trap:493 local r=1
    ++./stack.sh:exit_trap:494 jobs -p
    +./stack.sh:exit_trap:494 jobs=
    +./stack.sh:exit_trap:497 [[ -n '' ]]
    +./stack.sh:exit_trap:503 '[' -f /tmp/tmp.WM95BUHu4T ']'
    +./stack.sh:exit_trap:504 rm /tmp/tmp.WM95BUHu4T
    +./stack.sh:exit_trap:508 kill_spinner
    +./stack.sh:kill_spinner:407 '[' '!' -z '' ']'
    +./stack.sh:exit_trap:510 [[ 1 -ne 0 ]]
    +./stack.sh:exit_trap:511 echo 'Error on exit'
    Error on exit
    +./stack.sh:exit_trap:513 type -p generate-subunit
    +./stack.sh:exit_trap:514 generate-subunit 1511878021 235 fail
    +./stack.sh:exit_trap:516 [[ -z /opt/stack/logs ]]
    +./stack.sh:exit_trap:519 /home/stack/devstack/tools/worlddump.py -d
    /opt/stack/logs
    World dumping... see /opt/stack/logs/worlddump-2017-11-28-141057.txt for details
    +./stack.sh:exit_trap:528 exit 1

Not much interesting in the worlddump file.

    $ sudo systemctl start mysql

and stack.sh completes fine.

    2017-11-28 14:35:41.904 | stack.sh completed in 1113 seconds.

No errors in mysql error log.

I supose that at some point in the code it's needed to ensure mysql is running - though I'm not sure exactly where and how to make it work properly across multiple distributions, so I'll just drop the bug report here as for now.

Tobias Brox (c-tobias) wrote :
chandan kumar (chkumar246) wrote :

Can you please retry on a fresh vm? If it still comes, feel free to comment back.

mysql is in the default list of enabled services, so something must have going wrong.
Could you share your local.conf as well please?

Tobias Brox (c-tobias) wrote :

stack@devstackubuntu:~/devstack$ cat local.conf
[[local|localrc]]
ADMIN_PASSWORD=HorseStaple76
DATABASE_PASSWORD=HorseStaple76
RABBIT_PASSWORD=HorseStaple76
SERVICE_PASSWORD=HorseStaple76
stack@devstackubuntu:~/devstack$

Tobias Brox (c-tobias) wrote :

Deleted and recreated the vm, no running mysql so far but it's still in process. Mysql has been installed and initiated:

stack@devstackubuntu:~/devstack$ sudo cat /var/log/mysql/error.log
2017-12-07T09:42:09.334830Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-12-07T09:42:10.163135Z 0 [Warning] InnoDB: New log files created, LSN=45790
2017-12-07T09:42:10.465835Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2017-12-07T09:42:10.552452Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: e5a1d0d4-db32-11e7-8ca7-fa163e348a7d.
2017-12-07T09:42:10.561139Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2017-12-07T09:42:10.561554Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.

Tobias Brox (c-tobias) wrote :

Reproduced the bug once more.

Should I run stack.sh > stdout.log and attach the stdout.log?

Tobias Brox (c-tobias) wrote :
Dr. Jens Harbott (j-harbott) wrote :

With the instructions from https://docs.openstack.org/devstack/latest/guides/single-vm.html devstack runs without errors for me. If you are still seeing this issue, please provide full steps to reproduce, including a copy of your userdata file and the full log from /var/log/cloud-init-output.log

Changed in devstack:
status: New → Incomplete
Tobias Brox (c-tobias) wrote :
Download full text (4.8 KiB)

Still no dice for me. Weird.

Here is the cloud-config - copied directly from https://docs.openstack.org/devstack/latest/guides/single-vm.html this time, even without replacing the password "password".

    $ cat > devstack-userdata.yaml
    #cloud-config

    users:
      - default
      - name: stack
        lock_passwd: False
        sudo: ["ALL=(ALL) NOPASSWD:ALL\nDefaults:stack !requiretty"]
        shell: /bin/bash

    write_files:
      - content: |
            #!/bin/sh
            DEBIAN_FRONTEND=noninteractive sudo apt-get -qqy update || sudo yum update -qy
            DEBIAN_FRONTEND=noninteractive sudo apt-get install -qqy git || sudo yum install -qy git
            sudo chown stack:stack /home/stack
            cd /home/stack
            git clone https://git.openstack.org/openstack-dev/devstack
            cd devstack
            echo '[[local|localrc]]' > local.conf
            echo ADMIN_PASSWORD=password >> local.conf
            echo DATABASE_PASSWORD=password >> local.conf
            echo RABBIT_PASSWORD=password >> local.conf
            echo SERVICE_PASSWORD=password >> local.conf
            ./stack.sh
        path: /home/stack/start.sh
        permissions: 0755

    runcmd:
      - su -l stack ./start.sh

Here is how I launched the VM:

    $ openstack server create --os-cloud=ms1-test --image ubuntu16.04 --flavor large --key-name someprivatekey --nic net-id=3d2ec639-098b-47f8-896c-bc67cf321a2b --user-data=devstack-userdata.yaml devstackubuntu --availability-zone osl3
    +-----------------------------+------------------------------------------------------------------+
    | Field | Value |
    +-----------------------------+------------------------------------------------------------------+
    | OS-DCF:diskConfig | MANUAL |
    | OS-EXT-AZ:availability_zone | osl3 |
    | OS-EXT-STS:power_state | NOSTATE |
    | OS-EXT-STS:task_state | scheduling |
    | OS-EXT-STS:vm_state | building |
    | OS-SRV-USG:launched_at | None |
    | OS-SRV-USG:terminated_at | None |
    | accessIPv4 | |
    | accessIPv6 | |
    | addresses | |
    | adminPass | aB6ZSw6hWa9T |
    | config_drive | ...

Read more...

Tobias Brox (c-tobias) wrote :
Tobias Brox (c-tobias) wrote :

Sorry for not gzipping the output log.

I do see mysql-server-5.7 being installed, early on. Then it seems like the last lines of the log is most relevant:

+ ./stack.sh:main:1011 : is_service_enabled mysql postgresql
+ functions-common:is_service_enabled:1881 : local xtrace
++ functions-common:is_service_enabled:1882 : set +o
++ functions-common:is_service_enabled:1882 : grep xtrace
+ functions-common:is_service_enabled:1882 : xtrace='set -o xtrace'
+ functions-common:is_service_enabled:1883 : set +o xtrace
+ functions-common:is_service_enabled:1911 : return 0
+ ./stack.sh:main:1012 : configure_database
+ lib/database:configure_database:128 : configure_database_mysql
+ lib/databases/mysql:configure_database_mysql:61 : local my_conf mysql slow_log
+ lib/databases/mysql:configure_database_mysql:62 : echo_summary 'Configuring and starting MySQL'
+ ./stack.sh:echo_summary:434 : [[ -t 3 ]]
+ ./stack.sh:echo_summary:440 : echo -e Configuring and starting MySQL
2018-03-19 13:36:14.499 | Configuring and starting MySQL
+ lib/databases/mysql:configure_database_mysql:64 : is_ubuntu
+ functions-common:is_ubuntu:465 : [[ -z deb ]]
+ functions-common:is_ubuntu:468 : '[' deb = deb ']'
+ lib/databases/mysql:configure_database_mysql:65 : my_conf=/etc/mysql/my.cnf
+ lib/databases/mysql:configure_database_mysql:79 : is_fedora
+ functions-common:is_fedora:438 : [[ -z Ubuntu ]]
+ functions-common:is_fedora:442 : '[' Ubuntu = Fedora ']'
+ functions-common:is_fedora:442 : '[' Ubuntu = 'Red Hat' ']'
+ functions-common:is_fedora:443 : '[' Ubuntu = RedHatEnterpriseServer ']'
+ functions-common:is_fedora:444 : '[' Ubuntu = CentOS ']'
+ functions-common:is_fedora:444 : '[' Ubuntu = OracleServer ']'
+ functions-common:is_fedora:445 : '[' Ubuntu = Virtuozzo ']'
+ lib/databases/mysql:configure_database_mysql:79 : is_suse
+ functions-common:is_suse:453 : [[ -z Ubuntu ]]
+ functions-common:is_suse:457 : [[ Ubuntu =~ (openSUSE) ]]
+ functions-common:is_suse:457 : [[ Ubuntu == \S\U\S\E\ \L\I\N\U\X ]]
+ lib/databases/mysql:configure_database_mysql:87 : sudo mysqladmin -u root password password
^Gmysqladmin: connect to server at '127.0.0.1' failed
error: 'Can't connect to MySQL server on '127.0.0.1' (111)'
Check that mysqld is running on 127.0.0.1 and that the port is 3306.
You can check this by doing 'telnet 127.0.0.1 3306'
+ lib/databases/mysql:configure_database_mysql:87 : true
+ lib/databases/mysql:configure_database_mysql:90 : sudo mysql -uroot -ppassword -h127.0.0.1 -e 'GRANT ALL PRIVILEGES ON *.* TO '\''root'\''@'\''%'\'' identified by '\''password'\'';'
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
+ lib/databases/mysql:configure_database_mysql:1 : exit_trap

Tobias Brox (c-tobias) wrote :

Here is what's happening around line 80 in lib/databases/mysql:

    # Start mysql-server
    if is_fedora || is_suse; then
        # service is not started by default
        start_service $MYSQL_SERVICE_NAME
    fi

    # Set the root password - only works the first time. For Ubuntu, we already
    # did that with debconf before installing the package, but we still try,
    # because the package might have been installed already.
    sudo mysqladmin -u root password $DATABASE_PASSWORD || true

So, if we're on Ubuntu it asserts mysql was already started as it was installed.

The logs says ...

Selecting previously unselected package mysql-server-core-5.7.
Preparing to unpack .../mysql-server-core-5.7_5.7.21-0ubuntu0.16.04.1_amd64.deb ...
Unpacking mysql-server-core-5.7 (5.7.21-0ubuntu0.16.04.1) ...
Selecting previously unselected package mysql-server-5.7.
Preparing to unpack .../mysql-server-5.7_5.7.21-0ubuntu0.16.04.1_amd64.deb ...
Unpacking mysql-server-5.7 (5.7.21-0ubuntu0.16.04.1) ...
Selecting previously unselected package mysql-server.
Preparing to unpack .../mysql-server_5.7.21-0ubuntu0.16.04.1_all.deb ...
Unpacking mysql-server (5.7.21-0ubuntu0.16.04.1) ...
Setting up mysql-server-core-5.7 (5.7.21-0ubuntu0.16.04.1) ...
Setting up libevent-core-2.0-5:amd64 (2.0.21-stable-2ubuntu0.16.04.1) ...
Setting up mysql-server-5.7 (5.7.21-0ubuntu0.16.04.1) ...
Setting up mysql-server (5.7.21-0ubuntu0.16.04.1) ...

... but nothing about starting mysql-server.

I'd suggest replacing the "if suse || redhat; then start mysql" with "if mysql not running; then start mysql" or something similar, that ought to be a more robust check. Or just try to start mysql no matter if it's already running or not.

Changed in devstack:
status: Incomplete → New
Dr. Jens Harbott (j-harbott) wrote :

The only difference I am seeing compared to my setup is that your version of cloud-init looks pretty old, which makes me assume that you are using a rather old version of the 16.04 cloud image. Would it be possible for you to try with a current version?

Tobias Brox (c-tobias) wrote :

Indeed, it seems like the 16.04-image is pretty old.

Still, it seems more robust to me to replace the passage ...

    if is_fedora || is_suse; then
        # service is not started by default
        start_service $MYSQL_SERVICE_NAME
    fi

with something like

    start_service $MYSQL_SERVICE_NAME || true

Changed in devstack:
status: New → Triaged
importance: Undecided → Low

I got this on Ubuntu 18.04.

This happens when HOST_IP=192.168.1.n and MYSQL_HOST="".

The solution was to fix my local.conf setting both HOST_IP and MYSQL_HOST to same value.

Actually I am wrong because I still have the issue even when HOST_IP and MYSQL_HOST have same value. The value 127.0.0.1 is hardcoded in ./lib/databases/mysql

    # Update the DB to give user '$DATABASE_USER'@'%' full control of the all databases:
    sudo mysql -uroot -p$DATABASE_PASSWORD -h127.0.0.1 -e "GRANT ALL PRIVILEGES ON *.* TO '$DATABASE_USER'@'%' identified by '$DATABASE_PASSWORD';"

There is error in /var/log/syslog (similar to https://bugs.launchpad.net/ubuntu/+source/mysql-5.5/+bug/1637280). I had MariaDB installed before running devstack. Devstack purges mariadb* mysql* but maybe some residual configuration was left. will test further.

Dec 8 16:41:15 ubuntu1804 mysql-systemd-start[25028]: 2018-12-09T00:41:15.256901Z 0 [ERROR] Failed to access directory for --secure-file-priv. Please make sure that directory exists and is accessible by MySQL Server. Supplied value : /var/lib/mysql-files

To confirm this problem arises when you install mariaDb before running devstack - to avoid this issue don't install mariadb before you run devstack - I no longer see the issue.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers