Failin on mysql connection

[Martin Mágr]

My overcloud deployment is failing on mysql connection:

TASK [Debug output for task: Run docker-puppet tasks (bootstrap tasks) for step 2] ***
Friday 25 January 2019 10:24:16 +0000 (0:08:01.206) 0:44:16.955 ********
fatal: [overcloud-controller-0]: FAILED! => {
    "failed_when_result": true,
    "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
        "2019-01-25 10:16:14,806 INFO: 114942 -- Running docker-puppet",
        "2019-01-25 10:16:14,806 INFO: 114942 -- Service compilation completed.",
        "2019-01-25 10:16:14,807 INFO: 114942 -- Starting multiprocess configuration steps. Using 2 processes.",
        "2019-01-25 10:16:14,812 INFO: 114946 -- Starting configuration of rabbit_init_tasks using image 192.168.24.1:8787/tripleomaster/centos-binary-rabbitmq:current-tripleo-rdo",
        "2019-01-25 10:16:14,812 INFO: 114945 -- Starting configuration of mysql_init_tasks using image 192.168.24.1:8787/tripleomaster/centos-binary-mariadb:current-tripleo-rdo",
        "2019-01-25 10:16:14,929 INFO: 114946 -- Removing container: docker-puppet-rabbit_init_tasks",
        "2019-01-25 10:16:14,945 INFO: 114945 -- Removing container: docker-puppet-mysql_init_tasks",
        "2019-01-25 10:16:15,140 INFO: 114946 -- Image already exists: 192.168.24.1:8787/tripleomaster/centos-binary-rabbitmq:current-tripleo-rdo",
        "2019-01-25 10:16:15,141 INFO: 114945 -- Image already exists: 192.168.24.1:8787/tripleomaster/centos-binary-mariadb:current-tripleo-rdo",
        "2019-01-25 10:18:52,965 WARNING: 114946 -- + mkdir -p /etc/puppet",
        "+ cp -dR /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/hieradata /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet",
        "+ rm -Rf /etc/puppet/ssl",
        "+ echo '{\"step\": 2}'",
        "+ TAGS=",
        "+ '[' -n file,file_line,concat,augeas,cron,rabbitmq_policy,rabbitmq_user ']'",
        "+ TAGS='--tags file,file_line,concat,augeas,cron,rabbitmq_policy,rabbitmq_user'",
        "+ CHECK_MODE=",
        "+ '[' -d /tmp/puppet-check-mode ']'",
        "+ origin_of_time=/var/lib/config-data/rabbit_init_tasks.origin_of_time",
        "+ touch /var/lib/config-data/rabbit_init_tasks.origin_of_time",
        "+ sync",
        "+ export NET_HOST=true",
        "+ NET_HOST=true",
        "+ set +e",
        "+ '[' true == false ']'",
        "+ export FACTER_deployment_type=containers",
        "+ FACTER_deployment_type=containers",
        "++ tr '[:upper:]' '[:lower:]'",
        "++ cat /sys/class/dmi/id/product_uuid",
        "+ export FACTER_uuid=db47a556-4b8e-4cfe-97b7-6fac99e7bf87",
        "+ FACTER_uuid=db47a556-4b8e-4cfe-97b7-6fac99e7bf87",
        "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,rabbitmq_policy,rabbitmq_user /etc/config.pp",
        "Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5",
        " (file: /etc/puppet/hiera.yaml)",
        "Warning: Undefined variable '::deploy_config_name'; \\n (file & line not available)",
        "Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_language.html\\n (file & line not available)",
        "Warning: ModuleLoader: module 'rabbitmq' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
        "+ rc=2",
        "+ set -e",
        "+ '[' 2 -ne 2 -a 2 -ne 0 ']'",
        "+ '[' -z true ']'",
        "",
        "2019-01-25 10:18:52,966 INFO: 114946 -- Removing container: docker-puppet-rabbit_init_tasks",
        "2019-01-25 10:18:53,048 INFO: 114946 -- Finished processing puppet configs for rabbit_init_tasks",
        "2019-01-25 10:18:55,610 WARNING: 114945 -- ['/usr/bin/podman', 'run', '--user', 'root', '--name', u'docker-puppet-mysql_init_tasks', '--env', u'PUPPET_TAGS=file,file_line,concat,augeas,cron,mysql_database,mysql_grant,mysql_user', '--env', u'NAME=mysql_init_tasks', '--env', u'HOSTNAME=overcloud-controller-0', '--env', 'NO_ARCHIVE=true', '--env', 'STEP=2', '--env', 'NET_HOST=true', '--volume', '/etc/localtime:/etc/localtime:ro', '--volume', '/tmp/tmpr6AL71:/etc/config.pp:ro', '--volume', '/etc/puppet/:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume', '/var/lib/config-data:/var/lib/config-data/:rw', '--volume', '/dev/log:/dev/log:rw', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--volume', u'/var/lib/mysql:/var/lib/mysql/:rw', '--volume', u'/var/log/containers/mysql:/var/log/mariadb', '--volume', u'/var/lib/config-data/mysql/root:/root:rw', '--entrypoint', '/var/lib/docker-puppet/docker-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/docker-puppet/docker-puppet.sh:/var/lib/docker-puppet/docker-puppet.sh:ro', u'192.168.24.1:8787/tripleomaster/centos-binary-mariadb:current-tripleo-rdo'] run failed after + mkdir -p /etc/puppet",
        "+ '[' -n file,file_line,concat,augeas,cron,mysql_database,mysql_grant,mysql_user ']'",
        "+ TAGS='--tags file,file_line,concat,augeas,cron,mysql_database,mysql_grant,mysql_user'",
        "+ origin_of_time=/var/lib/config-data/mysql_init_tasks.origin_of_time",
        "+ touch /var/lib/config-data/mysql_init_tasks.origin_of_time",
        "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,mysql_database,mysql_grant,mysql_user /etc/config.pp",
        "Warning: This method is deprecated, please use the stdlib validate_legacy function,",
        " with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql.pp\", 119]:[\"/etc/config.pp\", 2]",
        " (location: /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')",
        "Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
        "Warning: ModuleLoader: module 'aodh' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
        " with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/aodh/manifests/db/mysql.pp\", 48]:[\"/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql.pp\", 200]",
        "Warning: ModuleLoader: module 'cinder' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
        "Warning: ModuleLoader: module 'glance' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
        "Warning: ModuleLoader: module 'gnocchi' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
        "Warning: ModuleLoader: module 'heat' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
        "Warning: ModuleLoader: module 'keystone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
        "Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
        "Warning: ModuleLoader: module 'nova' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
        "Warning: ModuleLoader: module 'panko' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
        "Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
        " with Pattern[]. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/openstacklib/manifests/db/mysql/host_access.pp\", 43]:",
        "Error: Failed to apply catalog: Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)",
        "+ rc=1",
        "+ '[' 1 -ne 2 -a 1 -ne 0 ']'",
        "+ exit 1",
        " attempt(s): 1",
        "2019-01-25 10:18:55,610 WARNING: 114945 -- Retrying running container: mysql_init_tasks",
        "2019-01-25 10:21:35,135 WARNING: 114945 -- ['/usr/bin/podman', 'start', '-a', u'docker-puppet-mysql_init_tasks'] run failed after + mkdir -p /etc/puppet",
        " attempt(s): 2",
        "2019-01-25 10:21:35,136 WARNING: 114945 -- Retrying running container: mysql_init_tasks",
        "2019-01-25 10:24:15,393 WARNING: 114945 -- ['/usr/bin/podman', 'start', '-a', u'docker-puppet-mysql_init_tasks'] run failed after + mkdir -p /etc/puppet",
        " attempt(s): 3",
        "2019-01-25 10:24:15,394 WARNING: 114945 -- Retrying running container: mysql_init_tasks",
        "2019-01-25 10:24:15,395 ERROR: 114945 -- Failed running container for mysql_init_tasksException occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 28, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 945, in take_action
    verbosity=self.app_args.verbose_level)
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 307, in config_download
    raise exceptions.DeploymentError("Overcloud configuration failed.")
DeploymentError: Overcloud configuration failed.
Overcloud configuration failed.
",
        "2019-01-25 10:24:15,395 INFO: 114945 -- Finished processing puppet configs for mysql_init_tasks",
        "2019-01-25 10:24:15,396 ERROR: 114942 -- ERROR configuring mysql_init_tasks"
    ]
}
skipping: [overcloud-novacompute-0] => {}

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
overcloud-controller-0 : ok=220 changed=71 unreachable=0 failed=1
overcloud-novacompute-0 : ok=162 changed=41 unreachable=0 failed=0
undercloud

--------------------------

Deployed master-tripleo-ci release via quickstart. Overcloud deploy is:
(undercloud) [stack@undercloud ~]$ ./overcloud-deploy.sh -e config.yaml
(undercloud) [stack@undercloud ~]$ cat config.yaml
## A Heat environment file which can be used to set up monitoring agents

resource_registry:
  OS::TripleO::Services::SensuClient: /usr/share/openstack-tripleo-heat-templates/docker/services/sensu-client.yaml

parameter_defaults:
  MonitoringRabbitHost: 10.10.10.10
  MonitoringRabbitPort: 5672
  MonitoringRabbitUserName: sensu
  MonitoringRabbitPassword: sensu

  ContainerCli: podman
  DockerInsecureRegistryAddress:
    - 192.168.24.1:8787
(undercloud) [stack@undercloud ~]$

[Rabi Mishra (rabi)]

Though may not be the same, I think this happens if you've a already running mysql container on the overcloud node and you're trying do a fresh deploy.

I had seen that after doing openstack overcloud delete and then running overcloud-deploy.sh again.

It seems we only bootstrap mysql[1], if /var/lib/mysql/mysql does not exist, which IMO is not correct as that can be from the previous deployment and we need the old passwords to reset the mysql root password which is re-generated for the new deployment.

Cleaning up the node and deploying works fine. I do the following to cleanup the node.

1. sudo systemctl stop $(ls /etc/systemd/system/tripleo_* | xargs -n 1 basename)
2. sudo podman rm $(sudo podman -aq)
3. sudo rm -Rf /var/lib/mysql
4. sudo rm -Rf /var/lib/rabbitmq

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/database/mysql.yaml#L148

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/633863

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rabi Mishra (<email address hidden>) on branch: master
Review: https://review.openstack.org/633863
Reason: Abandoned in favor of https://review.openstack.org/#/c/633768

[Rabi Mishra (rabi)]

Fixed with https://review.openstack.org/#/c/633768/