Failin on mysql connection
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Critical
|
Rabi Mishra |
Bug Description
My overcloud deployment is failing on mysql connection:
TASK [Debug output for task: Run docker-puppet tasks (bootstrap tasks) for step 2] ***
Friday 25 January 2019 10:24:16 +0000 (0:08:01.206) 0:44:16.955 ********
fatal: [overcloud-
"failed_
"outputs.
"2019-01-25 10:16:14,806 INFO: 114942 -- Running docker-puppet",
"2019-01-25 10:16:14,806 INFO: 114942 -- Service compilation completed.",
"2019-01-25 10:16:14,807 INFO: 114942 -- Starting multiprocess configuration steps. Using 2 processes.",
"2019-01-25 10:16:14,812 INFO: 114946 -- Starting configuration of rabbit_init_tasks using image 192.168.
"2019-01-25 10:16:14,812 INFO: 114945 -- Starting configuration of mysql_init_tasks using image 192.168.
"2019-01-25 10:16:14,929 INFO: 114946 -- Removing container: docker-
"2019-01-25 10:16:14,945 INFO: 114945 -- Removing container: docker-
"2019-01-25 10:16:15,140 INFO: 114946 -- Image already exists: 192.168.
"2019-01-25 10:16:15,141 INFO: 114945 -- Image already exists: 192.168.
"2019-01-25 10:18:52,965 WARNING: 114946 -- + mkdir -p /etc/puppet",
"+ cp -dR /tmp/puppet-
"+ rm -Rf /etc/puppet/ssl",
"+ echo '{\"step\": 2}'",
"+ TAGS=",
"+ '[' -n file,file_
"+ TAGS='--tags file,file_
"+ CHECK_MODE=",
"+ '[' -d /tmp/puppet-
"+ origin_
"+ touch /var/lib/
"+ sync",
"+ export NET_HOST=true",
"+ NET_HOST=true",
"+ set +e",
"+ '[' true == false ']'",
"+ export FACTER_
"+ FACTER_
"++ tr '[:upper:]' '[:lower:]'",
"++ cat /sys/class/
"+ export FACTER_
"+ FACTER_
"+ /usr/bin/puppet apply --summarize --detailed-
"Warning: /etc/puppet/
" (file: /etc/puppet/
"Warning: Undefined variable '::deploy_
"Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https:/
"Warning: ModuleLoader: module 'rabbitmq' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
"+ rc=2",
"+ set -e",
"+ '[' 2 -ne 2 -a 2 -ne 0 ']'",
"+ '[' -z true ']'",
"",
"2019-01-25 10:18:52,966 INFO: 114946 -- Removing container: docker-
"2019-01-25 10:18:53,048 INFO: 114946 -- Finished processing puppet configs for rabbit_init_tasks",
"2019-01-25 10:18:55,610 WARNING: 114945 -- ['/usr/bin/podman', 'run', '--user', 'root', '--name', u'docker-
"+ '[' -n file,file_
"+ TAGS='--tags file,file_
"+ origin_
"+ touch /var/lib/
"+ /usr/bin/puppet apply --summarize --detailed-
"Warning: This method is deprecated, please use the stdlib validate_legacy function,",
" with Stdlib:
" (location: /etc/puppet/
"Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
"Warning: ModuleLoader: module 'aodh' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
" with Stdlib:
"Warning: ModuleLoader: module 'cinder' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
"Warning: ModuleLoader: module 'glance' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
"Warning: ModuleLoader: module 'gnocchi' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
"Warning: ModuleLoader: module 'heat' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
"Warning: ModuleLoader: module 'keystone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
"Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
"Warning: ModuleLoader: module 'nova' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
"Warning: ModuleLoader: module 'panko' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
"Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
" with Pattern[]. There is further documentation for validate_legacy function in the README. at [\"/etc/
"Error: Failed to apply catalog: Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)",
"+ rc=1",
"+ '[' 1 -ne 2 -a 1 -ne 0 ']'",
"+ exit 1",
" attempt(s): 1",
"2019-01-25 10:18:55,610 WARNING: 114945 -- Retrying running container: mysql_init_tasks",
"2019-01-25 10:21:35,135 WARNING: 114945 -- ['/usr/bin/podman', 'start', '-a', u'docker-
" attempt(s): 2",
"2019-01-25 10:21:35,136 WARNING: 114945 -- Retrying running container: mysql_init_tasks",
"2019-01-25 10:24:15,393 WARNING: 114945 -- ['/usr/bin/podman', 'start', '-a', u'docker-
" attempt(s): 3",
"2019-01-25 10:24:15,394 WARNING: 114945 -- Retrying running container: mysql_init_tasks",
"2019-01-25 10:24:15,395 ERROR: 114945 -- Failed running container for mysql_init_
Traceback (most recent call last):
File "/usr/lib/
super(Command, self).run(
File "/usr/lib/
return super(Command, self).run(
File "/usr/lib/
return_code = self.take_
File "/usr/lib/
verbosity=
File "/usr/lib/
raise exceptions.
DeploymentError: Overcloud configuration failed.
Overcloud configuration failed.
",
"2019-01-25 10:24:15,395 INFO: 114945 -- Finished processing puppet configs for mysql_init_tasks",
"2019-01-25 10:24:15,396 ERROR: 114942 -- ERROR configuring mysql_init_tasks"
]
}
skipping: [overcloud-
NO MORE HOSTS LEFT *******
PLAY RECAP *******
overcloud-
overcloud-
undercloud
-------
Deployed master-tripleo-ci release via quickstart. Overcloud deploy is:
(undercloud) [stack@undercloud ~]$ ./overcloud-
(undercloud) [stack@undercloud ~]$ cat config.yaml
## A Heat environment file which can be used to set up monitoring agents
resource_registry:
OS::TripleO:
parameter_defaults:
MonitoringRab
MonitoringRab
MonitoringRab
MonitoringRab
ContainerCli: podman
DockerInsecur
- 192.168.24.1:8787
(undercloud) [stack@undercloud ~]$
Changed in tripleo: | |
importance: | Undecided → Critical |
Though may not be the same, I think this happens if you've a already running mysql container on the overcloud node and you're trying do a fresh deploy.
I had seen that after doing openstack overcloud delete and then running overcloud-deploy.sh again.
It seems we only bootstrap mysql[1], if /var/lib/ mysql/mysql does not exist, which IMO is not correct as that can be from the previous deployment and we need the old passwords to reset the mysql root password which is re-generated for the new deployment.
Cleaning up the node and deploying works fine. I do the following to cleanup the node.
1. sudo systemctl stop $(ls /etc/systemd/ system/ tripleo_ * | xargs -n 1 basename)
2. sudo podman rm $(sudo podman -aq)
3. sudo rm -Rf /var/lib/mysql
4. sudo rm -Rf /var/lib/rabbitmq
[1] https:/ /github. com/openstack/ tripleo- heat-templates/ blob/master/ docker/ services/ database/ mysql.yaml# L148