Failed running docker-puppet.py for mysql_init_tasks

Bug #1779900 reported by Harald Jensås on 2018-07-03
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
High
Unassigned

Bug Description

Consistently getting this error when deploying 1x Controller 1x Compute.

Mysql log says "Can't start server: listen() on TCP/IP port: Address already in use"

2018-07-03 17:32:30,329 p=13495 u=mistral | fatal: [overcloud-controller-0]: FAILED! => {
    "failed_when_result": true,
    "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
        "2018-07-03 15:32:18,962 INFO: 30319 -- Running docker-puppet",
        "2018-07-03 15:32:18,962 INFO: 30319 -- Service compilation completed.",
        "2018-07-03 15:32:18,963 INFO: 30319 -- Starting multiprocess configuration steps. Using 4 processes.",
        "2018-07-03 15:32:18,973 INFO: 30320 -- Starting configuration of mysql_init_tasks using image docker.io/tripleomaster/centos-binary-mariadb:9f3a41c2c752bd68a84b6bd72add18e8bc6f4c76_f1a4ee5a",
        "2018-07-03 15:32:18,975 INFO: 30320 -- Removing container: docker-puppet-mysql_init_tasks",
        "2018-07-03 15:32:19,017 INFO: 30320 -- Image already exists: docker.io/tripleomaster/centos-binary-mariadb:9f3a41c2c752bd68a84b6bd72add18e8bc6f4c76_f1a4ee5a",
        "2018-07-03 15:32:30,095 ERROR: 30320 -- Failed running docker-puppet.py for mysql_init_tasks",
        "2018-07-03 15:32:30,095 ERROR: 30320 -- Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend",
        "Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend",
        "Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 3.79 seconds",
        "Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{md5}af90358207ccfecae7af249d5ef7dd3e' to '{md5}ca60ca4934e101c81e74a6373973a80e'",
        "Changes:",
        " Total: 1",
        "Events:",
        " Success: 1",
        "Resources:",
        " Changed: 1",
        " Out of sync: 1",
        " Skipped: 6",
        " Total: 9",
        "Time:",
        " File: 0.03",
        " Last run: 1530631949",
        " Config retrieval: 4.12",
        " Total: 4.15",
        "Version:",
        " Config: 1530631945",
        " Puppet: 4.8.2",
        "",
        "2018-07-03 15:32:30,095 ERROR: 30320 -- + mkdir -p /etc/puppet",
        "+ cp -a /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/hieradata /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet",
        "+ rm -Rf /etc/puppet/ssl",
        "+ echo '{\"step\": 2}'",
        "+ TAGS=",
        "+ '[' -n file,file_line,concat,augeas,cron,mysql_database,mysql_grant,mysql_user ']'",
        "+ TAGS='--tags file,file_line,concat,augeas,cron,mysql_database,mysql_grant,mysql_user'",
        "+ origin_of_time=/var/lib/config-data/mysql_init_tasks.origin_of_time",
        "+ touch /var/lib/config-data/mysql_init_tasks.origin_of_time",
        "+ sync",
        "+ set +e",
        "+ FACTER_hostname=overcloud-controller-0",
        "+ FACTER_uuid=docker",
        "+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --logdest syslog --logdest console --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags file,file_line,concat,augeas,cron,mysql_database,mysql_grant,mysql_user /etc/config.pp",
        "Failed to get D-Bus connection: Operation not permitted",
        "Warning: Facter: Could not retrieve fact='nic_alias', resolution='<anonymous>': Could not execute '/usr/bin/os-net-config -i': command not found",
        "Warning: Undefined variable 'deploy_config_name'; ",
        " (file & line not available)",
        "Warning: This method is deprecated, please use the stdlib validate_legacy function,",
        " with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql.pp\", 103]:[\"/etc/config.pp\", 2]",
        " (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')",
        "Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
        "Warning: ModuleLoader: module 'aodh' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
        " with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/aodh/manifests/db/mysql.pp\", 57]:[\"/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql.pp\", 175]",
        "Warning: ModuleLoader: module 'cinder' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
        "Warning: ModuleLoader: module 'glance' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
        "Warning: ModuleLoader: module 'gnocchi' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
        "Warning: ModuleLoader: module 'heat' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
        "Warning: ModuleLoader: module 'keystone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
        "Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
        "Warning: ModuleLoader: module 'nova' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
        "Warning: ModuleLoader: module 'panko' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
        "Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules",
        " with Pattern[]. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/openstacklib/manifests/db/mysql/host_access.pp\", 43]:",
        "Error: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 \"No such file or directory\")",
        "+ rc=1",
        "+ set -e",
        "+ '[' 1 -ne 2 -a 1 -ne 0 ']'",
        "+ exit 1",
        "2018-07-03 15:32:30,095 INFO: 30320 -- Finished processing puppet configs for mysql_init_tasks",
        "2018-07-03 15:32:30,096 ERROR: 30319 -- ERROR configuring mysql_init_tasks"
    ]
}

[root@overcloud-controller-0 containers]# tail -n 10 mysql/mariadb.log
2018-07-03 16:00:26 139797265000640 [Note] InnoDB: Waiting for purge to start
2018-07-03 16:00:26 139797265000640 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 1622998
2018-07-03 16:00:26 139796454291200 [Note] InnoDB: Dumping buffer pool(s) not yet started
2018-07-03 16:00:26 139797265000640 [Note] Plugin 'FEEDBACK' is disabled.
2018-07-03 16:00:26 139797265000640 [Note] Server socket created on IP: '172.20.1.20'.
2018-07-03 16:00:26 139797265000640 [ERROR] Can't start server: listen() on TCP/IP port: Address already in use
2018-07-03 16:00:26 139797265000640 [ERROR] listen() on TCP/IP failed with error 98
2018-07-03 16:00:26 139797265000640 [ERROR] Aborting

180703 16:00:28 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended

Harald Jensås (harald-jensas) wrote :
Changed in tripleo:
milestone: rocky-3 → rocky-rc1
Changed in tripleo:
milestone: rocky-rc1 → stein-1
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3

Is this still an issue?

Changed in tripleo:
milestone: stein-3 → stein-rc1
Changed in tripleo:
milestone: stein-rc1 → train-1
Changed in tripleo:
milestone: train-1 → train-2
Changed in tripleo:
milestone: train-2 → train-3
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers