scn003 gate: [tripleo-mysql-client-conf]: Could not evaluate: Saving failed

Bug #1748180 reported by Rafael Folco
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Alex Schultz

Bug Description

tripleo-ci-centos-7-scenario003-multinode-oooq-container

http://logs.openstack.org/76/541376/2/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/f646d7a/logs/undercloud/home/zuul/failed_deployment_list.log.txt.gz

2018-02-08 09:06:03 | overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.0:
2018-02-08 09:06:03 | resource_type: OS::Heat::StructuredDeployment
2018-02-08 09:06:03 | physical_resource_id: 1b06f96a-5ca8-41fd-8868-cb6f26b80408
2018-02-08 09:06:03 | status: CREATE_FAILED
2018-02-08 09:06:03 | status_reason: |
2018-02-08 09:06:03 | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2018-02-08 09:06:03 | deploy_stdout: |
2018-02-08 09:06:03 | ...
2018-02-08 09:06:03 | " with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/ntp/manifests/init.pp\", 76]:[\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\", 29]",
2018-02-08 09:06:03 | " with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/ssh/manifests/server.pp\", 12]:[\"/var/lib/tripleo-config/puppet_step_config.pp\", 45]",
2018-02-08 09:06:03 | "Error: /Stage[main]/Tripleo::Profile::Base::Database::Mysql::Client/Augeas[tripleo-mysql-client-conf]: Could not evaluate: Saving failed, see debug"
2018-02-08 09:06:03 | ]
2018-02-08 09:06:03 | }

Another run:
http://logs.openstack.org/29/541829/1/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/7533271/logs/undercloud/home/zuul/failed_deployment_list.log.txt.gz

Revision history for this message
Michele Baldessari (michele) wrote :

So the complete augeas error is:
Debug: Augeas[tripleo-mysql-client-conf](provider=augeas): Augeas version 1.4.0 is installed",
        "Debug: Augeas[tripleo-mysql-client-conf](provider=augeas): Will attempt to save and only run if files changed",
        "Debug: Augeas[tripleo-mysql-client-conf](provider=augeas): sending command 'set' with params [\"/files/etc/my.cnf.d/tripleo.cnf/tripleo/bind-address\", \"\"]",
        "Debug: Augeas[tripleo-mysql-client-conf](provider=augeas): sending command 'rm' with params [\"/files/etc/my.cnf.d/tripleo.cnf/tripleo/ssl\"]",
        "Debug: Augeas[tripleo-mysql-client-conf](provider=augeas): sending command 'rm' with params [\"/files/etc/my.cnf.d/tripleo.cnf/tripleo/ssl-ca\"]",
        "Debug: Augeas[tripleo-mysql-client-conf](provider=augeas): Put failed on one or more files, output from /augeas//error:",
        "Debug: Augeas[tripleo-mysql-client-conf](provider=augeas): /augeas/files/etc/my.cnf.d/tripleo.cnf/error = put_failed",
        "Debug: Augeas[tripleo-mysql-client-conf](provider=augeas): /augeas/files/etc/my.cnf.d/tripleo.cnf/error/path = /files/etc/my.cnf.d/tripleo.cnf/tripleo/bind-address",
        "Debug: Augeas[tripleo-mysql-client-conf](provider=augeas): /augeas/files/etc/my.cnf.d/tripleo.cnf/error/lens = /usr/share/augeas/lenses/dist/inifile.aug:216.40-.94:",
        "Debug: Augeas[tripleo-mysql-client-conf](provider=augeas): /augeas/files/etc/my.cnf.d/tripleo.cnf/error/message = Value '' does not match regexp /([^\\t\\n\\r \"#;])((((([^\\n\\r#;])+)*)([^\\t\\n\\r \"#;])){0,1})/ in store lens",
        "Debug: Augeas[tripleo-mysql-client-conf](provider=augeas): Closed the augeas connection"

I wonder if the problem here is that we are passing mysql_client_bind_address = '' instead of leaving it undef (in the tripleo::profile::base::database::mysql::client class)

Revision history for this message
Michele Baldessari (michele) wrote :

Indeed in http://logs.openstack.org/76/541376/2/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/f646d7a/logs/subnode-2/etc/puppet/hieradata/service_configs.json.txt.gz we can see that the bind address is set to empty:
     "tripleo::profile::base::database::mysql::client_bind_address": "",

Now that hiera key is populated by:
puppet/services/database/mysql-client.yaml: tripleo::profile::base::database::mysql::client::mysql_client_bind_address: {get_param: [ServiceNetMap, MysqlNetwork]}

So I guess there are two points here:
A) The root issue: "Why is {get_param: [ServiceNetMap, MysqlNetwork]}" returning an empty string?
B) How do we fail more clearly in the puppet module so the error is more obvious (if we should at all)

Revision history for this message
Alex Schultz (alex-schultz) wrote :

A) is the problem but for B) we can just add a check to tripleo::profile::base::database::mysql::client to ensure mysql_client_bind_address is always an ip.

Revision history for this message
Alex Schultz (alex-schultz) wrote :

Also it might be because of the check in https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/database/mysql/client.pp#L56 which in puppet 4 '' the if condition returns true. I'll update it to check for an ip address.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.openstack.org/542372

Changed in tripleo:
assignee: nobody → Alex Schultz (alex-schultz)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/542479

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (master)

Reviewed: https://review.openstack.org/542372
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=ff88c8929ce884d613a77dcf5a00a51da7301ed6
Submitter: Zuul
Branch: master

commit ff88c8929ce884d613a77dcf5a00a51da7301ed6
Author: Alex Schultz <email address hidden>
Date: Thu Feb 8 11:53:03 2018 -0700

    Ensure mysql client ip address is an ip

    If the hieradata is set to '' for the client ip address, puppet may
    consider this as a true condition. This change changes out the check to
    ensure that the value is an ip address, otherwise remove it from the
    configuration file.

    Change-Id: Idbf6bec3ec66f7018dc30e5f1e6d1ba5f48180d7
    Partial-Bug: #1748180

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/542479

Revision history for this message
Alex Schultz (alex-schultz) wrote :

This appears to have been resolved. The original failure had empty addresses in the net_ip_map.json hieradata. http://logs.openstack.org/76/541376/2/gate/tripleo-ci-centos-7-scenario003-multinode-oooq-container/f646d7a/logs/subnode-2/etc/puppet/hieradata/net_ip_map.json.txt.gz They are now properly populated and we've improved the failure when an ip isn't passed in.

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.