Failed running docker-puppet.py for designate

[Sorin Sbarnea]

Promotion is failing because:

2018-08-24 01:52:18 | "Status: Downloaded newer image for 192.168.24.1:8787/tripleorocky/centos-binary-designate-worker:tripleo-ci-testing-updated-20180824005543",
2018-08-24 01:52:18 | "2018-08-24 01:52:09,298 DEBUG: 18250 -- NET_HOST enabled",
2018-08-24 01:52:18 | "2018-08-24 01:52:09,298 DEBUG: 18250 -- Running docker command: /usr/bin/docker run --user root --name docker-puppet-designate --env PUPPET_TAGS=file,file_line,concat,augeas,cron,designate_config,designate_config,designate_config,designate_config,dns_config,exec,designate_config --env NAME=designate --env HOSTNAME=upstream-centos-7-rdo-cloud-tripleo-0000115742 --env NO_ARCHIVE= --env STEP=6 --volume /etc/localtime:/etc/localtime:ro --volume /tmp/tmppf71KK:/etc/config.pp:ro,z --volume /etc/puppet/:/tmp/puppet-etc/:ro,z --volume /usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro,z --volume /var/lib/config-data:/var/lib/config-data/:z --volume tripleo_logs:/var/log/tripleo/ --volume /dev/log:/dev/log --volume /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume /var/lib/docker-puppet/docker-puppet.sh:/var/lib/docker-puppet/docker-puppet.sh:z --entrypoint /var/lib/docker-puppet/docker-puppet.sh --net host --volume /etc/hosts:/etc/hosts:ro 192.168.24.1:8787/tripleorocky/centos-binary-designate-worker:tripleo-ci-testing-updated-20180824005543",
2018-08-24 01:52:18 | "2018-08-24 01:52:15,627 ERROR: 18250 -- Failed running docker-puppet.py for designate",
2018-08-24 01:52:18 | "2018-08-24 01:52:15,628 ERROR: 18250 -- Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend",

FAILURE: http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-rocky/d3bc8e2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
SUCCESS: http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-rocky/b4e71cb/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[wes hayutin (weshayutin)]

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-rocky/d3bc8e2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-08-24_01_52_18

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-rocky/d3bc8e2/logs/subnode-2/var/log/journal.txt.gz#_Aug_24_01_52_14

file,file_line,concat,augeas,cron,designate_config,designate_config,designate_config,designate_config,dns_config,exec,designate_config /etc/config.pp
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 puppet-user[26521]: (Facter) Could not retrieve fact='nic_alias', resolution='<anonymous>': Could not execute '/usr/bin/os-net-config -i': command not found
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 puppet-user[26521]: (Facter) Could not retrieve fact='nic_alias', resolution='<anonymous>': Could not execute '/usr/bin/os-net-config -i': command not found
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 dockerd-current[8044]: Warning: Facter: Could not retrieve fact='nic_alias', resolution='<anonymous>': Could not execute '/usr/bin/os-net-config -i': command not found
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 dockerd-current[8044]: Warning: Facter: Could not retrieve fact='nic_alias', resolution='<anonymous>': Could not execute '/usr/bin/os-net-config -i': command not found
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 puppet-user[26521]: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 dockerd-current[8044]: Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 puppet-user[26521]: Undefined variable 'deploy_config_name';
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 dockerd-current[8044]: Warning: Undefined variable 'deploy_config_name';
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 puppet-user[26521]: (file & line not available)
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 dockerd-current[8044]: (file & line not available)

This evaluation error looks like the real error:

 Evaluation Error: Error while evaluating a Resource Statement, Class[Dns]: parameter 'controls' expects a Hash[String, Hash[String, Data]] value, got Hash[Undef, Struct[{'port' => Integer[953, 953], 'keys' => Tuple[String[8, 8]], NotUndef['allowed_addresses'] => Undef}]] at /etc/puppet/modules/designate/manifests/backend/bind9.pp:38:5 on node upstream-centos-7-rdo-cloud-tripleo-0000115742.localdomain
Aug 24 01:52:15 upstream-centos-7-rdo-cloud-tripleo-0000115742 dockerd-current[8044]: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Dns]: parameter 'controls' expects a Hash[String, Hash[String, Data]] value, got Hash[Undef, Struct[{'port' => Integer[953, 953], 'keys' => Tuple[String[8, 8]], NotU...

[wes hayutin (weshayutin)]

Class[Dns]: parameter 'controls' expects a Hash[String, Hash[String, Data]] value, got Hash[Undef, Struct[{'port' => Integer[953, 953], 'keys' => Tuple[String[8, 8]], NotUndef['allowed_addresses'] => Undef}]] at /etc/puppet/modules/designate/manifests/backend/bind9.pp:38:5

[wes hayutin (weshayutin)]

https://github.com/openstack/puppet-designate/commit/df4991fbe2c9f7e5b457ae8ed0f6e401fc4e9c1d

[wes hayutin (weshayutin)]

Also failing in check, adding alert

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/596453

[wes hayutin (weshayutin)]

Fix is here ---> https://review.openstack.org/#/c/596450/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/596453
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=b918c0b5b7faa5d170982265e7d990cd6f7ba0ac
Submitter: Zuul
Branch: master

commit b918c0b5b7faa5d170982265e7d990cd6f7ba0ac
Author: Wes Hayutin <email address hidden>
Date: Fri Aug 24 15:53:16 2018 -0400

    add scenario003 tests when designate changes

    Related-Bug: #1788907
    Change-Id: I8fb4047596068ca02c812dc9545d74436992c5c3

[wes hayutin (weshayutin)]

Looks like https://review.openstack.org/#/c/583670/ will fix it

[yatin (yatinkarel)]

<< Looks like https://review.openstack.org/#/c/583670/ will fix it

So it's not just 583670, it includes lot's of commits under it. and yes with all those commit fs018 promotion job is passing, tested here:- https://logs.rdoproject.org/43/13943/24/check/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-master/999bb47/job-output.txt.gz#_2018-08-26_20_39_54_361781, and the reverted comment should not be needed to fix the issue, but i think bnemec is trying to fix that circular dependency with that commit.

And since now tripleo-heat-templates is branched(stable/rocky), all these tht patches would required to be cherry-picked there to unblock rocky promotions(as today we will be branching(rocky-rdo) it in RDO) and i think it would take time to get all those cherry-picks. So, if required can pin puppet-tripleo to a good hash in RDO and unblock rocky promotion.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/596697

[Sorin Sbarnea (ssbarnea)]

Any update on this? The list of patches is growing and I am afriad that the chance of getting them merged succesfully goes down.

[Ben Nemec (bnemec)]

Since fixing the problem was pretty much as simple as squashing reverts, I've proposed https://review.openstack.org/#/c/596467 which fixes the bug (scenario003 has already passed on it, although it's waiting on some other jobs ATM). That will also need to be backported to Rocky, but once that is done this should be resolved.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/596697
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=cda96f8bc79ff4fbf8a26351eaac7045b5197681
Submitter: Zuul
Branch: stable/rocky

commit cda96f8bc79ff4fbf8a26351eaac7045b5197681
Author: Wes Hayutin <email address hidden>
Date: Fri Aug 24 15:53:16 2018 -0400

    add scenario003 tests when designate changes

    Related-Bug: #1788907
    Change-Id: I8fb4047596068ca02c812dc9545d74436992c5c3
    (cherry picked from commit b918c0b5b7faa5d170982265e7d990cd6f7ba0ac)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/596467
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=0ed2cf7aea9f9e6be9e2ddf0fe7b5c6311e35994
Submitter: Zuul
Branch: master

commit 0ed2cf7aea9f9e6be9e2ddf0fe7b5c6311e35994
Author: Ben Nemec <email address hidden>
Date: Wed Jul 11 20:57:51 2018 +0000

    Conditionalize rndc changes

    The changes to allow configuration of more rndc options had a
    cross-repo dependency on the tripleo-heat-templates changes that
    made use of the new options. Unfortunately, the puppet changes
    landed first and are now broken because necessary inputs are not
    being provided. This change conditionalizes the new configurations
    so that if the parameters aren't passed we won't attempt to do
    invalid configuration.

    Closes-Bug: #1788907
    Change-Id: Ia82f1c6be0d5504498e77a90268cad8abecdeae2

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/596864
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=0134f91b947774cdea5e7616536ea5bdd3ceade0
Submitter: Zuul
Branch: stable/rocky

commit 0134f91b947774cdea5e7616536ea5bdd3ceade0
Author: Ben Nemec <email address hidden>
Date: Wed Jul 11 20:57:51 2018 +0000

    Conditionalize rndc changes

    The changes to allow configuration of more rndc options had a
    cross-repo dependency on the tripleo-heat-templates changes that
    made use of the new options. Unfortunately, the puppet changes
    landed first and are now broken because necessary inputs are not
    being provided. This change conditionalizes the new configurations
    so that if the parameters aren't passed we won't attempt to do
    invalid configuration.

    Closes-Bug: #1788907
    Change-Id: Ia82f1c6be0d5504498e77a90268cad8abecdeae2
    (cherry picked from commit 3add939d4c723dec9cc1c2538ea28271d5769041)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ben Nemec (<email address hidden>) on branch: master
Review: https://review.openstack.org/596450
Reason: Fixed instead of reverted.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 10.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 9.4.0 release.