Failed running docker-puppet.py for designate

Bug #1788907 reported by Sorin Sbarnea on 2018-08-24
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Ben Nemec

Bug Description

Promotion is failing because:

2018-08-24 01:52:18 | "Status: Downloaded newer image for 192.168.24.1:8787/tripleorocky/centos-binary-designate-worker:tripleo-ci-testing-updated-20180824005543",
2018-08-24 01:52:18 | "2018-08-24 01:52:09,298 DEBUG: 18250 -- NET_HOST enabled",
2018-08-24 01:52:18 | "2018-08-24 01:52:09,298 DEBUG: 18250 -- Running docker command: /usr/bin/docker run --user root --name docker-puppet-designate --env PUPPET_TAGS=file,file_line,concat,augeas,cron,designate_config,designate_config,designate_config,designate_config,dns_config,exec,designate_config --env NAME=designate --env HOSTNAME=upstream-centos-7-rdo-cloud-tripleo-0000115742 --env NO_ARCHIVE= --env STEP=6 --volume /etc/localtime:/etc/localtime:ro --volume /tmp/tmppf71KK:/etc/config.pp:ro,z --volume /etc/puppet/:/tmp/puppet-etc/:ro,z --volume /usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro,z --volume /var/lib/config-data:/var/lib/config-data/:z --volume tripleo_logs:/var/log/tripleo/ --volume /dev/log:/dev/log --volume /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume /var/lib/docker-puppet/docker-puppet.sh:/var/lib/docker-puppet/docker-puppet.sh:z --entrypoint /var/lib/docker-puppet/docker-puppet.sh --net host --volume /etc/hosts:/etc/hosts:ro 192.168.24.1:8787/tripleorocky/centos-binary-designate-worker:tripleo-ci-testing-updated-20180824005543",
2018-08-24 01:52:18 | "2018-08-24 01:52:15,627 ERROR: 18250 -- Failed running docker-puppet.py for designate",
2018-08-24 01:52:18 | "2018-08-24 01:52:15,628 ERROR: 18250 -- Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend",

FAILURE: http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-rocky/d3bc8e2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
SUCCESS: http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-rocky/b4e71cb/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

wes hayutin (weshayutin) on 2018-08-24
tags: added: promotion-blocker
removed: alert
wes hayutin (weshayutin) wrote :
Download full text (3.4 KiB)

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-rocky/d3bc8e2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-08-24_01_52_18

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-rocky/d3bc8e2/logs/subnode-2/var/log/journal.txt.gz#_Aug_24_01_52_14

file,file_line,concat,augeas,cron,designate_config,designate_config,designate_config,designate_config,dns_config,exec,designate_config /etc/config.pp
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 puppet-user[26521]: (Facter) Could not retrieve fact='nic_alias', resolution='<anonymous>': Could not execute '/usr/bin/os-net-config -i': command not found
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 puppet-user[26521]: (Facter) Could not retrieve fact='nic_alias', resolution='<anonymous>': Could not execute '/usr/bin/os-net-config -i': command not found
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 dockerd-current[8044]: Warning: Facter: Could not retrieve fact='nic_alias', resolution='<anonymous>': Could not execute '/usr/bin/os-net-config -i': command not found
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 dockerd-current[8044]: Warning: Facter: Could not retrieve fact='nic_alias', resolution='<anonymous>': Could not execute '/usr/bin/os-net-config -i': command not found
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 puppet-user[26521]: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 dockerd-current[8044]: Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 puppet-user[26521]: Undefined variable 'deploy_config_name';
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 dockerd-current[8044]: Warning: Undefined variable 'deploy_config_name';
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 puppet-user[26521]: (file & line not available)
Aug 24 01:52:14 upstream-centos-7-rdo-cloud-tripleo-0000115742 dockerd-current[8044]: (file & line not available)

This evaluation error looks like the real error:

 Evaluation Error: Error while evaluating a Resource Statement, Class[Dns]: parameter 'controls' expects a Hash[String, Hash[String, Data]] value, got Hash[Undef, Struct[{'port' => Integer[953, 953], 'keys' => Tuple[String[8, 8]], NotUndef['allowed_addresses'] => Undef}]] at /etc/puppet/modules/designate/manifests/backend/bind9.pp:38:5 on node upstream-centos-7-rdo-cloud-tripleo-0000115742.localdomain
Aug 24 01:52:15 upstream-centos-7-rdo-cloud-tripleo-0000115742 dockerd-current[8044]: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Dns]: parameter 'controls' expects a Hash[String, Hash[String, Data]] value, got Hash[Undef, Struct[{'port' => Integer[953, 953], 'keys' => Tuple[String[8, 8]], NotU...

Read more...

wes hayutin (weshayutin) wrote :

Class[Dns]: parameter 'controls' expects a Hash[String, Hash[String, Data]] value, got Hash[Undef, Struct[{'port' => Integer[953, 953], 'keys' => Tuple[String[8, 8]], NotUndef['allowed_addresses'] => Undef}]] at /etc/puppet/modules/designate/manifests/backend/bind9.pp:38:5

wes hayutin (weshayutin) wrote :

Also failing in check, adding alert

tags: added: alert

Related fix proposed to branch: master
Review: https://review.openstack.org/596453

Changed in tripleo:
assignee: nobody → wes hayutin (weshayutin)
status: New → In Progress
Changed in tripleo:
milestone: none → rocky-rc2
importance: Undecided → Critical

Reviewed: https://review.openstack.org/596453
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=b918c0b5b7faa5d170982265e7d990cd6f7ba0ac
Submitter: Zuul
Branch: master

commit b918c0b5b7faa5d170982265e7d990cd6f7ba0ac
Author: Wes Hayutin <email address hidden>
Date: Fri Aug 24 15:53:16 2018 -0400

    add scenario003 tests when designate changes

    Related-Bug: #1788907
    Change-Id: I8fb4047596068ca02c812dc9545d74436992c5c3

wes hayutin (weshayutin) on 2018-08-26
Changed in tripleo:
assignee: wes hayutin (weshayutin) → nobody
Changed in tripleo:
assignee: nobody → yatin (yatinkarel)
wes hayutin (weshayutin) wrote :
yatin (yatinkarel) wrote :

<< Looks like https://review.openstack.org/#/c/583670/ will fix it

So it's not just 583670, it includes lot's of commits under it. and yes with all those commit fs018 promotion job is passing, tested here:- https://logs.rdoproject.org/43/13943/24/check/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-master/999bb47/job-output.txt.gz#_2018-08-26_20_39_54_361781, and the reverted comment should not be needed to fix the issue, but i think bnemec is trying to fix that circular dependency with that commit.

And since now tripleo-heat-templates is branched(stable/rocky), all these tht patches would required to be cherry-picked there to unblock rocky promotions(as today we will be branching(rocky-rdo) it in RDO) and i think it would take time to get all those cherry-picks. So, if required can pin puppet-tripleo to a good hash in RDO and unblock rocky promotion.

Changed in tripleo:
assignee: yatin (yatinkarel) → Ben Nemec (bnemec)
Sorin Sbarnea (ssbarnea) wrote :

Any update on this? The list of patches is growing and I am afriad that the chance of getting them merged succesfully goes down.

Ben Nemec (bnemec) wrote :

Since fixing the problem was pretty much as simple as squashing reverts, I've proposed https://review.openstack.org/#/c/596467 which fixes the bug (scenario003 has already passed on it, although it's waiting on some other jobs ATM). That will also need to be backported to Rocky, but once that is done this should be resolved.

Reviewed: https://review.openstack.org/596697
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=cda96f8bc79ff4fbf8a26351eaac7045b5197681
Submitter: Zuul
Branch: stable/rocky

commit cda96f8bc79ff4fbf8a26351eaac7045b5197681
Author: Wes Hayutin <email address hidden>
Date: Fri Aug 24 15:53:16 2018 -0400

    add scenario003 tests when designate changes

    Related-Bug: #1788907
    Change-Id: I8fb4047596068ca02c812dc9545d74436992c5c3
    (cherry picked from commit b918c0b5b7faa5d170982265e7d990cd6f7ba0ac)

tags: added: in-stable-rocky
Changed in tripleo:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/596467
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=0ed2cf7aea9f9e6be9e2ddf0fe7b5c6311e35994
Submitter: Zuul
Branch: master

commit 0ed2cf7aea9f9e6be9e2ddf0fe7b5c6311e35994
Author: Ben Nemec <email address hidden>
Date: Wed Jul 11 20:57:51 2018 +0000

    Conditionalize rndc changes

    The changes to allow configuration of more rndc options had a
    cross-repo dependency on the tripleo-heat-templates changes that
    made use of the new options. Unfortunately, the puppet changes
    landed first and are now broken because necessary inputs are not
    being provided. This change conditionalizes the new configurations
    so that if the parameters aren't passed we won't attempt to do
    invalid configuration.

    Closes-Bug: #1788907
    Change-Id: Ia82f1c6be0d5504498e77a90268cad8abecdeae2

Reviewed: https://review.openstack.org/596864
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=0134f91b947774cdea5e7616536ea5bdd3ceade0
Submitter: Zuul
Branch: stable/rocky

commit 0134f91b947774cdea5e7616536ea5bdd3ceade0
Author: Ben Nemec <email address hidden>
Date: Wed Jul 11 20:57:51 2018 +0000

    Conditionalize rndc changes

    The changes to allow configuration of more rndc options had a
    cross-repo dependency on the tripleo-heat-templates changes that
    made use of the new options. Unfortunately, the puppet changes
    landed first and are now broken because necessary inputs are not
    being provided. This change conditionalizes the new configurations
    so that if the parameters aren't passed we won't attempt to do
    invalid configuration.

    Closes-Bug: #1788907
    Change-Id: Ia82f1c6be0d5504498e77a90268cad8abecdeae2
    (cherry picked from commit 3add939d4c723dec9cc1c2538ea28271d5769041)

Change abandoned by Ben Nemec (<email address hidden>) on branch: master
Review: https://review.openstack.org/596450
Reason: Fixed instead of reverted.

This issue was fixed in the openstack/puppet-tripleo 10.0.0 release.

This issue was fixed in the openstack/puppet-tripleo 9.4.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers