docker-puppet.py failed for haproxy no parameter manage_firewall

[Steven Brzozowski]

Undercloud was deployed with "./quickstart.sh --release master-tripleo-ci" on Aug 3, 2017 around 3:00 PM EST

Successfully deployed minimal containerized overcloud with 1 controller, 1 compute, and 3 ceph-storage nodes

Upon trying to deploy a hyperconverged containerized overcloud (1 controller and 3 osd-compute nodes), found that with some probability, the deployment would fail with the following output:
http://paste.openstack.org/show/617563/

The following command was used to deploy the latter overcloud:
http://paste.openstack.org/show/617564/

The failed playbook found in the controller node:
http://paste.openstack.org/show/617565/

[John Fulton (jfulton-org)]

I get this error too regardless of using HCI. Deleting the overcloud and redeploying works as a workaround but it happens often enough to affect usability.

(undercloud) [stack@undercloud ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: cce4e14b-7790-4261-b964-ccf8686a8d0a
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
        ],
        "changed": false,
        "failed": true,
        "failed_when_result": true
    }
     to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/ff8d76eb-7025-49af-a12e-85796df294f2_playbook.retry

    PLAY RECAP *********************************************************************
    localhost : ok=5 changed=1 unreachable=0 failed=1

[John Fulton (jfulton-org)]

- Reproduced and ran the steps of the local playbook [1] separately on the controller
- Second task `python /var/lib/docker-puppet/docker-puppet.py` returned 'ERROR: 60501 -- ERROR configuring haproxy'
- The downloaded image [2]
- docker inspect 87bf3ffe5d60 output at http://sprunge.us/NbMG

[1] http://paste.openstack.org/show/617818/
[2]
[root@overcloud-controller-0 heat-config-ansible]# docker images | grep haproxy
docker.io/tripleoupstream/centos-binary-haproxy latest 87bf3ffe5d60 8 days ago 664.2 MB
[root@overcloud-controller-0 heat-config-ansible]#

[John Fulton (jfulton-org)]

Following http://tripleo.org/install/containers_deployment/tips_tricks.html#debugging-docker-puppet-py ...
 [root@overcloud-controller-0 ~]# HOW_DIFF=True ; PROCESS_COUNT=1 ; CONFIG=/var/lib/docker-puppet/docker-puppet.json ; python /var/lib/docker-puppet/docker-puppet.py

2017-08-08 19:27:30,734 INFO: 74622 -- Pulling image: tripleoupstream/centos-binary-haproxy:latest
2017-08-08 19:27:37,112 ERROR: 74622 -- Failed running docker-puppet.py for haproxy
2017-08-08 19:27:37,113 ERROR: 74622 -- Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend

2017-08-08 19:27:37,113 ERROR: 74622 -- + mkdir -p /etc/puppet
+ cp -a /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/hieradata /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet
+ rm -Rf /etc/puppet/ssl
+ echo '{"step": 6}'
+ TAGS=
+ '[' -n file,file_line,concat,augeas,cron,haproxy_config ']'
+ TAGS='--tags file,file_line,concat,augeas,cron,haproxy_config'
+ touch /tmp/the_origin_of_time
+ sync
+ FACTER_hostname=overcloud-controller-0
+ FACTER_uuid=docker
+ /usr/bin/puppet apply --color=false --logdest syslog --logdest console --tags file,file_line,concat,augeas,cron,haproxy_config /etc/config.pp
Failed to get D-Bus connection: Operation not permitted
Warning: Undefined variable 'deploy_config_name';
   (file & line not available)
Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Tripleo::Profile::Base::Haproxy]: has no parameter named 'manage_firewall' at /etc/config.pp:2:1 on node overcloud-controller-0.openstacklocal

[John Fulton (jfulton-org)]

This issue might be happening because of an older version of puppet-tripleo. The commit below introduced the manage_firewall parameter.

 https://github.com/openstack/puppet-tripleo/commit/50f160a148b6a973891ffc6d0882f4c0d597336e