Containers scenario001 broken -- resources.ControllerDockerConfig.properties.value: Cannot merge <type 'unicode'> with <type 'tuple'>

[Attila Darazs]

gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container fails with the following error:

[overcloud.AllNodesDeploySteps]: CREATE_FAILED resources.AllNodesDeploySteps: Property error: resources.ControllerDockerConfig.properties.value: Cannot merge <type 'unicode'> with <type 'tuple'>

Searching for ControllerDockerConfig in THT doesn't turn up anything, so I'm trying to reproduce this locally and check the generated templates.

Related change that fixes a couple of problems already: https://review.openstack.org/476693

[Jiří Stránský (jistr)]

The ControllerDockerConfig mentioned above gets generated from jinja2 this way:

https://github.com/openstack/tripleo-heat-templates/blob/05ec01df4325a50de5114ef28ae0cde3fd4b843a/docker/docker-steps.j2#L171-L182

Given the error message, this could be caused by somehow malformed service template, something in the lines of the manila-api issue for scenario004:

https://review.openstack.org/#/c/476916

[Jiří Stránský (jistr)]

It'll be a needle in haystack search but i think we should be looking at the service_name and docker_config sections of the services enabled here:

https://review.openstack.org/#/c/476693/1/ci/environments/scenario001-multinode-containers.yaml

[Jiří Stránský (jistr)]

We could probably get more info by removing the `type()` wrapper calls here

https://github.com/openstack/yaql/blob/c9d1f7bf756d922f13d93f80e64a40d5f9d21d34/yaql/standard_library/queries.py#L1395-L1407

So that we get the values printed, rather than the data types.

[Attila Darazs (adarazs)]

I managed to reproduce it locally. For the record it needed this change: https://review.openstack.org/477892 and adding this line to /usr/share/openstack-tripleo-heat-templates/environments/docker-network.yaml

OS::TripleO::Controller::Net::SoftwareConfig: ../net-config-bridge.yaml

After editing /usr/lib/python2.7/site-packages/yaql/standard_library/queries.py as suggested by Jiří, it resulted in

    resources.AllNodesDeploySteps: Property error: resources.ControllerDockerConfig.properties.value: Cannot merge /usr/bin/bootstrap_host_exec ceilometer su ceilometer -s /bin/bash -c '/usr/bin/ceilometer-upgrade --skip-metering-database' with (u'/usr/bin/ceilometer-upgrade', u'--skip-metering-database')

So the problem is with that tuple. It appears as a list in two files, and as a string in one, so I assume the string version is the correct one:

list:

- https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/ceilometer-agent-central.yaml#L109
- https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/ceilometer-agent-notification.yaml#L109

string:

https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/ceilometer-agent-ipmi.yaml#L109

[Jiří Stránský (jistr)]

So the issue is that we have a duplicate container defined, and the way we approach templates processing nowadays doesn't deal well with that. Possible solutions i see:

1) Only do the ceilometer upgrade from one of the agent containers instead of all 3, if there's an agent that we know we *always* have to deploy when deploying ceilometer. (Could that be the central agent?)

2) Extract the ceilometer upgrade to a separate service template (not a great solution...).

3) Rename the 3 ceilometer upgrade containers so that each has a unique name. This would mean that we run the ceilometer upgrade multiple times potentially.

4) Accept duplicate containers, and only spawn one eventually, and write validations so that we require that the container occurences are always the same (we'd have to change the ceilometer upgrade containers to use a single common image), and they are defined in the same step everywhere.

We need to sync with the telemetry folks on this, to pick the right solution.

[Jiří Stránský (jistr)]

I think it would be ideal to go with 1) if possible. Even in this case it would be good to add validations to make sure that the root cause of this bug cannot happen again.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/478554

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/478554
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=244b8bf13e04dafb11cfde02dddfff68a6e272fd
Submitter: Jenkins
Branch: master

commit 244b8bf13e04dafb11cfde02dddfff68a6e272fd
Author: Pradeep Kilambi <email address hidden>
Date: Wed Jun 28 12:22:37 2017 -0400

    Remove ceilometer upgrade from other ceilometer containers

    Lets just run the ceilometer upgrade once in central agent container

    Change-Id: If5e5ca6122f8583c6221bc6b343e483e41f04d29
    Closes-bug: #1700056

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b3 development milestone.