get_attr seems to be occasionally returning before the output is ready

[Giulio Fidente]

In TripleO we seem to be hitting an issue where get_attr is returning before the output of the referenced resource is ready; this can be seen in the submission at: https://review.openstack.org/#/c/305751

That submission is creating a new output in the endpoint_map.yaml, see [1] and then it's updating the glance-registry role so that it gets its mysql connection string from the new output [2].

The mysql connection string is collected from the endpoint_map output in and passed as parameter to the role in [3]

As a result, the glance connection string ends up being invalid and the puppet module will default to sqlite [4]

1. https://review.openstack.org/#/c/305751/8/network/endpoints/endpoint_map.yaml
2. https://review.openstack.org/#/c/305751/8/puppet/services/glance-registry.yaml
3. https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.yaml#L878
4. http://logs.openstack.org/51/305751/8/check-tripleo/gate-tripleo-ci-f22-nonha/ab26e4d/logs/overcloud-controller-0.tar.xz

[Giulio Fidente (gfidente)]

In the controller logs we have the puppet hieradata provisioned by heat. When the problem occures, the puppet param related to the connection string looks like the following:

glance::registry::database_connection: ://glance:G9dBqYZQwgH6rJ4NqgTbaECeK@/glance

so it misses a couple of values supposed to be gathered from the EndpointMap resource in [1]

1. https://review.openstack.org/#/c/305751/8/puppet/services/glance-registry.yaml

[Zane Bitter (zaneb)]

One possibility to look into would be a race such that the stack reports as COMPLETE over RPC but when the parent stack then reads the outputs (which, IIRC, does not go over RPC) they're not yet available.

[Steven Hardy (shardy)]

Possibly related to the comment from Zane - in https://github.com/openstack/heat/commit/095a5eae434417ed8ba7bb3d73aacf493ac51387 we made a change that means the parent Stack object is only partially refreshed while we're polling for state.

While we don't store the outputs in the stack record, the previous approach did a force reload of the entire stack object every time we polled for state, which I think has the side-effect of re-resolving the outputs every time. So it'd be interesting to put in an explicit self.resolve_static_data(self.t[self.t.OUTPUTS]) somewhere like we'd previously have been doing via the Stack constructor?

[Jay Dobies (jdob)]

Steve - Thanks for the input. One (probably more, but one for now) question:

I see in the patch where we removed the force_reload in the nested stack retrieval, replacing it with the lightweight DB call for just the status (and some other small data).

What I'm not understanding is how that would affect the parent stack. I see a reference to nested.set_parent_stack removed from _parse_nested_stack, but the latter is only called during validate, which AFAICT doesn't come into play during that status check load.

I'm waiting on a run of TripleO CI with some added logging to finish, but after that, I'll see about kicking one off with the explicit resolve call at the tail end of the nested stack creation to see if it's any more stable.

[Zane Bitter (zaneb)]

No activity here in a while. Is it safe to say this is no longer an issue?