get_attr seems to be occasionally returning before the output is ready

Bug #1575898 reported by Giulio Fidente
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Incomplete
High
Jay Dobies

Bug Description

In TripleO we seem to be hitting an issue where get_attr is returning before the output of the referenced resource is ready; this can be seen in the submission at: https://review.openstack.org/#/c/305751

That submission is creating a new output in the endpoint_map.yaml, see [1] and then it's updating the glance-registry role so that it gets its mysql connection string from the new output [2].

The mysql connection string is collected from the endpoint_map output in and passed as parameter to the role in [3]

As a result, the glance connection string ends up being invalid and the puppet module will default to sqlite [4]

1. https://review.openstack.org/#/c/305751/8/network/endpoints/endpoint_map.yaml
2. https://review.openstack.org/#/c/305751/8/puppet/services/glance-registry.yaml
3. https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud.yaml#L878
4. http://logs.openstack.org/51/305751/8/check-tripleo/gate-tripleo-ci-f22-nonha/ab26e4d/logs/overcloud-controller-0.tar.xz

Revision history for this message
Giulio Fidente (gfidente) wrote :

In the controller logs we have the puppet hieradata provisioned by heat. When the problem occures, the puppet param related to the connection string looks like the following:

glance::registry::database_connection: ://glance:G9dBqYZQwgH6rJ4NqgTbaECeK@/glance

so it misses a couple of values supposed to be gathered from the EndpointMap resource in [1]

1. https://review.openstack.org/#/c/305751/8/puppet/services/glance-registry.yaml

Jay Dobies (jdob)
Changed in heat:
assignee: nobody → Jay Dobies (jdob)
Revision history for this message
Zane Bitter (zaneb) wrote :

One possibility to look into would be a race such that the stack reports as COMPLETE over RPC but when the parent stack then reads the outputs (which, IIRC, does not go over RPC) they're not yet available.

Thomas Herve (therve)
description: updated
Changed in heat:
importance: Undecided → High
Revision history for this message
Steven Hardy (shardy) wrote :

Possibly related to the comment from Zane - in https://github.com/openstack/heat/commit/095a5eae434417ed8ba7bb3d73aacf493ac51387 we made a change that means the parent Stack object is only partially refreshed while we're polling for state.

While we don't store the outputs in the stack record, the previous approach did a force reload of the entire stack object every time we polled for state, which I think has the side-effect of re-resolving the outputs every time. So it'd be interesting to put in an explicit self.resolve_static_data(self.t[self.t.OUTPUTS]) somewhere like we'd previously have been doing via the Stack constructor?

Revision history for this message
Jay Dobies (jdob) wrote :

Steve - Thanks for the input. One (probably more, but one for now) question:

I see in the patch where we removed the force_reload in the nested stack retrieval, replacing it with the lightweight DB call for just the status (and some other small data).

What I'm not understanding is how that would affect the parent stack. I see a reference to nested.set_parent_stack removed from _parse_nested_stack, but the latter is only called during validate, which AFAICT doesn't come into play during that status check load.

I'm waiting on a run of TripleO CI with some added logging to finish, but after that, I'll see about kicking one off with the explicit resolve call at the tail end of the nested stack creation to see if it's any more stable.

Revision history for this message
Zane Bitter (zaneb) wrote :

No activity here in a while. Is it safe to say this is no longer an issue?

Zane Bitter (zaneb)
Changed in heat:
status: New → Incomplete
Rico Lin (rico-lin)
Changed in heat:
milestone: none → no-priority-tag-bugs
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.