Scaling up with heat stack-update

[Ladislav Smola]

After successful stack creation with COMPUTESCALE=1

scaling up and updating doesn't work on latest devtest

make -C $TRIPLEO_ROOT/tripleo-heat-templates overcloud.yaml COMPUTESCALE=2
heat syack-update -f $TRIPLEO_ROOT/tripleo-heat-templates/overcloud.yaml \
    -P "AdminToken=${OVERCLOUD_ADMIN_TOKEN}" \
    -P "AdminPassword=${OVERCLOUD_ADMIN_PASSWORD}" \
    -P "CinderPassword=${OVERCLOUD_CINDER_PASSWORD}" \
    -P "GlancePassword=${OVERCLOUD_GLANCE_PASSWORD}" \
    -P "HeatPassword=${OVERCLOUD_HEAT_PASSWORD}" \
    -P "NeutronFlatNetworks=${OVERCLOUD_FLAT_NETWORKS}" \
    -P "NeutronPassword=${OVERCLOUD_NEUTRON_PASSWORD}" \
    -P "NovaPassword=${OVERCLOUD_NOVA_PASSWORD}" \
    -P "NeutronPublicInterface=${NeutronPublicInterface}" \
    -P "SwiftPassword=${OVERCLOUD_SWIFT_PASSWORD}" \
    -P "SwiftHashSuffix=${OVERCLOUD_SWIFT_HASH}" \
    -P "NovaComputeLibvirtType=${OVERCLOUD_LIBVIRT_TYPE}" \
    -P "SSLCertificate=${OVERCLOUD_SSL_CERT}" \
    -P "SSLKey=${OVERCLOUD_SSL_KEY}" \
    overcloud

Logs

http://paste.openstack.org/show/64344/

the heat log shows nothing interesting

[James Slagle (james-slagle)]

the "syack-update" is just a typo in the description

[Ladislav Smola (lsmola)]

confirmed by rlandy, it doesnt work in undercloud-live nor in the devtest

[Robert Collins (lifeless)]

This is important to implement, but noone has done so yet. THanks for figuring out that it doesn't work :)

[Tomas Sedovic (tsedovic)]

Looks like a Heat bug. Here's a stack trace from the logs:

Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.624 7144 INFO heat.engine.resource [-] updating AccessPolicy "ComputeAccessPolicy" Stack "overcloud"
Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.663 7144 DEBUG heat.engine.resource [-] Resource ComputeAccessPolicy update requires replacement upda
Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.664 7144 DEBUG heat.engine.update [-] Backing up existing Resource ComputeAccessPolicy _create_resour
Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.694 7144 INFO heat.engine.resource [-] creating AccessPolicy "ComputeAccessPolicy" Stack "overcloud"
Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.709 7144 ERROR heat.engine.resources.user [-] AccessPolicy resource NovaCompute1 not in stack
Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.710 7144 ERROR heat.engine.resource [-] CREATE : AccessPolicy "ComputeAccessPolicy" Stack "overcloud"
Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.710 7144 TRACE heat.engine.resource Traceback (most recent call last):
Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.710 7144 TRACE heat.engine.resource File "/opt/stack/venvs/heat/lib/python2.7/site-packages/heat/en
Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.710 7144 TRACE heat.engine.resource handle())
Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.710 7144 TRACE heat.engine.resource File "/opt/stack/venvs/heat/lib/python2.7/site-packages/heat/en
Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.710 7144 TRACE heat.engine.resource stack_name=self.stack.name)
Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.710 7144 TRACE heat.engine.resource ResourceNotFound: The Resource (NovaCompute1) could not be found
Feb 28 11:57:24 undercloud-undercloud-ymr7vupmq3pt heat-engine[7144]: 2014-02-28 11:57:24.710 7144 TRACE heat.engine.resource

Here's what's raising the exception:

https://github.com/openstack/heat/blob/65fbc4f3632cf2d04f57dbe6b5b9e6f5e16583f1/heat/engine/resources/user.py#L325

If you look at the submitted event-list, AccessPolicy is created before any of the AllowedResouces it references exist. The same goes for update.

However, the `handle_create` method checks that the stack being updated at least knows about the listed resources (NovaCompute0 and NovaCompute0Config during create, the same plus NovaCompute1 and NovaCompute1Config during update).

So it looks like the new AccessPolicy is being create in the context of the old stack/template without NovaCompute1. Which is why it fails.

I'll poke into it a bit more and see what comes up. We'll probably have to file bug with Heat.

[Tomas Sedovic (tsedovic)]

sorry, log-pasting fail. Here's the uncut version:

http://paste.openstack.org/show/70606/

[Tomas Sedovic (tsedovic)]

I think I found the issue, will submit it to Heat later today.

[Tomas Sedovic (tsedovic)]

Here's the related Heat bug: https://bugs.launchpad.net/heat/+bug/1286307

[Ladislav Smola (lsmola)]

Great work, the bug is fixed!

I have updated from 1 compute to 2 and it has correctly created new node. (active node in nova-list)

Though, I am getting timeout on NovaCompute1CompletionCondition (the new instance) will investigate this and create another bug if needed.