ParallelDeploymentsTest intermittent failure

[Thomas Herve]

ParallelDeploymentsTest.test_deployments_metadata rarely fails with a timeout error "failed to reach CREATE_COMPLETE status within the required time".

I've found the following traceback which looks related:
Exception during message handling: deployment_id must be specified
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
    executor_callback))
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
    executor_callback)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
    result = func(ctxt, **new_args)
  File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
    return f(*args, **kwargs)
  File "/opt/stack/new/heat/heat/common/context.py", line 325, in wrapped
    return func(self, ctx, *args, **kwargs)
  File "/opt/stack/new/heat/heat/engine/service.py", line 1810, in signal_software_deployment
    updated_at=updated_at)
  File "/opt/stack/new/heat/heat/engine/service_software_config.py", line 228, in signal_software_deployment
    raise ValueError(_('deployment_id must be specified'))
ValueError: deployment_id must be specified

[Thomas Herve (therve)]

I think I've found the issue: we're signaling the resource as soon as the metadata is available, but the resource is not finished creating, thus signal fails because resource_id is not set.

I can fix the test, but I wonder if it's not a race condition that can happen in the wild.

[Thomas Herve (therve)]

Looks very similar to what tried to fix https://review.openstack.org/218627

[Steve Baker (steve-stevebaker)]

I think fixing the test would be appropriate, this seems very unlikely to happen in the wild.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/265714

[Thomas Herve (therve)]

I agree that the window is small, but considering things like zaqar queues were the subscriber can be notified as soon as the message is sent, I think it's there, so I tried something to remove it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/265714
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=b3a56659ee6e8b54b6bacc864635116983ec562c
Submitter: Jenkins
Branch: master

commit b3a56659ee6e8b54b6bacc864635116983ec562c
Author: Thomas Herve <email address hidden>
Date: Mon Jan 11 09:22:33 2016 +0100

    Provide software deployment id on create

    There is currently a race condition where software deployment populates
    its metadata when being created, before the parent resource receives an
    ID. This creates a window where the metadata is available but the
    resource can't be signalled. This fixes it by providing the creation ID.

    Change-Id: If58679138db95cf1da7ab5a4906217da446a391f
    Closes-Bug: #1532445

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/heat 6.0.0.0b2 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/heat 6.0.0.0b2 development milestone.