Duplicate signals create events with reason "Unknown"

Bug #1564627 reported by Steve Baker
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
New
Medium
Unassigned
tripleo
Fix Released
Medium
Steven Hardy

Bug Description

All SIGNAL events currently look like this:

[1]: SIGNAL_COMPLETE Unknown

There should be a useful reason which can be extracted from the signal, but it is being lost.

This has been fixed before in #1300679 and #1300679 but has still been occurring for a couple of releases.

Changed in heat:
milestone: none → newton-1
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
Rabi Mishra (rabi) wrote :

I think shardy submitted a patch[1] for this sometime back.

[1] https://review.openstack.org/#/c/291660/

Changed in heat:
assignee: nobody → Sharat Sharma (sharat-sharma)
status: Triaged → In Progress
Revision history for this message
Steven Hardy (shardy) wrote :

Actually the bug is that we log events for signals, even when they are ignored:

https://github.com/openstack/heat/blob/master/heat/engine/service_software_config.py#L264

We return none from signal_software_deployment if we get a spurious signal to a deployment that's already COMPLETE, which we then try to derive a reason from here:

https://github.com/openstack/heat/blob/master/heat/engine/resource.py#L1916

Probably the heat fix is to not log events for signals we're actually ignoring, but I couldn't see a simple way to do that, so instead I fixed the script that's sending duplicate signals for os-apply-config deployments:

https://review.openstack.org/#/c/314226/

Steven Hardy (shardy)
Changed in tripleo:
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Steven Hardy (shardy)
milestone: none → newton-1
summary: - All SIGNAL events have a reason "Unknown"
+ Duplicate signals create events with reason "Unknown"
Revision history for this message
Steven Hardy (shardy) wrote :

2016-05-11 17:49:12 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded

This is what the actual signal we don't ignore looks like - we could probably improve this by including the deployment id in the "deployment succeeded" case, and in particular that would be useful when the signal indicates failure

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/315420

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/315476

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to heat (master)

Reviewed: https://review.openstack.org/315476
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=530544b3df7513a132c0b81e5c1b6a802959cb1f
Submitter: Jenkins
Branch: master

commit 530544b3df7513a132c0b81e5c1b6a802959cb1f
Author: Steven Hardy <email address hidden>
Date: Wed May 11 18:56:41 2016 +0100

    Add deployment ID to deploy signal events

    Curently you get events that look like this with the default
    event formatter:

    2016-05-11 17:49:12 [0]: SIGNAL_IN_PROGRESS Signal: deployment succeeded

    Including the ID in the reason string will help (particularly in the
    case of failure), as you can then easily show the deployment and
    thus the stderr associated with the failure.

    Change-Id: I59a3e9bf527040c4d807adc4f822bfa2c1591585
    Related-Bug: #1564627

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-image-elements (master)

Reviewed: https://review.openstack.org/314226
Committed: https://git.openstack.org/cgit/openstack/tripleo-image-elements/commit/?id=de7012df7cef4c46cd1434233ffa090482e05f27
Submitter: Jenkins
Branch: master

commit de7012df7cef4c46cd1434233ffa090482e05f27
Author: Steven Hardy <email address hidden>
Date: Mon May 9 19:32:34 2016 +0100

    Don't send duplicate signals via 99-refresh-completed

    Currently we send signals for every deployment, even those already
    completed, so instead keep a record of already-signalled deployments
    similar to how heat-config does, which will remove the annoying
    SIGNAL_COMPLETE: Unknown events (and possibly be slightly faster too).

    Partial-Bug: #1564627
    Change-Id: I0ec7ed4cc4e58a43a91e8323cb9a1bcaaafc9752

Steven Hardy (shardy)
Changed in tripleo:
status: In Progress → Fix Committed
Rabi Mishra (rabi)
Changed in heat:
milestone: newton-1 → newton-2
Steven Hardy (shardy)
Changed in tripleo:
status: Fix Committed → Fix Released
Thomas Herve (therve)
Changed in heat:
milestone: newton-2 → newton-3
Thomas Herve (therve)
Changed in heat:
milestone: newton-3 → next
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (master)

Change abandoned by Rico Lin (<email address hidden>) on branch: master
Review: https://review.openstack.org/315420
Reason: Hi, this patch stays untouched for few months, would like to abandon for now. Feel free to restore it if you still working on this. Thank you!

Changed in heat:
assignee: Sharat Sharma (sharat-sharma) → nobody
status: In Progress → New
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.