| 2020-06-11 20:20:09 |
Anujeyan Manokeran |
bug |
|
|
added bug |
| 2020-06-11 20:20:55 |
Anujeyan Manokeran |
summary |
patch orchestration failed on unlocking controllers |
Patch orchestration failed on unlocking controllers |
|
| 2020-06-11 20:30:43 |
Anujeyan Manokeran |
tags |
|
stx.retestneeded |
|
| 2020-06-11 20:34:17 |
Anujeyan Manokeran |
attachment added |
|
collect logs https://bugs.launchpad.net/starlingx/+bug/1883176/+attachment/5383101/+files/ALL_NODES_20200611.182201.tar |
|
| 2020-06-12 18:17:23 |
Ghada Khalil |
tags |
stx.retestneeded |
stx.retestneeded stx.update |
|
| 2020-06-12 18:18:09 |
Ghada Khalil |
description |
Brief Description
-----------------
During test patch apply using patch orchestration controller unlock was failed . When controller was unlocked manually it was successful . Further investigation by Bart following findings.
2020-06-11 17:06:43 – The VIM issues a host-install request to patching for controller-0. This seems to complete immediately.
The VIM then waits 15 seconds (intentionally).
2020-06-11 17:06:59 – The VIM issues a host-lock request for controller-0 to sysinv:
Sysinv queries patching to see if the host is patch current.
The patcher seems to say the host is not patch current:
sysinv 2020-06-11 17:07:00.289 254793 WARNING wsme.api [-] Client-side error: host-unlock rejected: Not patch current. 'sw-patch host-install controller-0' is required.: ClientSideError: host-unlock rejected: Not patch current. 'sw-patch host-install controller-0' is required.
Patch orchestration details
apply-phase:
total-stages: 2
current-stage: 0
stop-at-stage: 2
timeout: 10173 seconds
completion-percentage: 100%
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:07:00
result: failed
reason: host unlock failed
stages:
stage-id: 0
stage-name: sw-patch-controllers
total-steps: 7
current-step: 5
timeout: 5536 seconds
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:07:00
result: failed
reason: host unlock failed
steps:
step-id: 0
step-name: query-alarms
timeout: 60 seconds
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:00:16
result: success
reason:
step-id: 1
step-name: swact-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 900 seconds
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:03:56
result: success
reason:
step-id: 2
step-name: lock-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 900 seconds
start-date-time: 2020-06-11 17:03:56
end-date-time: 2020-06-11 17:06:43
result: success
reason:
step-id: 3
step-name: sw-patch-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 1800 seconds
start-date-time: 2020-06-11 17:06:43
step-name: sw-patch-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 1800 seconds
start-date-time: 2020-06-11 17:06:43
end-date-time: 2020-06-11 17:06:43
result: success
reason:
step-id: 4
step-name: system-stabilize
timeout: 15 seconds
start-date-time: 2020-06-11 17:06:43
end-date-time: 2020-06-11 17:06:59
result: success
reason:
step-id: 5
step-name: unlock-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 1800 seconds
start-date-time: 2020-06-11 17:06:59
end-date-time: 2020-06-11 17:07:00
result: failed
reason: host unlock failed
step-id: 6
step-name: system-stabilize
timeout: 60 seconds
result: initial
reason:
stage-id: 1
stage-name: sw-patch-worker-hosts
total-steps: 6
current-step: 0
timeout: 4636 seconds
start-date-time:
end-date-time:
result: initial
reason:
steps:
step-id: 0
step-name: query-alarms
timeout: 60 seconds
result: initial
reason:
step-id: 1
step-name: lock-hosts
entity-type: hosts
Severity
--------
Major
System Configuration
--------------------
wcp-71-75
Expected Behavior
------------------
No failure on unlock
Actual Behavior
----------------
As description says unlock by patch orchestration fails
Reproducibility
---------------
Tried only once with this load.
Load
-------
2020-06-10_20-00-00
Last Pass
---------
It was passed on 2020-06-10_20-00-00 with different
Reboot able patch . Above failure was large test patch.
Timestamp/Logs
--------------
2020-06-11 17:06:43
Test Activity
-------------
Regression test |
Brief Description
-----------------
During test patch apply using patch orchestration controller unlock was failed . When controller was unlocked manually it was successful . Further investigation by Bart following findings.
2020-06-11 17:06:43 – The VIM issues a host-install request to patching for controller-0. This seems to complete immediately.
The VIM then waits 15 seconds (intentionally).
2020-06-11 17:06:59 – The VIM issues a host-lock request for controller-0 to sysinv:
Sysinv queries patching to see if the host is patch current.
The patcher seems to say the host is not patch current:
sysinv 2020-06-11 17:07:00.289 254793 WARNING wsme.api [-] Client-side error: host-unlock rejected: Not patch current. 'sw-patch host-install controller-0' is required.: ClientSideError: host-unlock rejected: Not patch current. 'sw-patch host-install controller-0' is required.
Patch orchestration details
apply-phase:
total-stages: 2
current-stage: 0
stop-at-stage: 2
timeout: 10173 seconds
completion-percentage: 100%
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:07:00
result: failed
reason: host unlock failed
stages:
stage-id: 0
stage-name: sw-patch-controllers
total-steps: 7
current-step: 5
timeout: 5536 seconds
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:07:00
result: failed
reason: host unlock failed
steps:
step-id: 0
step-name: query-alarms
timeout: 60 seconds
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:00:16
result: success
reason:
step-id: 1
step-name: swact-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 900 seconds
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:03:56
result: success
reason:
step-id: 2
step-name: lock-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 900 seconds
start-date-time: 2020-06-11 17:03:56
end-date-time: 2020-06-11 17:06:43
result: success
reason:
step-id: 3
step-name: sw-patch-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 1800 seconds
start-date-time: 2020-06-11 17:06:43
step-name: sw-patch-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 1800 seconds
start-date-time: 2020-06-11 17:06:43
end-date-time: 2020-06-11 17:06:43
result: success
reason:
step-id: 4
step-name: system-stabilize
timeout: 15 seconds
start-date-time: 2020-06-11 17:06:43
end-date-time: 2020-06-11 17:06:59
result: success
reason:
step-id: 5
step-name: unlock-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 1800 seconds
start-date-time: 2020-06-11 17:06:59
end-date-time: 2020-06-11 17:07:00
result: failed
reason: host unlock failed
step-id: 6
step-name: system-stabilize
timeout: 60 seconds
result: initial
reason:
stage-id: 1
stage-name: sw-patch-worker-hosts
total-steps: 6
current-step: 0
timeout: 4636 seconds
start-date-time:
end-date-time:
result: initial
reason:
steps:
step-id: 0
step-name: query-alarms
timeout: 60 seconds
result: initial
reason:
step-id: 1
step-name: lock-hosts
entity-type: hosts
Severity
--------
Major
System Configuration
--------------------
wcp-71-75
Expected Behavior
------------------
No failure on unlock
Actual Behavior
----------------
As description says unlock by patch orchestration fails
Reproducibility
---------------
Tried only once with this load.
Load
-------
2020-06-10_20-00-00
Last Pass
---------
It was passed on 2020-06-10_20-00-00 with different
Reboot able patch . Above failure was large test patch.
Timestamp/Logs
--------------
2020-06-11 17:06:43
Test Activity
-------------
Regression test |
|
| 2020-06-12 18:21:56 |
Ghada Khalil |
starlingx: status |
New |
Triaged |
|
| 2020-06-12 18:21:58 |
Ghada Khalil |
starlingx: importance |
Undecided |
Low |
|
| 2020-06-12 18:22:12 |
Ghada Khalil |
starlingx: assignee |
|
Don Penney (dpenney) |
|
| 2020-06-12 18:22:27 |
Ghada Khalil |
bug |
|
|
added subscriber Bart Wensley |
| 2020-06-12 19:51:45 |
Bart Wensley |
bug |
|
|
added subscriber Eric MacDonald |
| 2020-06-12 19:58:16 |
Bart Wensley |
starlingx: assignee |
Don Penney (dpenney) |
Eric MacDonald (rocksolidmtce) |
|
| 2020-06-12 19:58:31 |
Bart Wensley |
tags |
stx.retestneeded stx.update |
stx.nfv stx.retestneeded |
|
| 2020-06-12 20:58:45 |
Ghada Khalil |
bug |
|
|
added subscriber Daniel Badea |
| 2020-06-12 20:58:48 |
Ghada Khalil |
starlingx: importance |
Low |
High |
|
| 2020-06-12 21:02:57 |
Ghada Khalil |
description |
Brief Description
-----------------
During test patch apply using patch orchestration controller unlock was failed . When controller was unlocked manually it was successful . Further investigation by Bart following findings.
2020-06-11 17:06:43 – The VIM issues a host-install request to patching for controller-0. This seems to complete immediately.
The VIM then waits 15 seconds (intentionally).
2020-06-11 17:06:59 – The VIM issues a host-lock request for controller-0 to sysinv:
Sysinv queries patching to see if the host is patch current.
The patcher seems to say the host is not patch current:
sysinv 2020-06-11 17:07:00.289 254793 WARNING wsme.api [-] Client-side error: host-unlock rejected: Not patch current. 'sw-patch host-install controller-0' is required.: ClientSideError: host-unlock rejected: Not patch current. 'sw-patch host-install controller-0' is required.
Patch orchestration details
apply-phase:
total-stages: 2
current-stage: 0
stop-at-stage: 2
timeout: 10173 seconds
completion-percentage: 100%
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:07:00
result: failed
reason: host unlock failed
stages:
stage-id: 0
stage-name: sw-patch-controllers
total-steps: 7
current-step: 5
timeout: 5536 seconds
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:07:00
result: failed
reason: host unlock failed
steps:
step-id: 0
step-name: query-alarms
timeout: 60 seconds
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:00:16
result: success
reason:
step-id: 1
step-name: swact-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 900 seconds
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:03:56
result: success
reason:
step-id: 2
step-name: lock-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 900 seconds
start-date-time: 2020-06-11 17:03:56
end-date-time: 2020-06-11 17:06:43
result: success
reason:
step-id: 3
step-name: sw-patch-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 1800 seconds
start-date-time: 2020-06-11 17:06:43
step-name: sw-patch-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 1800 seconds
start-date-time: 2020-06-11 17:06:43
end-date-time: 2020-06-11 17:06:43
result: success
reason:
step-id: 4
step-name: system-stabilize
timeout: 15 seconds
start-date-time: 2020-06-11 17:06:43
end-date-time: 2020-06-11 17:06:59
result: success
reason:
step-id: 5
step-name: unlock-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 1800 seconds
start-date-time: 2020-06-11 17:06:59
end-date-time: 2020-06-11 17:07:00
result: failed
reason: host unlock failed
step-id: 6
step-name: system-stabilize
timeout: 60 seconds
result: initial
reason:
stage-id: 1
stage-name: sw-patch-worker-hosts
total-steps: 6
current-step: 0
timeout: 4636 seconds
start-date-time:
end-date-time:
result: initial
reason:
steps:
step-id: 0
step-name: query-alarms
timeout: 60 seconds
result: initial
reason:
step-id: 1
step-name: lock-hosts
entity-type: hosts
Severity
--------
Major
System Configuration
--------------------
wcp-71-75
Expected Behavior
------------------
No failure on unlock
Actual Behavior
----------------
As description says unlock by patch orchestration fails
Reproducibility
---------------
Tried only once with this load.
Load
-------
2020-06-10_20-00-00
Last Pass
---------
It was passed on 2020-06-10_20-00-00 with different
Reboot able patch . Above failure was large test patch.
Timestamp/Logs
--------------
2020-06-11 17:06:43
Test Activity
-------------
Regression test |
Brief Description
-----------------
During test patch apply using patch orchestration controller unlock was failed . When controller was unlocked manually it was successful . Further investigation by Bart following findings.
2020-06-11 17:06:43 – The VIM issues a host-install request to patching for controller-0. This seems to complete immediately.
The VIM then waits 15 seconds (intentionally).
2020-06-11 17:06:59 – The VIM issues a host-lock request for controller-0 to sysinv:
Sysinv queries patching to see if the host is patch current.
The patcher seems to say the host is not patch current:
sysinv 2020-06-11 17:07:00.289 254793 WARNING wsme.api [-] Client-side error: host-unlock rejected: Not patch current. 'sw-patch host-install controller-0' is required.: ClientSideError: host-unlock rejected: Not patch current. 'sw-patch host-install controller-0' is required.
Patch orchestration details
apply-phase:
total-stages: 2
current-stage: 0
stop-at-stage: 2
timeout: 10173 seconds
completion-percentage: 100%
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:07:00
result: failed
reason: host unlock failed
stages:
stage-id: 0
stage-name: sw-patch-controllers
total-steps: 7
current-step: 5
timeout: 5536 seconds
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:07:00
result: failed
reason: host unlock failed
steps:
step-id: 0
step-name: query-alarms
timeout: 60 seconds
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:00:16
result: success
reason:
step-id: 1
step-name: swact-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 900 seconds
start-date-time: 2020-06-11 17:00:16
end-date-time: 2020-06-11 17:03:56
result: success
reason:
step-id: 2
step-name: lock-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 900 seconds
start-date-time: 2020-06-11 17:03:56
end-date-time: 2020-06-11 17:06:43
result: success
reason:
step-id: 3
step-name: sw-patch-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 1800 seconds
start-date-time: 2020-06-11 17:06:43
step-name: sw-patch-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 1800 seconds
start-date-time: 2020-06-11 17:06:43
end-date-time: 2020-06-11 17:06:43
result: success
reason:
step-id: 4
step-name: system-stabilize
timeout: 15 seconds
start-date-time: 2020-06-11 17:06:43
end-date-time: 2020-06-11 17:06:59
result: success
reason:
step-id: 5
step-name: unlock-hosts
entity-type: hosts
entity-names: [u'controller-0']
timeout: 1800 seconds
start-date-time: 2020-06-11 17:06:59
end-date-time: 2020-06-11 17:07:00
result: failed
reason: host unlock failed
step-id: 6
step-name: system-stabilize
timeout: 60 seconds
result: initial
reason:
stage-id: 1
stage-name: sw-patch-worker-hosts
total-steps: 6
current-step: 0
timeout: 4636 seconds
start-date-time:
end-date-time:
result: initial
reason:
steps:
step-id: 0
step-name: query-alarms
timeout: 60 seconds
result: initial
reason:
step-id: 1
step-name: lock-hosts
entity-type: hosts
Severity
--------
Major
System Configuration
--------------------
wcp-71-75
Expected Behavior
------------------
No failure on unlock
Actual Behavior
----------------
As description says unlock by patch orchestration fails
Reproducibility
---------------
Tried only once with this load.
Load
-------
2020-06-10_20-00-00
Last Pass
---------
It was passed on 2020-06-10_20-00-00 with different
Reboot able patch . Above failure was large test patch.
Timestamp/Logs
--------------
2020-06-11 17:06:43
Test Activity
-------------
Regression test |
|
| 2020-06-12 21:03:28 |
Ghada Khalil |
tags |
stx.nfv stx.retestneeded |
stx.4.0 stx.nfv stx.retestneeded |
|
| 2020-06-16 15:17:41 |
Ghada Khalil |
starlingx: assignee |
Eric MacDonald (rocksolidmtce) |
Bart Wensley (bartwensley) |
|
| 2020-06-16 15:19:36 |
Ghada Khalil |
starlingx: assignee |
Bart Wensley (bartwensley) |
Eric MacDonald (rocksolidmtce) |
|
| 2020-06-16 15:19:53 |
Ghada Khalil |
starlingx: status |
Triaged |
In Progress |
|
| 2020-06-16 16:19:49 |
OpenStack Infra |
starlingx: status |
In Progress |
Fix Released |
|
| 2020-06-27 16:53:47 |
Yosief Gebremariam |
tags |
stx.4.0 stx.nfv stx.retestneeded |
|
|
| 2020-06-28 01:32:30 |
Ghada Khalil |
tags |
|
stx.4.0 stx.nfv |
|
