host-delete was rejected while host was powered off and hostname is none

Bug #1850683 reported by Anujeyan Manokeran
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
John Kung

Bug Description

Brief Description
-----------------
    host-delete was rejected for hostname name with none and powered off. Newly added host after powering off the host when it was deleted it was rejected . It was tried many times. Below shows the rejected message .
system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 3 | compute-1 | worker | unlocked | disabled | offline |
| 4 | controller-1 | controller | unlocked | enabled | available |
| 5 | None | None | locked | disabled | offline |
+----+--------------+-------------+----------------+-------------+--------------+
[sysadmin@controller-0 log(keystone_admin)]$ system host-delete 5
Mtce rejected delete request.Please retry and if problem persists then contact your system administrator.

2019-10-30 17:37:13.918 113029 INFO sysinv.api.controllers.v1.host [-] sysinv notify vim add host None personality=None
2019-10-30 17:37:13.918 113029 INFO sysinv.api.controllers.v1.host [-] update_subfunctions: subfunctions not set. personality=
2019-10-30 17:37:13.918 113029 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_add hostname=None, subfunctions= locked-disabled-offline subfunction_oper=disabled subfunction_avail=not-installed
2019-10-30 17:37:13.918 113029 WARNING sysinv.api.controllers.v1.vim_api [-] vim_host_add api_cmd=http://localhost:30001/nfvi-plugins/v1/hosts/ headers={'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload={'subfunction_oper': u'disabled', 'subfunction_avail': u'not-installed', 'uuid': u'b5cde097-b887-4c47-bcdf-6699da6b624d', 'subfunctions': '', 'hostname': None, 'operational': u'disabled', 'administrative': u'locked', 'availability': u'offline'}
2019-10-30 17:37:13.919 113029 INFO sysinv.api.controllers.v1.rest_api [-] POST cmd:http://localhost:30001/nfvi-plugins/v1/hosts/ hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"subfunction_oper": "disabled", "subfunction_avail": "not-installed", "uuid": "b5cde097-b887-4c47-bcdf-6699da6b624d", "subfunctions": "", "hostname": null, "operational": "disabled", "administrative": "locked", "availability": "offline"}
2019-10-30 17:37:13.920 113029 INFO sysinv.api.controllers.v1.rest_api [-] Response={}
2019-10-30 17:37:13.921 113029 WARNING wsme.api [-] Client-side error: Mtce rejected delete request.Please retry and if problem persists then contact your system administrator.
2019-10-30 17:37:40.979 113028 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-10-30 17:39:41.175 1076747 INFO sysinv.cmd.dnsmasq_lease_update [-] Called 'old' for mac 'a4:bf:01:00:0a:1f' with ip 'face::1d4f:a3b5:9024:8d8a'

6.274 [112384.00540] controller-0 mtcAgent hbs nodeClass.cpp (2968) clear_service_readies : Info : compute-1 clearing service ready events
2019-10-30T17:36:56.279 [112384.00541] controller-0 mtcAgent inv mtcInvApi.cpp ( 334) mtcInvApi_update_task : Info : compute-1 Task: Booting (seq:121)
2019-10-30T17:36:56.284 [112384.00542] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1035) enable_handler : Info : compute-1 Booting (timeout: 720 secs) (2)
2019-10-30T17:36:56.307 fmAlarmUtils.cpp(524): Sending FM raise alarm request: alarm_id (200.022), entity_id (host=compute-1.status=offline)
2019-10-30T17:36:56.348 fmAlarmUtils.cpp(558): FM Response for raise alarm: (0), alarm_id (200.022), entity_id (host=compute-1.status=offline)
2019-10-30T17:37:13.917 [112384.00543] controller-0 mtcAgent svr mtcHttpSvr.cpp ( 866) mtcHttpSvr_handler : Warn : uuid to host lookup failed ; 'b5cde097-b887-4c47-bcdf-6699da6b624d' not found
2019-10-30T17:37:13.917 [112384.00544] controller-0 mtcAgent svr mtcHttpSvr.cpp (1006) mtcHttpSvr_handler : Warn : sending 404 response for DELETE:/v1/hosts/b5cde097-b887-4c47-bcdf-6699da6b624d request ; response:{ "status" : "fail","reason" : "not found","action" : "retry with valid host"}

Severity
--------
Major

Steps to Reproduce
------------------
1. Verify system is up and good health. No alarm
2. lock one of the compute host-lock
3. Delete the locked compute system host-lock compute-0
4. Since compute is not powered off it will re added after it was rebooted host-list will have hostname
with none
5. Power off the compute that was deleted
6. Delete the host that was added last . As per above description id=5. host delete was failed.

System Configuration
--------------------
Regular system wcp-63-66 IPV6

Expected Behavior
------------------
Able delete hosts when hosts name is none .

Actual Behavior
----------------
As per description hosts delete failed for hostname is none

Reproducibility
---------------
100% reproduce able.

Load
----
Build was on "2019-10-29 20:02:14 -0400"
Last Pass
---------
Timestamp/Logs
--------------
2019-10-30 17:37:13.919

Test Activity
-------------
Regression test

summary: - host-delete was rejected when host was powered off
+ host-delete was rejected while it is host was powered off and hostname
+ is none
Revision history for this message
Anujeyan Manokeran (anujeyan) wrote :
summary: - host-delete was rejected while it is host was powered off and hostname
- is none
+ host-delete was rejected while host was powered off and hostname is none
Yang Liu (yliu12)
tags: added: stx.retestneeded
Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Low
status: New → Triaged
importance: Low → Undecided
status: Triaged → New
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Marking as stx.3.0 / medium priority - it's expected that the host-delete would succeed regardless of the actual state of the node

Changed in starlingx:
importance: Undecided → Medium
status: New → Triaged
tags: added: stx.3.0 stx.config
Changed in starlingx:
assignee: nobody → John Kung (john-kung)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/696629

Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (master)

Reviewed: https://review.opendev.org/696629
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=64e097037a1c0cd1aedcb3d1220f4f48ffb379f4
Submitter: Zuul
Branch: master

commit 64e097037a1c0cd1aedcb3d1220f4f48ffb379f4
Author: John Kung <email address hidden>
Date: Thu Nov 28 16:15:53 2019 -0500

    Allow host-delete for unprovisioned hosts without mtce notify

    In the case of host-delete in which hostname or personality has not yet
    been provisioned, host-delete must not send the notification to mtce.
    This is because mtce is not notified by sysinv about hosts which have
    not yet been provisioned with a hostname or personality.

    Change-Id: Iec310641411520ab2e1b711632c6c64e251ac64b
    Closes-Bug: 1850683
    Signed-off-by: John Kung <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
Ghada Khalil (gkhalil) wrote :

The next step is to cherrypick to the r/stx.3.0 branch

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (r/stx.3.0)

Fix proposed to branch: r/stx.3.0
Review: https://review.opendev.org/697096

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (r/stx.3.0)

Reviewed: https://review.opendev.org/697096
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=df3165209a05f8d9c99beb4048b10bd623bd5fc4
Submitter: Zuul
Branch: r/stx.3.0

commit df3165209a05f8d9c99beb4048b10bd623bd5fc4
Author: John Kung <email address hidden>
Date: Thu Nov 28 16:15:53 2019 -0500

    Allow host-delete for unprovisioned hosts without mtce notify

    In the case of host-delete in which hostname or personality has not yet
    been provisioned, host-delete must not send the notification to mtce.
    This is because mtce is not notified by sysinv about hosts which have
    not yet been provisioned with a hostname or personality.

    Change-Id: Iec310641411520ab2e1b711632c6c64e251ac64b
    Closes-Bug: 1850683
    Signed-off-by: John Kung <email address hidden>
    (cherry picked from commit 64e097037a1c0cd1aedcb3d1220f4f48ffb379f4)

Ghada Khalil (gkhalil)
tags: added: in-r-stx30
Revision history for this message
Anujeyan Manokeran (anujeyan) wrote :

verified in load 2020-02-19 02:30:00 +0000

tags: removed: in-r-stx30 stx.retestneeded
Ghada Khalil (gkhalil)
tags: added: stx.4.0
tags: added: in-r-stx30
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.