Comment 0 for bug 1843082

Brief Description
-----------------
      Unable to lock any host. System host-lock command was in stucked state . This was observed during the patch install test on IPv6 lab(wolfpass-03-07). Prior to this test lab was healthy able to lock and unlock all the hosts. Test scenario as below
Launched 50 pods using resource-consumer –image . Pods are active
upload test patch successful
apply test patch successful
Using horizon patch orchestration sw-patch strategy created successfully
orchestration sw-patch strategy applied successfully.
Orchestration locked standby controller-1 and swacted successfully then failed on locking new standby lock.
Manually tried to lock new standby failed and the prompt was not returned. This was continue to be there and lock was tried on other hosts it was same issue. After deleting resource consumer pods also same behavior.

sudo sw-patch show 2019-09-04_00-10-00_RR_ALLNODES
Password:
2019-09-04_00-10-00_RR_ALLNODES:
    Release: 19.09
    Patch State: Partial-Apply
    RR: Y
    Summary: Patch to /etc/init.d/logmgmt
    Contents:
                    RR_ALLNODES-1.0-2.tis.x86_64.rpm
                    logmgmt-1.0-6.tis.x86_64.rpm
                    logmgmt-wheels-1.0-6.tis.x86_64.rpm

sw-patch query-hosts
  Hostname IP Address Patch Current Reboot Required Release State
============ ========================= ============= =============== ======= =====
compute-0 face::fb1:f2dc:4b40:2ed4 No Yes 19.09 idle
compute-1 face::5455:e33e:5332:120c No Yes 19.09 idle
compute-2 face::75a5:57f:709b:1c3e No Yes 19.09 idle
controller-0 face::3 No Yes 19.09 idle
controller-1 face::4 Yes No 19.09 idle

$ sw-patch query
           Patch ID RR Release Patch State
=============================== == ======= =============
2019-09-04_00-10-00_RR_ALLNODES Y 19.09 Partial-Apply

Sysinv logs showing which is not sure
19-09-06 15:51:56.996 110960 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:54:31.036 110951 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:54:31.141 110951 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:54:31.181 110957 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:54:31.289 110957 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:54:31.292 110960 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:54:31.389 110957 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:54:31.397 110960 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:54:31.492 110960 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:55:13.765 110957 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:55:28.128 282169 INFO sysinv.cmd.dnsmasq_lease_update [-] Called 'old' for mac '3c:fd:fe:af:fc:ec' with ip 'face::fb1:f2dc:4b40:2ed4'
2019-09-06 15:56:03.015 286216 INFO sysinv.cmd.dnsmasq_lease_update [-] Called 'old' for mac '3c:fd:fe:af:fc:e8' with ip 'face::75a5:57f:709b:1c3e'
2019-09-06 15:56:14.259 287377 INFO sysinv.cmd.dnsmasq_lease_update [-] Called 'old' for mac '3c:fd:fe:af:fb:84' with ip 'face::5455:e33e:5332:120c'
2019-09-06 15:56:15.769 110953 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:56:15.886 110960 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:57:24.770 110960 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:59:31.052 110951 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 15:59:32.760 110951 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 16:00:33.771 110951 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 16:00:33.776 110957 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 16:00:33.812 110960 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 16:00:33.820 110960 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 16:01:40.769 110951 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 16:04:31.220 110957 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-06 16:04:31.325 110957 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []

Severity
--------
Major
Steps to Reproduce
------------------
1.Create 50 pods using resource consumer
kubectl run resource-consumer --image=gcr.io/kubernetes-e2e-test-images/resource-consumer:1.4 --expose --service-overrides='{ "spec": { "type": "LoadBalancer" } }' --port 8080 --requests='cpu=500m,memory=256Mi'
 kubectl get services resource-consumer
kubectl scale deploy/resource-consumer --replicas 50

2. Upload patches
3. Apply patch
3. Create patch strategy using horizon orchestion
4. Apply patch strategy and monitor for failure. Host lock standby and swact was successful new standby lock failed.
5. Tried host-lock manually failed as per description

System Configuration
--------------------
Regular system

Expected Behavior
------------------
Host-lock should be locking host and returning prompt. If there is a error should show and error and return prompt
Actual Behavior
----------------
As per description hosts fail to lock host.

Reproducibility
---------------
Unable to lock all the time . Lab was in that state.
Above test was tried only once
System Configuration
--------------------
Regular system IPV6
Load
----

Last Pass
---------
Not available
Timestamp/Logs
--------------
Manual lock
2019-09-06T16:01:52.000 controller-1 -sh: info HISTORY: PID=323823 UID=42425 date;system host-lock controller-0
Test Activity
-------------
Regression test