Steps (roughly) prior to the problem
[Unlocked controller-1 then swact was initiated here] 2019-11-04 15:25:10,211 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/ 302] parameters:[{"action": "hostscontroller__swact__1", "csrfmiddlewaretoken": "a7HwE603lUKsk7PEWLbWXQVPf7rwYttoWJPWmDWqLrm1kEsOowBQcze9tVPperiZ"}] message:[success: Swact Initiated Host: controller-0]
[Lock controller-1 then elastic labels added (via horizon) the controller-1 was unlocked - elastic-data, elastic-controller,elastic client, elastic-master]
2019-11-04 15:46:43,736 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/ 302] parameters:[{"action": "hostscontroller__lock__2", "csrfmiddlewaretoken": "r3wLFmZ4VcOyMt4wTm5sSQDgRGbAJTUzUR1fc0Vg13qFx4b8obqectAGIsOnFp94"}] message:[success: Locking Host: controller-1]
2019-11-04 15:47:45,148 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/2/assignlabel/ 200] parameters:[{"clabelvalue": "enabled", "clabelkey": "elastic-data", "host_id": "2", "csrfmiddlewaretoken": "UQbQ6qKK1jvcAkPYavBAQoJHheGRFZstnEGkD4GW7a7jlVWAFkWma1G780jEBvHY", "host_uuid": "1c95aa14-657e-494d-a617-76e460981339", "labelkey": "customized_label"}] message:[success: Label "elastic-data" was successfully created.]
2019-11-04 15:47:55,759 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/2/assignlabel/ 200] parameters:[{"clabelvalue": "enabled", "clabelkey": "elastic-controller", "host_id": "2", "csrfmiddlewaretoken": "R8lSVzWJzbExf28mf6nLP6tg1fOsBJyHkWQmsdSVF2gE0DfYKVIx9JqGS1rfxfNc", "host_uuid": "1c95aa14-657e-494d-a617-76e460981339", "labelkey": "customized_label"}] message:[success: Label "elastic-controller" was successfully created.]
2019-11-04 15:48:05,502 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/2/assignlabel/ 200] parameters:[{"clabelvalue": "enabled", "clabelkey": "elastic-client", "host_id": "2", "csrfmiddlewaretoken": "2yMXXqt9UlscnQdltdvz5DHU9eJRjuVYvmhru4pl0c4j8rkXY2QlpgEk00mEf0at", "host_uuid": "1c95aa14-657e-494d-a617-76e460981339", "labelkey": "customized_label"}] message:[success: Label "elastic-client" was successfully created.]
2019-11-04 15:48:14,962 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/2/assignlabel/ 200] parameters:[{"clabelvalue": "enabled", "clabelkey": "elastic-master", "host_id": "2", "csrfmiddlewaretoken": "PPIuuORIADvIkgBaWCjljpqp17Jqy377iDdY1sNUGu7P5RIMrrE7D2nPSTmduzmC", "host_uuid": "1c95aa14-657e-494d-a617-76e460981339", "labelkey": "customized_label"}] message:[success: Label "elastic-master" was successfully created.]
[Unlock controller-1] 2019-11-04 15:48:27,365 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/ 302] parameters:[{"action": "hostscontroller__unlock__2", "csrfmiddlewaretoken": "NyG412UsUjYBpPElWveljpTak6S9vrIDgmbyyGQE0aAIaqLXrkz7D2QAbSvWrXX8"}] message:[success: Unlocked Host: controller-1]
Steps (roughly) prior to the problem
[Unlocked controller-1 then swact was initiated here] operation_ log: [admin 0b18a42bf702449 a83245107e80739 1f] [admin 801af43854bb4bc 4a8527e9409fa6a fd] [POST /admin/ 302] parameters: [{"action" : "hostscontrolle r__swact_ _1", "csrfmiddleware token": "a7HwE603lUKsk7 PEWLbWXQVPf7rwY ttoWJPWmDWqLrm1 kEsOowBQcze9tVP periZ"} ] message:[success: Swact Initiated Host: controller-0]
2019-11-04 15:25:10,211 [INFO] horizon.
[Lock controller-1 then elastic labels added (via horizon) the controller-1 was unlocked - elastic-data, elastic- controller, elastic client, elastic-master]
2019-11-04 15:46:43,736 [INFO] horizon. operation_ log: [admin 0b18a42bf702449 a83245107e80739 1f] [admin 801af43854bb4bc 4a8527e9409fa6a fd] [POST /admin/ 302] parameters: [{"action" : "hostscontrolle r__lock_ _2", "csrfmiddleware token": "r3wLFmZ4VcOyMt 4wTm5sSQDgRGbAJ TUzUR1fc0Vg13qF x4b8obqectAGIsO nFp94"} ] message:[success: Locking Host: controller-1]
2019-11-04 15:47:45,148 [INFO] horizon. operation_ log: [admin 0b18a42bf702449 a83245107e80739 1f] [admin 801af43854bb4bc 4a8527e9409fa6a fd] [POST /admin/ 2/assignlabel/ 200] parameters: [{"clabelvalue" : "enabled", "clabelkey": "elastic-data", "host_id": "2", "csrfmiddleware token": "UQbQ6qKK1jvcAk PYavBAQoJHheGRF ZstnEGkD4GW7a7j lVWAFkWma1G780j EBvHY", "host_uuid": "1c95aa14- 657e-494d- a617-76e4609813 39", "labelkey": "customized_ label"} ] message:[success: Label "elastic-data" was successfully created.]
2019-11-04 15:47:55,759 [INFO] horizon. operation_ log: [admin 0b18a42bf702449 a83245107e80739 1f] [admin 801af43854bb4bc 4a8527e9409fa6a fd] [POST /admin/ 2/assignlabel/ 200] parameters: [{"clabelvalue" : "enabled", "clabelkey": "elastic- controller" , "host_id": "2", "csrfmiddleware token": "R8lSVzWJzbExf2 8mf6nLP6tg1fOsB JyHkWQmsdSVF2gE 0DfYKVIx9JqGS1r fxfNc", "host_uuid": "1c95aa14- 657e-494d- a617-76e4609813 39", "labelkey": "customized_ label"} ] message:[success: Label "elastic- controller" was successfully created.]
2019-11-04 15:48:05,502 [INFO] horizon. operation_ log: [admin 0b18a42bf702449 a83245107e80739 1f] [admin 801af43854bb4bc 4a8527e9409fa6a fd] [POST /admin/ 2/assignlabel/ 200] parameters: [{"clabelvalue" : "enabled", "clabelkey": "elastic-client", "host_id": "2", "csrfmiddleware token": "2yMXXqt9UlscnQ dltdvz5DHU9eJRj uVYvmhru4pl0c4j 8rkXY2QlpgEk00m Ef0at", "host_uuid": "1c95aa14- 657e-494d- a617-76e4609813 39", "labelkey": "customized_ label"} ] message:[success: Label "elastic-client" was successfully created.]
2019-11-04 15:48:14,962 [INFO] horizon. operation_ log: [admin 0b18a42bf702449 a83245107e80739 1f] [admin 801af43854bb4bc 4a8527e9409fa6a fd] [POST /admin/ 2/assignlabel/ 200] parameters: [{"clabelvalue" : "enabled", "clabelkey": "elastic-master", "host_id": "2", "csrfmiddleware token": "PPIuuORIADvIkg BaWCjljpqp17Jqy 377iDdY1sNUGu7P 5RIMrrE7D2nPSTm duzmC", "host_uuid": "1c95aa14- 657e-494d- a617-76e4609813 39", "labelkey": "customized_ label"} ] message:[success: Label "elastic-master" was successfully created.]
[Unlock controller-1] operation_ log: [admin 0b18a42bf702449 a83245107e80739 1f] [admin 801af43854bb4bc 4a8527e9409fa6a fd] [POST /admin/ 302] parameters: [{"action" : "hostscontrolle r__unlock_ _2", "csrfmiddleware token": "NyG412UsUjYBpP ElWveljpTak6S9v rIDgmbyyGQE0aAI aqLXrkz7D2QAbSv WrXX8"} ] message:[success: Unlocked Host: controller-1]
2019-11-04 15:48:27,365 [INFO] horizon.