Steps (roughly) prior to the problem

[Unlocked controller-1 then swact was initiated here]
2019-11-04 15:25:10,211 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/ 302] parameters:[{"action": "hostscontroller__swact__1", "csrfmiddlewaretoken": "a7HwE603lUKsk7PEWLbWXQVPf7rwYttoWJPWmDWqLrm1kEsOowBQcze9tVPperiZ"}] message:[success: Swact Initiated Host: controller-0]

[Lock controller-1 then elastic labels added (via horizon) the controller-1 was unlocked - elastic-data, elastic-controller,elastic client, elastic-master]

2019-11-04 15:46:43,736 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/ 302] parameters:[{"action": "hostscontroller__lock__2", "csrfmiddlewaretoken": "r3wLFmZ4VcOyMt4wTm5sSQDgRGbAJTUzUR1fc0Vg13qFx4b8obqectAGIsOnFp94"}] message:[success: Locking Host: controller-1]

2019-11-04 15:47:45,148 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/2/assignlabel/ 200] parameters:[{"clabelvalue": "enabled", "clabelkey": "elastic-data", "host_id": "2", "csrfmiddlewaretoken": "UQbQ6qKK1jvcAkPYavBAQoJHheGRFZstnEGkD4GW7a7jlVWAFkWma1G780jEBvHY", "host_uuid": "1c95aa14-657e-494d-a617-76e460981339", "labelkey": "customized_label"}] message:[success: Label "elastic-data" was successfully created.]

2019-11-04 15:47:55,759 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/2/assignlabel/ 200] parameters:[{"clabelvalue": "enabled", "clabelkey": "elastic-controller", "host_id": "2", "csrfmiddlewaretoken": "R8lSVzWJzbExf28mf6nLP6tg1fOsBJyHkWQmsdSVF2gE0DfYKVIx9JqGS1rfxfNc", "host_uuid": "1c95aa14-657e-494d-a617-76e460981339", "labelkey": "customized_label"}] message:[success: Label "elastic-controller" was successfully created.]

2019-11-04 15:48:05,502 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/2/assignlabel/ 200] parameters:[{"clabelvalue": "enabled", "clabelkey": "elastic-client", "host_id": "2", "csrfmiddlewaretoken": "2yMXXqt9UlscnQdltdvz5DHU9eJRjuVYvmhru4pl0c4j8rkXY2QlpgEk00mEf0at", "host_uuid": "1c95aa14-657e-494d-a617-76e460981339", "labelkey": "customized_label"}] message:[success: Label "elastic-client" was successfully created.]

2019-11-04 15:48:14,962 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/2/assignlabel/ 200] parameters:[{"clabelvalue": "enabled", "clabelkey": "elastic-master", "host_id": "2", "csrfmiddlewaretoken": "PPIuuORIADvIkgBaWCjljpqp17Jqy377iDdY1sNUGu7P5RIMrrE7D2nPSTmduzmC", "host_uuid": "1c95aa14-657e-494d-a617-76e460981339", "labelkey": "customized_label"}] message:[success: Label "elastic-master" was successfully created.]

[Unlock controller-1]
2019-11-04 15:48:27,365 [INFO] horizon.operation_log: [admin 0b18a42bf702449a83245107e807391f] [admin 801af43854bb4bc4a8527e9409fa6afd] [POST /admin/ 302] parameters:[{"action": "hostscontroller__unlock__2", "csrfmiddlewaretoken": "NyG412UsUjYBpPElWveljpTak6S9vrIDgmbyyGQE0aAIaqLXrkz7D2QAbSvWrXX8"}] message:[success: Unlocked Host: controller-1]