kibana-6 migration will not run

[Alexander Diana]

**Bug Report**

What happened:

upgrading to kibana-6 index crashes with syntax errors regarding status codes, and also may get stuck on pre-existing index errors.

What you expected to happen:

kibana-6 upgrade does not crash

How to reproduce it (minimal and precise):

try and migrate from kibana to kibana-6 index, will receive syntax errors and problems which were reported and closed without changes in other tickets:
https://bugs.launchpad.net/kolla-ansible/+bug/1875812
https://bugs.launchpad.net/kolla-ansible/+bug/1881862

these issues are still present.

I have attached a fixed playbook which worked for our deployments.

**Environment**:
* Kolla-Ansible version: stable/ussuri (same in master)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/759833

[Mark Goddard (mgoddard)]

^ Added a CI test for the migration. Let's see what it does.

[Mark Goddard (mgoddard)]

The CI test passed in Train. I notice you are using Ussuri though. Is that upgrading from Train to Ussuri? Which OS are you using?

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a53/759833/2/check/kolla-ansible-centos-source-prometheus-efk/a5364d1/primary/logs/ansible/migrate-elk

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a53/759833/2/check/kolla-ansible-centos-source-prometheus-efk/a5364d1/primary/logs/ansible/test-prometheus-efk

[Mark Goddard (mgoddard)]

I had wondered if it was an issue with idempotence, but it seems not. I ran the upgrade twice (on Train) and it worked.

[Alexander Diana (rouk)]

I can still reproduce it, running debian 10 kolla host (py3.7, ansible 2.9.14), latest ussuri containers, running stable/ussuri kolla.

[Mark Goddard (mgoddard)]

It's still not clear to me why you are running upgrade rather than deploy. Are you upgrading from Train to Ussuri?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/760084

[Mark Goddard (mgoddard)]

Trying again on Ussuri. One difference is the Ansible version.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/760182

[Mark Goddard (mgoddard)]

Failed on Ussuri. I changed status_code to a comma-separated list and it worked.

[Alexander Diana (rouk)]

the other issue is that elastic will return errors due to ansible eating the escaped quotes in the reindex .kibana step.

In my playbook, i had to convert from escaped double quotes, to unescaped single quotes, which is valid elasticsearch "painless" syntax and removes the possibility of issues.

[Mark Goddard (mgoddard)]

How do those errors show? The deployment succeeded without that for me. Are they runtime errors seen when using ES after deployment?

[Alexander Diana (rouk)]

They are reported by ES as rejected requests.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Mark Goddard <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/759833

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Radosław Piliszek <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/759833
Reason: I was just testing something gerrit