"Find gospel node" task crashes during upgrade

[Vikram Hosakote]

The "Find gospel node" task crashes during upgrade. Below are the steps to reproduce the bug.

$ grep release /etc/kolla/globals.yml
openstack_release: "2.0.1"

$ kolla-build --base ubuntu --type source --debug --registry 192.168.122.201:4000 --tag 2.0.1 --push

$ sudo kolla-ansible upgrade

Below are the errors.

TASK: [rabbitmq | Find gospel node] *******************************************
failed: [localhost] => {"changed": true, "cmd": ["docker", "exec", "-t", "rabbitmq", "/usr/local/bin/rabbitmq_get_gospel_node"], "delta": "0:00:01.052517", "end": "2016-04-22 10:42:46.944975", "failed": true, "failed_when_result": true, "rc": 0, "start": "2016-04-22 10:42:45.892458", "stdout_lines": ["{\"failed\": true, \"changed\": true, \"error\": \"Traceback (most recent call last):\\n File \\\"/usr/local/bin/rabbitmq_get_gospel_node\\\", line 38, in main\\n '\\\\n'.join(raw_status.split('\\\\n')[1:-3])\\n File \\\"/usr/local/bin/rabbitmq_get_gospel_node\\\", line 26, in convert_erlang_to_json\\n return json.loads(term)\\n File \\\"/usr/lib/python2.7/json/__init__.py\\\", line 338, in loads\\n return _default_decoder.decode(s)\\n File \\\"/usr/lib/python2.7/json/decoder.py\\\", line 366, in decode\\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\\n File \\\"/usr/lib/python2.7/json/decoder.py\\\", line 382, in raw_decode\\n obj, end = self.scan_once(s, idx)\\nValueError: Expecting property name: line 1 column 16 (char 15)\\n\"}"], "warnings": []}
stdout: {"failed": true, "changed": true, "error": "Traceback (most recent call last):\n File \"/usr/local/bin/rabbitmq_get_gospel_node\", line 38, in main\n '\\n'.join(raw_status.split('\\n')[1:-3])\n File \"/usr/local/bin/rabbitmq_get_gospel_node\", line 26, in convert_erlang_to_json\n return json.loads(term)\n File \"/usr/lib/python2.7/json/__init__.py\", line 338, in loads\n return _default_decoder.decode(s)\n File \"/usr/lib/python2.7/json/decoder.py\", line 366, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n File \"/usr/lib/python2.7/json/decoder.py\", line 382, in raw_decode\n obj, end = self.scan_once(s, idx)\nValueError: Expecting property name: line 1 column 16 (char 15)\n"}

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/home/kolla/site.retry

localhost : ok=52 changed=7 unreachable=0 failed=1

Command failed ansible-playbook -i /usr/local/share/kolla/ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=upgrade /usr/local/share/kolla/ansible/site.yml

[Vikram Hosakote (vhosakot)]

Review is at https://review.openstack.org/#/c/309448/.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309448
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=22a07075d278d592bccfe9ca3fc1b670248d706d
Submitter: Jenkins
Branch: master

commit 22a07075d278d592bccfe9ca3fc1b670248d706d
Author: Michal (inc0) Jastrzebski <email address hidden>
Date: Fri Apr 22 14:44:54 2016 +0000

    Robustify of rabbitmq upgrade

    String parsing for gospel node detection was overly complex and not
    robust. Hopefully this will make it better.

    Closes-bug: #1573684
    Change-Id: I12e869eff274d286d239c4f55cc4b4ddca389bbd

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/309528

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/309529

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309529
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=5ef62c121bdf6b3fa4f386a2b3921cf8c14a13bc
Submitter: Jenkins
Branch: stable/liberty

commit 5ef62c121bdf6b3fa4f386a2b3921cf8c14a13bc
Author: Michal (inc0) Jastrzebski <email address hidden>
Date: Fri Apr 22 14:44:54 2016 +0000

    Robustify of rabbitmq upgrade

    String parsing for gospel node detection was overly complex and not
    robust. Hopefully this will make it better.

    Closes-bug: #1573684
    Change-Id: I12e869eff274d286d239c4f55cc4b4ddca389bbd

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309528
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=b63771f1c874864441ae82abc00422fb369905f7
Submitter: Jenkins
Branch: stable/mitaka

commit b63771f1c874864441ae82abc00422fb369905f7
Author: Michal (inc0) Jastrzebski <email address hidden>
Date: Fri Apr 22 14:44:54 2016 +0000

    Robustify of rabbitmq upgrade

    String parsing for gospel node detection was overly complex and not
    robust. Hopefully this will make it better.

    Closes-bug: #1573684
    Change-Id: I12e869eff274d286d239c4f55cc4b4ddca389bbd

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 2.0.0 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 1.1.0 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 3.0.0.0b1 development milestone.