Task "haproxy : Checking if kolla_internal_vip_address" fails when adding a new controller

[Paulo Angelo]

Hi all,

We deployed two controllers via kolla-ansible. Now, we tried to deploy a third controller, but its failing in task:

TASK [haproxy : Checking if kolla_internal_vip_address and kolla_external_vip_address are not pingable from any node]

Is there a workaround for this?

Best regards,

Paulo Angelo

============== Environment information ==============

Name: kolla-ansible
Version: 9.0.1

Linux controller2.gigaponto.local 4.15.0-109-generic #110-Ubuntu SMP Tue Jun 23 02:39:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Ubuntu 18.04 LTS

Client: Docker Engine - Community
 Version: 19.03.5
 API version: 1.40
 Go version: go1.12.12
 Git commit: 633a0ea838
 Built: Wed Nov 13 07:29:52 2019
 OS/Arch: linux/amd64
 Experimental: false

Server: Docker Engine - Community
 Engine:
  Version: 19.03.6
  API version: 1.40 (minimum version 1.12)
  Go version: go1.12.16
  Git commit: 369ce74a3c
  Built: Thu Feb 13 01:26:21 2020
  OS/Arch: linux/amd64
  Experimental: false
 containerd:
  Version: 1.2.10
  GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version: 1.0.0-rc8+dev
  GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version: 0.18.0
  GitCommit: fec3683

============== Debug msgs ==============

TASK [haproxy : Checking if kolla_internal_vip_address and kolla_external_vip_address are not pingable from any node] *****************************************************************************************
task path: /usr/local/share/kolla-ansible/ansible/roles/haproxy/tasks/precheck.yml:120
Using module file /usr/local/lib/python3.6/dist-packages/ansible/modules/commands/command.py
<controller3> ESTABLISH SSH CONNECTION FOR USER: openstack
<controller3> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=160s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/openstack/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openstack -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/08ac2ef8a3 controller3 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-eqoncevscfrrlzknnouiiicyznohcpzg; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<controller3> (0, b'\n{"changed": true, "end": "2020-07-29 11:13:31.005552", "stdout": "PING 10.2.1.10 (10.2.1.10) 56(84) bytes of data.\\n64 bytes from 10.2.1.10: icmp_seq=1 ttl=64 time=0.301 ms\\n64 bytes from 10.2.1.10: icmp_seq=2 ttl=64 time=0.216 ms\\n64 bytes from 10.2.1.10: icmp_seq=2 ttl=64 time=0.235 ms (DUP!)\\n64 bytes from 10.2.1.10: icmp_seq=3 ttl=64 time=0.231 ms\\n\\n--- 10.2.1.10 ping statistics ---\\n3 packets transmitted, 3 received, +1 duplicates, 0% packet loss, time 2030ms\\nrtt min/avg/max/mdev = 0.216/0.245/0.301/0.037 ms", "cmd": ["ping", "-c", "3", "10.2.1.10"], "rc": 0, "start": "2020-07-29 11:13:28.972279", "stderr": "", "delta": "0:00:02.033273", "invocation": {"module_args": {"creates": null, "executable": null, "_uses_shell": false, "_raw_params": "ping -c 3 10.2.1.10", "removes": null, "argv": null, "warn": true, "chdir": null, "stdin": null}}}\n', b'')
failed: [controller3] (item={'address': '10.2.1.10', 'command': 'ping'}) => {
    "changed": false,
    "cmd": [
        "ping",
        "-c",
        "3",
        "10.2.1.10"
    ],
    "delta": "0:00:02.033273",
    "end": "2020-07-29 11:13:31.005552",
    "failed_when_result": true,
    "invocation": {
        "module_args": {
            "_raw_params": "ping -c 3 10.2.1.10",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": {
        "address": "10.2.1.10",
        "command": "ping"
    },
    "rc": 0,
    "start": "2020-07-29 11:13:28.972279",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "PING 10.2.1.10 (10.2.1.10) 56(84) bytes of data.\n64 bytes from 10.2.1.10: icmp_seq=1 ttl=64 time=0.301 ms\n64 bytes from 10.2.1.10: icmp_seq=2 ttl=64 time=0.216 ms\n64 bytes from 10.2.1.10: icmp_seq=2 ttl=64 time=0.235 ms (DUP!)\n64 bytes from 10.2.1.10: icmp_seq=3 ttl=64 time=0.231 ms\n\n--- 10.2.1.10 ping statistics ---\n3 packets transmitted, 3 received, +1 duplicates, 0% packet loss, time 2030ms\nrtt min/avg/max/mdev = 0.216/0.245/0.301/0.037 ms",
    "stdout_lines": [
        "PING 10.2.1.10 (10.2.1.10) 56(84) bytes of data.",
        "64 bytes from 10.2.1.10: icmp_seq=1 ttl=64 time=0.301 ms",
        "64 bytes from 10.2.1.10: icmp_seq=2 ttl=64 time=0.216 ms",
        "64 bytes from 10.2.1.10: icmp_seq=2 ttl=64 time=0.235 ms (DUP!)",
        "64 bytes from 10.2.1.10: icmp_seq=3 ttl=64 time=0.231 ms",
        "",
        "--- 10.2.1.10 ping statistics ---",
        "3 packets transmitted, 3 received, +1 duplicates, 0% packet loss, time 2030ms",
        "rtt min/avg/max/mdev = 0.216/0.245/0.301/0.037 ms"
    ]
}
Using module file /usr/local/lib/python3.6/dist-packages/ansible/modules/commands/command.py
<controller3> ESTABLISH SSH CONNECTION FOR USER: openstack
<controller3> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=160s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/openstack/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=openstack -o ConnectTimeout=30 -o ControlPath=/root/.ansible/cp/08ac2ef8a3 controller3 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-onjzeikfkwrbtlediwmqxskhoepeocdj; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<controller3> (0, b'\n{"changed": true, "end": "2020-07-29 11:13:33.277774", "stdout": "PING 10.2.1.9 (10.2.1.9) 56(84) bytes of data.\\n64 bytes from 10.2.1.9: icmp_seq=1 ttl=64 time=0.195 ms\\n64 bytes from 10.2.1.9: icmp_seq=2 ttl=64 time=0.342 ms\\n64 bytes from 10.2.1.9: icmp_seq=3 ttl=64 time=0.256 ms\\n\\n--- 10.2.1.9 ping statistics ---\\n3 packets transmitted, 3 received, 0% packet loss, time 2055ms\\nrtt min/avg/max/mdev = 0.195/0.264/0.342/0.061 ms", "cmd": ["ping", "-c", "3", "10.2.1.9"], "rc": 0, "start": "2020-07-29 11:13:31.219510", "stderr": "", "delta": "0:00:02.058264", "invocation": {"module_args": {"creates": null, "executable": null, "_uses_shell": false, "_raw_params": "ping -c 3 10.2.1.9", "removes": null, "argv": null, "warn": true, "chdir": null, "stdin": null}}}\n', b'')
failed: [controller3] (item={'address': '10.2.1.9', 'command': 'ping'}) => {
    "changed": false,
    "cmd": [
        "ping",
        "-c",
        "3",
        "10.2.1.9"
    ],
    "delta": "0:00:02.058264",
    "end": "2020-07-29 11:13:33.277774",
    "failed_when_result": true,
    "invocation": {
        "module_args": {
            "_raw_params": "ping -c 3 10.2.1.9",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": {
        "address": "10.2.1.9",
        "command": "ping"
    },
    "rc": 0,
    "start": "2020-07-29 11:13:31.219510",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "PING 10.2.1.9 (10.2.1.9) 56(84) bytes of data.\n64 bytes from 10.2.1.9: icmp_seq=1 ttl=64 time=0.195 ms\n64 bytes from 10.2.1.9: icmp_seq=2 ttl=64 time=0.342 ms\n64 bytes from 10.2.1.9: icmp_seq=3 ttl=64 time=0.256 ms\n\n--- 10.2.1.9 ping statistics ---\n3 packets transmitted, 3 received, 0% packet loss, time 2055ms\nrtt min/avg/max/mdev = 0.195/0.264/0.342/0.061 ms",
    "stdout_lines": [
        "PING 10.2.1.9 (10.2.1.9) 56(84) bytes of data.",
        "64 bytes from 10.2.1.9: icmp_seq=1 ttl=64 time=0.195 ms",
        "64 bytes from 10.2.1.9: icmp_seq=2 ttl=64 time=0.342 ms",
        "64 bytes from 10.2.1.9: icmp_seq=3 ttl=64 time=0.256 ms",
        "",
        "--- 10.2.1.9 ping statistics ---",
        "3 packets transmitted, 3 received, 0% packet loss, time 2055ms",
        "rtt min/avg/max/mdev = 0.195/0.264/0.342/0.061 ms"
    ]
}

LAY RECAP ****************************************************************************************************************************************************************************************************
controller3 : ok=30 changed=0 unreachable=0 failed=1

[Mark Goddard (mgoddard)]

Please update kolla-ansible to a new version in the Train series - this has been fixed.

[Alysson Filipe (alyfilipe)]

Hi.

We upgraded the kolla to version 10.1.0. However, now we have a new error. At the time of the deployment he complains about an image of "kolla/ubuntu-source-mariadb-clustercheck:train". I thought it was something new regarding the new version, but when I try to execute the pull, it also gives the error. It seems to me that there is no such thing in the docker hub.

Any ideas about this error?

TASK [Pulling mariadb image] ****************************************************************************************************************************************************************************************
ok: [localhost] => (item={'key': 'mariadb', 'value': {'container_name': 'mariadb', 'group': 'mariadb', 'enabled': True, 'image': 'kolla/ubuntu-source-mariadb:train', 'volumes': ['/etc/kolla/mariadb/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'mariadb:/var/lib/mysql', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'haproxy': {'mariadb': {'enabled': True, 'mode': 'tcp', 'port': '3306', 'listen_port': '3306', 'frontend_tcp_extra': ['option clitcpka', 'timeout client 3600s'], 'backend_tcp_extra': ['option srvtcpka', 'timeout server 3600s', 'option httpchk'], 'custom_member_list': ['server kolla-ansible 10.2.1.252:3306 check port 4569 inter 2000 rise 2 fall 5', '']}, 'mariadb_external_lb': {'enabled': False, 'mode': 'tcp', 'port': '3306', 'listen_port': '3306', 'frontend_tcp_extra': ['option clitcpka', 'timeout client 3600s'], 'backend_tcp_extra': ['option srvtcpka', 'timeout server 3600s'], 'custom_member_list': ['server localhost localhost:3306 check port 4569 inter 2000 rise 2 fall 5', '']}}}})
failed: [localhost] (item={'key': 'mariadb-clustercheck', 'value': {'container_name': 'mariadb_clustercheck', 'group': 'mariadb', 'enabled': True, 'image': 'kolla/ubuntu-source-mariadb-clustercheck:train', 'volumes': ['/etc/kolla/mariadb-clustercheck/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/etc/timezone:/etc/timezone:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}, 'environment': {'MYSQL_USERNAME': 'haproxy', 'MYSQL_PASSWORD': '', 'MYSQL_HOST': '10.2.1.252', 'AVAILABLE_WHEN_DONOR': '1'}}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "mariadb-clustercheck", "value": {"container_name": "mariadb_clustercheck", "dimensions": {}, "enabled": true, "environment": {"AVAILABLE_WHEN_DONOR": "1", "MYSQL_HOST": "10.2.1.252", "MYSQL_PASSWORD": "", "MYSQL_USERNAME": "haproxy"}, "group": "mariadb", "image": "kolla/ubuntu-source-mariadb-clustercheck:train", "volumes": ["/etc/kolla/mariadb-clustercheck/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "/etc/timezone:/etc/timezone:ro", "kolla_logs:/var/log/kolla/"]}}, "msg": "'Traceback (most recent call last):\\n File \"/usr/local/lib/python3.6/dist-packages/docker/api/client.py\", line 261, in _raise_for_status\\n response.raise_for_status()\\n File \"/usr/local/lib/python3.6/dist-packages/requests/models.py\", line 940, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+doc...

[Radosław Piliszek (yoctozepto)]

You upgraded Kolla-Ansible to Ussuri. It won't work with Train. Please keep to 9.x.y versions for Train.