rabbitmq wait task defaults to 10s which is not enough

Bug #2057676 reported by Michal Nasiadka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Status tracked in Caracal
Antelope
Fix Committed
Low
Unassigned
Bobcat
Fix Committed
Low
Unassigned
Caracal
Fix Released
Low
Michal Nasiadka
Zed
Fix Released
Low
Unassigned

Bug Description

See https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b1a/899626/3/gate/kolla-ansible-debian-upgrade/b1a6397/primary/logs/ansible/deploy

TASK [rabbitmq : Waiting for rabbitmq to start] ********************************
task path: /home/zuul/kolla-ansible-venv/share/kolla-ansible/ansible/roles/rabbitmq/tasks/restart_services.yml:34
Tuesday 12 March 2024 12:32:58 +0000 (0:00:00.784) 0:02:14.500 *********
Using module file /home/zuul/kolla-ansible-venv/lib/python3.9/site-packages/ansible/modules/command.py
Pipelining is enabled.
<173.231.255.72> ESTABLISH SSH CONNECTION FOR USER: None
<173.231.255.72> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=300 -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/zuul/.ansible/cp/8de95d9bf1"' 173.231.255.72 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-foijwjarqwypnevnzipwhwukspqtkhqe ; /usr/bin/python3'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<173.231.255.72> (1, b'\n{"changed": true, "stdout": "Waiting for pid file \'/var/lib/rabbitmq/mnesia/rabbitmq.pid\' to appear", "stderr": "Error: operation wait on node rabbit@primary timed out. Timeout value used: 10000", "rc": 75, "cmd": ["docker", "exec", "rabbitmq", "rabbitmqctl", "wait", "/var/lib/rabbitmq/mnesia/rabbitmq.pid"], "start": "2024-03-12 12:32:59.032248", "end": "2024-03-12 12:33:10.020800", "delta": "0:00:10.988552", "failed": true, "msg": "non-zero return code", "invocation": {"module_args": {"_raw_params": "docker exec rabbitmq rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbitmq.pid", "_uses_shell": false, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}}\n', b'')
<173.231.255.72> Failed to connect to the host via ssh:
fatal: [primary]: FAILED! => {
    "changed": true,
    "cmd": [
        "docker",
        "exec",
        "rabbitmq",
        "rabbitmqctl",
        "wait",
        "/var/lib/rabbitmq/mnesia/rabbitmq.pid"
    ],
    "delta": "0:00:10.988552",
    "end": "2024-03-12 12:33:10.020800",
    "invocation": {
        "module_args": {
            "_raw_params": "docker exec rabbitmq rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbitmq.pid",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true
        }
    },
    "msg": "non-zero return code",
    "rc": 75,
    "start": "2024-03-12 12:32:59.032248",
    "stderr": "Error: operation wait on node rabbit@primary timed out. Timeout value used: 10000",
    "stderr_lines": [
        "Error: operation wait on node rabbit@primary timed out. Timeout value used: 10000"
    ],
    "stdout": "Waiting for pid file '/var/lib/rabbitmq/mnesia/rabbitmq.pid' to appear",
    "stdout_lines": [
        "Waiting for pid file '/var/lib/rabbitmq/mnesia/rabbitmq.pid' to appear"
    ]
}

2024-03-12 12:33:14.757358+00:00 [info] <0.9.0> Time to start RabbitMQ: 14752747 us - which is nearly 15 seconds

Changed in kolla-ansible:
status: New → Triaged
Changed in kolla-ansible:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/912586
Committed: https://opendev.org/openstack/kolla-ansible/commit/7bb50ee05e402899deaa488ff22ea3c94b70b86f
Submitter: "Zuul (22348)"
Branch: master

commit 7bb50ee05e402899deaa488ff22ea3c94b70b86f
Author: Michal Nasiadka <email address hidden>
Date: Tue Mar 12 15:30:04 2024 +0100

    rabbitmq: bump wait timeout to 60 seconds

    Closes-Bug: #2057676

    Change-Id: I9e0287a4e80b1ebcecf9e3b66c11d4233970a30b

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/912079

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/912080

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/912741

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/912080
Committed: https://opendev.org/openstack/kolla-ansible/commit/edc8ef8c13b797092361857d617c329a77a220fc
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit edc8ef8c13b797092361857d617c329a77a220fc
Author: Michal Nasiadka <email address hidden>
Date: Tue Mar 12 15:30:04 2024 +0100

    rabbitmq: bump wait timeout to 60 seconds

    Closes-Bug: #2057676

    Change-Id: I9e0287a4e80b1ebcecf9e3b66c11d4233970a30b
    (cherry picked from commit 7bb50ee05e402899deaa488ff22ea3c94b70b86f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/912079
Committed: https://opendev.org/openstack/kolla-ansible/commit/d82c2deb80f1259d02bf3868b451f500f0a24546
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit d82c2deb80f1259d02bf3868b451f500f0a24546
Author: Michal Nasiadka <email address hidden>
Date: Tue Mar 12 15:30:04 2024 +0100

    rabbitmq: bump wait timeout to 60 seconds

    Closes-Bug: #2057676

    Change-Id: I9e0287a4e80b1ebcecf9e3b66c11d4233970a30b
    (cherry picked from commit 7bb50ee05e402899deaa488ff22ea3c94b70b86f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/912741
Committed: https://opendev.org/openstack/kolla-ansible/commit/d5ab0e71071dfe90ddd3ea121dc64df941d03d13
Submitter: "Zuul (22348)"
Branch: stable/zed

commit d5ab0e71071dfe90ddd3ea121dc64df941d03d13
Author: Michal Nasiadka <email address hidden>
Date: Tue Mar 12 15:30:04 2024 +0100

    rabbitmq: bump wait timeout to 60 seconds

    Closes-Bug: #2057676

    Change-Id: I9e0287a4e80b1ebcecf9e3b66c11d4233970a30b
    (cherry picked from commit 7bb50ee05e402899deaa488ff22ea3c94b70b86f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible zed-eom

This issue was fixed in the openstack/kolla-ansible zed-eom release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.