tripleo

Bug #1951260
Comment #1

Comment 1 for bug 1951260

Revision history for this message

Juan Badia Payno (jbadiapa) wrote on 2021-11-17:

well, The workaround makes the job fail later (still on the provisioning)

PLAY [Overcloud Node Grow Volumes] *********************************************
2021-11-17 12:37:33.240702 | 002d6d46-e31e-3db7-c63e-00000000000f | TASK | Wait for provisioned nodes to boot
<overcloud-controller-1> Attempting python interpreter discovery
<overcloud-controller-0> Attempting python interpreter discovery
<overcloud-controller-2> Attempting python interpreter discovery
<192.168.24.24> ESTABLISH LOCAL CONNECTION FOR USER: stack
<192.168.24.8> ESTABLISH LOCAL CONNECTION FOR USER: stack
<192.168.24.8> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<192.168.24.24> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<192.168.24.16> ESTABLISH LOCAL CONNECTION FOR USER: stack
<192.168.24.16> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<192.168.24.8> EXEC /bin/sh -c '/usr/bin/python3.6 && sleep 0'
<192.168.24.24> EXEC /bin/sh -c '/usr/bin/python3.6 && sleep 0'
<192.168.24.16> EXEC /bin/sh -c '/usr/bin/python3.6 && sleep 0'
Using module file /usr/lib/python3.6/site-packages/ansible/modules/utilities/logic/wait_for.py
Using module file /usr/lib/python3.6/site-packages/ansible/modules/utilities/logic/wait_for.py
Pipelining is enabled.
<192.168.24.24> EXEC /bin/sh -c '/usr/libexec/platform-python && sleep 0'
Using module file /usr/lib/python3.6/site-packages/ansible/modules/utilities/logic/wait_for.py
Pipelining is enabled.
<192.168.24.16> EXEC /bin/sh -c '/usr/libexec/platform-python && sleep 0'
Pipelining is enabled.
<192.168.24.8> EXEC /bin/sh -c '/usr/libexec/platform-python && sleep 0'

2021-11-17 12:47:35.540329 | 002d6d46-e31e-3db7-c63e-00000000000f | FATAL | Wait for provisioned nodes to boot | overcloud-controller-0 | error={
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "elapsed": 601,
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED",
                "FIN_WAIT1",
                "FIN_WAIT2",
                "SYN_RECV",
                "SYN_SENT",
                "TIME_WAIT"
            ],
            "connect_timeout": 5,
            "delay": 10,
            "exclude_hosts": null,
            "host": "192.168.24.8",
            "msg": "Timeout waiting for provisioned nodes to become available",
            "path": null,
            "port": 22,
            "search_regex": "OpenSSH",
            "sleep": 1,
            "state": "started",
            "timeout": 600
        }
    },
    "msg": "Timeout waiting for provisioned nodes to become available"
}
2021-11-17 12:47:35.542276 | 002d6d46-e31e-3db7-c63e-00000000000f | TIMING | Wait for provisioned nodes to boot | overcloud-controller-0 | 0:10:02.336808 | 602.30s
2021-11-17 12:47:35.544356 | 002d6d46-e31e-3db7-c63e-00000000000f | FATAL | Wait for provisioned nodes to boot | overcloud-controller-2 | error={
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "elapsed": 601,
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED",
                "FIN_WAIT1",
                "FIN_WAIT2",
                "SYN_RECV",
                "SYN_SENT",
                "TIME_WAIT"
            ],
            "connect_timeout": 5,
            "delay": 10,
            "exclude_hosts": null,
            "host": "192.168.24.24",
            "msg": "Timeout waiting for provisioned nodes to become available",
            "path": null,
            "port": 22,
            "search_regex": "OpenSSH",
            "sleep": 1,
            "state": "started",
            "timeout": 600
        }
    },
    "msg": "Timeout waiting for provisioned nodes to become available"
}
2021-11-17 12:47:35.545929 | 002d6d46-e31e-3db7-c63e-00000000000f | TIMING | Wait for provisioned nodes to boot | overcloud-controller-2 | 0:10:02.340467 | 602.28s
2021-11-17 12:47:35.548195 | 002d6d46-e31e-3db7-c63e-00000000000f | FATAL | Wait for provisioned nodes to boot | overcloud-controller-1 | error={
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "elapsed": 601,
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED",
                "FIN_WAIT1",
                "FIN_WAIT2",
                "SYN_RECV",
                "SYN_SENT",
                "TIME_WAIT"
            ],
            "connect_timeout": 5,
            "delay": 10,
            "exclude_hosts": null,
            "host": "192.168.24.16",
            "msg": "Timeout waiting for provisioned nodes to become available",
            "path": null,
            "port": 22,
            "search_regex": "OpenSSH",
            "sleep": 1,
            "state": "started",
            "timeout": 600
        }
    },
    "msg": "Timeout waiting for provisioned nodes to become available"
}
2021-11-17 12:47:35.549798 | 002d6d46-e31e-3db7-c63e-00000000000f | TIMING | Wait for provisioned nodes to boot | overcloud-controller-1 | 0:10:02.344344 | 602.30s

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
overcloud-controller-0 : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
overcloud-controller-1 : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
overcloud-controller-2 : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
2021-11-17 12:47:35.556990 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-11-17 12:47:35.557762 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-11-17 12:47:35.558318 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:10:02.352874 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-11-17 12:47:35.562738 | UUID | Info | Host | Task Name | Run Time
2021-11-17 12:47:35.563645 | 002d6d46-e31e-3db7-c63e-00000000000f | SUMMARY | overcloud-controller-0 | Wait for provisioned nodes to boot | 602.30s
2021-11-17 12:47:35.564726 | 002d6d46-e31e-3db7-c63e-00000000000f | SUMMARY | overcloud-controller-1 | Wait for provisioned nodes to boot | 602.30s
2021-11-17 12:47:35.565755 | 002d6d46-e31e-3db7-c63e-00000000000f | SUMMARY | overcloud-controller-2 | Wait for provisioned nodes to boot | 602.28s
2021-11-17 12:47:35.566623 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-11-17 12:47:35.567422 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-11-17 12:47:35.568458 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 3 ~~~~~~~~~~~~~~~~~
2021-11-17 12:47:35.569352 | The following node(s) had failures: overcloud-controller-0, overcloud-controller-1, overcloud-controller-2

The console of the overcloud-controller-X:

Booting from Hard Disk

well, The workaround makes the job fail later (still on the provisioning)

PLAY [Overcloud Node Grow Volumes] *********************************************
2021-11-17 12:37:33.240702 | 002d6d46-e31e-3db7-c63e-00000000000f |       TASK | Wait for provisioned nodes to boot
<overcloud-controller-1> Attempting python interpreter discovery
<overcloud-controller-0> Attempting python interpreter discovery
<overcloud-controller-2> Attempting python interpreter discovery
<192.168.24.24> ESTABLISH LOCAL CONNECTION FOR USER: stack
<192.168.24.8> ESTABLISH LOCAL CONNECTION FOR USER: stack
<192.168.24.8> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<192.168.24.24> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<192.168.24.16> ESTABLISH LOCAL CONNECTION FOR USER: stack
<192.168.24.16> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<192.168.24.8> EXEC /bin/sh -c '/usr/bin/python3.6 && sleep 0'
<192.168.24.24> EXEC /bin/sh -c '/usr/bin/python3.6 && sleep 0'
<192.168.24.16> EXEC /bin/sh -c '/usr/bin/python3.6 && sleep 0'
Using module file /usr/lib/python3.6/site-packages/ansible/modules/utilities/logic/wait_for.py
Using module file /usr/lib/python3.6/site-packages/ansible/modules/utilities/logic/wait_for.py
Pipelining is enabled.
<192.168.24.24> EXEC /bin/sh -c '/usr/libexec/platform-python && sleep 0'
Using module file /usr/lib/python3.6/site-packages/ansible/modules/utilities/logic/wait_for.py
Pipelining is enabled.
<192.168.24.16> EXEC /bin/sh -c '/usr/libexec/platform-python && sleep 0'
Pipelining is enabled.
<192.168.24.8> EXEC /bin/sh -c '/usr/libexec/platform-python && sleep 0'

2021-11-17 12:47:35.540329 | 002d6d46-e31e-3db7-c63e-00000000000f |      FATAL | Wait for provisioned nodes to boot | overcloud-controller-0 | error={
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "elapsed": 601,
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED",
                "FIN_WAIT1",
                "FIN_WAIT2",
                "SYN_RECV",
                "SYN_SENT",
                "TIME_WAIT"
            ],
            "connect_timeout": 5,
            "delay": 10,
            "exclude_hosts": null,
            "host": "192.168.24.8",
            "msg": "Timeout waiting for provisioned nodes to become available",
            "path": null,
            "port": 22,
            "search_regex": "OpenSSH",
            "sleep": 1,
            "state": "started",
            "timeout": 600
        }
    },
    "msg": "Timeout waiting for provisioned nodes to become available"
}
2021-11-17 12:47:35.542276 | 002d6d46-e31e-3db7-c63e-00000000000f |     TIMING | Wait for provisioned nodes to boot | overcloud-controller-0 | 0:10:02.336808 | 602.30s
2021-11-17 12:47:35.544356 | 002d6d46-e31e-3db7-c63e-00000000000f |      FATAL | Wait for provisioned nodes to boot | overcloud-controller-2 | error={
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "elapsed": 601,
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED",
                "FIN_WAIT1",
                "FIN_WAIT2",
                "SYN_RECV",
                "SYN_SENT",
                "TIME_WAIT"
            ],
            "connect_timeout": 5,
            "delay": 10,
            "exclude_hosts": null,
            "host": "192.168.24.24",
            "msg": "Timeout waiting for provisioned nodes to become available",
            "path": null,
            "port": 22,
            "search_regex": "OpenSSH",
            "sleep": 1,
            "state": "started",
            "timeout": 600
        }
    },
    "msg": "Timeout waiting for provisioned nodes to become available"
}
2021-11-17 12:47:35.545929 | 002d6d46-e31e-3db7-c63e-00000000000f |     TIMING | Wait for provisioned nodes to boot | overcloud-controller-2 | 0:10:02.340467 | 602.28s
2021-11-17 12:47:35.548195 | 002d6d46-e31e-3db7-c63e-00000000000f |      FATAL | Wait for provisioned nodes to boot | overcloud-controller-1 | error={
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "elapsed": 601,
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED",
                "FIN_WAIT1",
                "FIN_WAIT2",
                "SYN_RECV",
                "SYN_SENT",
                "TIME_WAIT"
            ],
            "connect_timeout": 5,
            "delay": 10,
            "exclude_hosts": null,
            "host": "192.168.24.16",
            "msg": "Timeout waiting for provisioned nodes to become available",
            "path": null,
            "port": 22,
            "search_regex": "OpenSSH",
            "sleep": 1,
            "state": "started",
            "timeout": 600
        }
    },
    "msg": "Timeout waiting for provisioned nodes to become available"
}
2021-11-17 12:47:35.549798 | 002d6d46-e31e-3db7-c63e-00000000000f |     TIMING | Wait for provisioned nodes to boot | overcloud-controller-1 | 0:10:02.344344 | 602.30s

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
overcloud-controller-0     : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
overcloud-controller-1     : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
overcloud-controller-2     : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
2021-11-17 12:47:35.556990 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-11-17 12:47:35.557762 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-11-17 12:47:35.558318 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:10:02.352874 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-11-17 12:47:35.562738 |                                 UUID |       Info |       Host |   Task Name |   Run Time
2021-11-17 12:47:35.563645 | 002d6d46-e31e-3db7-c63e-00000000000f |    SUMMARY | overcloud-controller-0 | Wait for provisioned nodes to boot | 602.30s
2021-11-17 12:47:35.564726 | 002d6d46-e31e-3db7-c63e-00000000000f |    SUMMARY | overcloud-controller-1 | Wait for provisioned nodes to boot | 602.30s
2021-11-17 12:47:35.565755 | 002d6d46-e31e-3db7-c63e-00000000000f |    SUMMARY | overcloud-controller-2 | Wait for provisioned nodes to boot | 602.28s
2021-11-17 12:47:35.566623 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-11-17 12:47:35.567422 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-11-17 12:47:35.568458 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 3 ~~~~~~~~~~~~~~~~~
2021-11-17 12:47:35.569352 |  The following node(s) had failures: overcloud-controller-0, overcloud-controller-1, overcloud-controller-2

The console of the overcloud-controller-X:

Booting from Hard Disk