ci.centos gate job fails at Taks: Wait for cloud-init to finish, if enabled

[yatin]

It fails as below:-
2021-01-28 12:45:56.219646 | 00cec312-dbfb-8f09-106f-000000000029 | RETRY | Wait for cloud-init to finish, if enabled | standalone | 4 retries left
2021-01-28 12:46:01.570955 | 00cec312-dbfb-8f09-106f-000000000029 | RETRY | Wait for cloud-init to finish, if enabled | standalone | 3 retries left
2021-01-28 12:46:06.915090 | 00cec312-dbfb-8f09-106f-000000000029 | RETRY | Wait for cloud-init to finish, if enabled | standalone | 2 retries left
2021-01-28 12:46:12.264037 | 00cec312-dbfb-8f09-106f-000000000029 | RETRY | Wait for cloud-init to finish, if enabled | standalone | 1 retries left
2021-01-28 12:46:17.616718 | 00cec312-dbfb-8f09-106f-000000000029 | FATAL | Wait for cloud-init to finish, if enabled | standalone | error={"ansible_facts": {"cloud_init_data_facts": {"status": {}}}, "attempts": 50, "changed": false, "cloud_init_data_facts": {"status": {}}}
2021-01-28 12:46:17.618451 | 00cec312-dbfb-8f09-106f-000000000029 | TIMING | Wait for cloud-init to finish, if enabled | standalone | 0:05:14.082095 | 268.28s

Example log:- https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-gate-master-delorean-full-featureset052-483/undercloud/home/stack/standalone_deploy.log

In these env, cloud-init is installed and enabled but not started, leading to res.cloud_init_data_facts.status.v1.stage never get defined.

Info from system:-
[stack@undercloud ~]$ sudo systemctl status cloud-init
● cloud-init.service - Initial cloud-init job (metadata service crawler)
   Loaded: loaded (/usr/lib/systemd/system/cloud-init.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
[stack@undercloud ~]$ sudo systemctl is-enabled cloud-init
enabled

"cloud-config.service": {
            "name": "cloud-config.service",
            "source": "systemd",
            "state": "inactive",
            "status": "enabled"
        },
        "cloud-final.service": {
            "name": "cloud-final.service",
            "source": "systemd",
            "state": "inactive",
            "status": "enabled"
        },
        "cloud-init-local.service": {
            "name": "cloud-init-local.service",
            "source": "systemd",
            "state": "inactive",
            "status": "enabled"
        },
        "cloud-init.service": {
            "name": "cloud-init.service",
            "source": "systemd",
            "state": "inactive",
            "status": "enabled"
        },

"res": {
        "ansible_facts": {
            "cloud_init_data_facts": {
                "status": {}
            }
        },
        "attempts": 2,
        "changed": false,
        "cloud_init_data_facts": {
            "status": {}
        },
        "failed": true
    }

[Rabi Mishra (rabi)]

so cloud-init services run after NetworkManager-wait-online.service and that has failed. So cloud-init did not run.

-- Logs begin at Thu 2021-01-28 11:20:18 UTC, end at Thu 2021-01-28 12:49:18 UTC. --
Jan 28 11:20:23 undercloud systemd[1]: Starting Network Manager Wait Online...
Jan 28 11:20:53 undercloud systemd[1]: NetworkManager-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Jan 28 11:20:53 undercloud systemd[1]: NetworkManager-wait-online.service: Failed with result 'exit-code'.
Jan 28 11:20:53 undercloud systemd[1]: Failed to start Network Manager Wait Online.

[yatin (yatinkarel)]

Proposed Fix:- https://review.opendev.org/c/openstack/tripleo-heat-templates/+/772979

[yatin (yatinkarel)]

Also proposed fix:- https://review.opendev.org/c/openstack/tripleo-quickstart/+/773294.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "yatin <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/772979
Reason: as per discussions in comments.