periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-wallaby is failing standalone upgrade - "Failed containers: nova_api_map_cell0"

[Ronelle Landy]

periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-wallaby is failing to upgrade the standalone deploy in step_3 with the following error:

2022-01-20 20:58:34 | + CMD='/usr/bin/bootstrap_host_exec nova_api /nova_api_map_cell0.sh'
2022-01-20 20:58:34 | + ARGS=
2022-01-20 20:58:34 | + [[ ! -n '' ]]
2022-01-20 20:58:34 | + . kolla_extend_start
2022-01-20 20:58:34 | + echo 'Running command: '\''/usr/bin/bootstrap_host_exec nova_api /nova_api_map_cell0.sh'\'''
2022-01-20 20:58:34 | + exec /usr/bin/bootstrap_host_exec nova_api /nova_api_map_cell0.sh
2022-01-20 20:58:34 | 2022-01-20 20:58:34.149193 | fa163e65-0b12-31ea-dcc6-000000001cd2 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 | standalone | error={"changed": false, "msg": "Failed containers: nova_api_map_cell0"}
2022-01-20 20:58:34 | 2022-01-20 20:58:34.150791 | fa163e65-0b12-31ea-dcc6-000000001cd2 | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 | standalone | 0:15:40.818489 | 38.81s

The full upgrade log is below:

https://logserver.rdoproject.org/48/38348/1/check/periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-wallaby/5f5a606/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz

The test has been failing since: 01/18 and is blocking the promotion of the tripleo component for wallaby and therefore rhos-17:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-wallaby

[Ronelle Landy (rlandy)]

nova_api_map_cell0 log:

https://logserver.rdoproject.org/48/38348/1/check/periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-wallaby/5f5a606/logs/undercloud/var/log/extra/podman/containers/nova_api_map_cell0/stdout.log.txt.gz

+ exec /usr/bin/bootstrap_host_exec nova_api /nova_api_map_cell0.sh
(cellv2) Updating cell0 cell_v2 cell 00000000-0000-0000-0000-000000000000
--transport-url not provided in the command line, using the value [DEFAULT]/transport_url from the configuration file
The specified transport_url and/or database_connection combination already exists for another cell with uuid 2b80d767-04ec-4520-a83b-394fca03ad58.

Compare with passing log:

https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-wallaby/246bb1d/logs/undercloud/var/log/extra/podman/containers/nova_api_map_cell0/stdout.log.txt.gz

[Ronelle Landy (rlandy)]

Possible impact: https://github.com/openstack/tripleo-heat-templates/commit/81c4c23f70fc4a917afce35e3588eee8081cf1e2

[Ronelle Landy (rlandy)]

Proposed a revert and marked it code-review -1 - dvd is working on another fix.

https://review.opendev.org/c/openstack/tripleo-heat-templates/+/825468

We can choose the right approach tomorrow

[Bogdan Dobrelya (bogdando)]

Could you please add periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-wallaby to check/gate for W? that would improve upgrades regression testing coverage

[Marios Andreou (marios-b)]

@Bogdan (re comment #4) - as discussed on irc last week - it was decided that the standalone-upgrade job and scen0 multinode upgrade job were not giving us good 'value' so we removed them.

Personally I disagreed with this but a decision was made. We kept undercloud-upgrade and minor update in the check/gate from wallaby onwards (we kept the ussuri/victoria ones that were already there).

As some mitigation we added these jobs to the periodic promotion lines (so this bug is about standalone upgrade job in the tripleo component promotion line).

I believe the upgrades team is working on a standalone-ffu - we can add that one into master/wallaby gate once it is ready.

[Marios Andreou (marios-b)]

Are we waiting for something further here? The revert merged @ https://review.opendev.org/c/openstack/tripleo-heat-templates/+/825468/2#message-92bc639a22f69427c4014292939d29dec21f381d

It looks to me the job is green - at least the last 3 runs on openstack-component-tripleo [1] are green

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-tripleo-wallaby