[queens] update fails on nova_api_ensure_cell0_database_url

[Sofer Athlan-Guyot]

Hi,

originally reported there https://bugzilla.redhat.com/show_bug.cgi?id=1813642

Update fails with that error:

"Running container: nova_api_ensure_cell0_database_url",
"$ docker ps -a --filter label=container_name=nova_api_ensure_cell0_database_url --filter label=config_id=tripleo_step3 --format {{.Names}}",
"Did not find container with \"['docker', 'ps', '-a', '--filter', 'label=container_name=nova_api_ensure_cell0_database_url', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\"
 - retrying without config_id",
 "$ docker ps -a --filter label=container_name=nova_api_ensure_cell0_database_url --format {{.Names}}",
"nova_api_ensure_cell0_database_url",
"$ docker run --name nova_api_ensure_cell0_database_url --label config_id=tripleo_step3 --label container_name=nova_api_ensure_cell0_database_url --label managed_by=paunch --label config_data={\"start_order\": 3, \"image\": \"registry.access.redhat.com/rhosp13/openstack-nova-api:13.0-114\", \"environment\": [\"TRIPLEO_CONFIG_HASH=249fea517d1efae4911e08838277b246\"], \"command\": \"/usr/bin/bootstrap_host_exec nova_api /nova_api_ensure_cell0_database_url.sh\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/log/containers/httpd/nova-api:/var/log/httpd\", \"/var/lib/config-data/nova/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro\", \"/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/lib/config-data/puppet-generated/nova/:/var/lib/kolla/config_files/src:ro\", \"/var/lib/docker-config-scripts/nova_api_ensure_cell0_database_url.sh:/nova_api_ensure_cell0_database_url.sh:ro\"], \"net\": \"host\", \"detach\": false} --env=TRIPLEO_CONFIG_HASH=249fea517d1efae4911e08838277b246 --net=host --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/var/log/containers/nova:/var/log/nova --volume=/var/log/containers/httpd/nova-api:/var/log/httpd --volume=/var/lib/config-data/nova/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro --volume=/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro --volume=/var/log/containers/nova:/var/log/nova --volume=/var/lib/config-data/puppet-generated/nova/:/var/lib/kolla/config_files/src:ro --volume=/var/lib/docker-config-scripts/nova_api_ensure_cell0_database_url.sh:/nova_api_ensure_cell0_database_url.sh:ro --cpuset-cpus=0,1,2,3 registry.access.redhat.com/rhosp13/openstack-nova-api:13.0-114 /usr/bin/bootstrap_host_exec nova_api /nova_api_ensure_cell0_database_url.sh",
"/usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/nova_api_ensure_cell0_database_url\" is already in use by container 580df6343e9af347fdf157e1f00fe37e0155c02ed368263ce8fc08466fcf7824.
You have to remove (or rename) that container to be able to reuse that name..",
"See '/usr/bin/docker-current run --help'.",
"Error running ['docker', 'run', '--name', u'nova_api_ensure_cell0_database_url', '--label', 'config_id=tripleo_step3', '--label', 'container_name=nova_api_ensure_cell0_database_url', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 3, \"image\": \"registry.access.redhat.com/rhosp13/openstack-nova-api:13.0-114\", \"environment\": [\"TRIPLEO_CONFIG_HASH=249fea517d1efae4911e08838277b246\"], \"command\": \"/usr/bin/bootstrap_host_exec nova_api /nova_api_ensure_cell0_database_url.sh\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/log/containers/httpd/nova-api:/var/log/httpd\", \"/var/lib/config-data/nova/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro\", \"/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro\", \"/var/log/containers/nova:/var/log/nova\", \"/var/lib/config-data/puppet-generated/nova/:/var/lib/kolla/config_files/src:ro\", \"/var/lib/docker-config-scripts/nova_api_ensure_cell0_database_url.sh:/nova_api_ensure_cell0_database_url.sh:ro\"], \"net\": \"host\", \"detach\": false}', '--env=TRIPLEO_CONFIG_HASH=249fea517d1efae4911e08838277b246', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/log/containers/nova:/var/log/nova', '--volume=/var/log/containers/httpd/nova-api:/var/log/httpd', '--volume=/var/lib/config-data/nova/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro', '--volume=/var/lib/config-data/nova/etc/nova/:/etc/nova/:ro', '--volume=/var/log/containers/nova:/var/log/nova', '--volume=/var/lib/config-data/puppet-generated/nova/:/var/lib/kolla/config_files/src:ro', '--volume=/var/lib/docker-config-scripts/nova_api_ensure_cell0_database_url.sh:/nova_api_ensure_cell0_database_url.sh:ro', '--cpuset-cpus=0,1,2,3', 'registry.access.redhat.com/rhosp13/openstack-nova-api:13.0-114', '/usr/bin/bootstrap_host_exec', 'nova_api', '/nova_api_ensure_cell0_database_url.sh']. [125]",
"stderr: /usr/bin/docker-current: Error response from daemon:
Conflict. The container name \"/nova_api_ensure_cell0_database_url\" is already in use by container 580df6343e9af347fdf157e1f00fe37e0155c02ed368263ce8fc08466fcf7824. You have to remove (or rename) that container to be able to reuse that name.."

The crux of it seems that he move of nova_api_ensure_cell0_database_url from start_order 5 to start_order 3 causes paunch to not delete it when it reaches the start_order 3 as in the live container, start_order is still 5.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/720090

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Note that is is queen only, the title has not been set properly.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/720090
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2fdda9b31884385d88bfa433535234c305bb01de
Submitter: Zuul
Branch: stable/queens

commit 2fdda9b31884385d88bfa433535234c305bb01de
Author: Sofer Athlan-Guyot <email address hidden>
Date: Tue Apr 14 23:55:54 2020 +0200

    Make sure nova_api_ensure_cell0_database_url is deleted.

    After moving the nova_api_ensure_cell0_database_url from tripleo_step5
    to tripleo_step3 we encounter an error where that container could not
    be created during update because it was not being deleted.

    The reasoning here is that paunch may look for container to destroy
    based on container live information while it creates them based of
    file configuration. Then moving container at an earlier step is
    not possible.

    Here we work around the issue by making sure that this container is
    destroyed before reaching the common deploy step task (where paunch is
    triggered) in the update tasks (which happen before).

    We look for the current live information of the container and if it is
    still in step5 then we delete it.

    To prevent such thing to happen again we would need to make sure that
    those container are ephemeral and get destroyed after being used.

    Change-Id: Iaa00c0971199b335eca415aa69af9132a3ddf225
    Closes-Bug: #1872828

[Marios Andreou (marios-b)]

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates queens-eol release.