Temporary containers created in docker_config not updated on update/upgrade

Bug #1860571 reported by Brent Eagles
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Brent Eagles

Bug Description

Containers created in the docker_config section to perform various config tasks, such as running container_puppet_apply.sh to create sidecar container scripts in neutron are not run during update/upgrades because the previous container instance prevents it from being defined and run. This is pretty serious because upgrades/updates won't update the sidecar scripts so old container images will be used by the sidecar processes.

Log output:

    "failed_when_result": false,
    "start_containers_outputs.stdout_lines | default([]) | union(start_containers_outputs.stderr_lines | default([]))": [
        "$ podman image exists undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-metadata-agent-ovn:20200110.1",
        "b''",
        "$ podman image exists undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1",
        "$ podman ps -a --filter label=managed_by=tripleo-Compute --filter label=config_id=tripleo_step2 --format {{.Names}} {{.Labels.container_name}}",
        "b'create_haproxy_wrapper create_haproxy_wrapper\\nnova_compute_init_log nova_compute_init_log\\n'",
        "$ podman inspect --type container --format {{index .Config.Labels \"config_data\"}} create_haproxy_wrapper",
        "b'{\"command\": [\"/container_puppet_apply.sh\", \"4\", \"file\", \"include ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers\"], \"detach\": false, \"image\": \"undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-metadata-agent-ovn:20200110.1\", \"net\": \"host\", \"pid\": \"host\", \"start_order\": 1, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro\", \"/etc/puppet:/tmp/puppet-etc:ro\", \"/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro\", \"/run/openvswitch:/run/openvswitch:shared,z\", \"/var/lib/neutron:/var/lib/neutron:shared,z\"]}\\n'",
        "$ podman inspect --type container --format {{index .Config.Labels \"config_data\"}} nova_compute_init_log",
        "b'{\"command\": [\"/bin/bash\", \"-c\", \"chown -R nova:nova /var/log/nova\"], \"environment\": {\"TRIPLEO_DEPLOY_IDENTIFIER\": \"1579554251\"}, \"image\": \"undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1\", \"net\": \"none\", \"privileged\": false, \"user\": \"root\", \"volumes\": [\"/var/log/containers/nova:/var/log/nova:z\"]}\\n'",
        "$ podman ps -a --filter label=managed_by=tripleo-Compute --format {{.Names}} {{.Labels.container_name}}",
        "b'create_haproxy_wrapper create_haproxy_wrapper\\nnova_statedir_owner nova_statedir_owner\\nnova_compute_init_log nova_compute_init_log\\nnova_wait_for_compute_service nova_wait_for_compute_service\\nnova_compute nova_compute\\novn_metadata_agent ovn_metadata_agent\\novn_controller ovn_controller\\nsetup_ovs_manager setup_ovs_manager\\nnova_migration_target nova_migration_target\\nlogrotate_crond logrotate_crond\\nconfigure_cms_options configure_cms_options\\niscsid iscsid\\nnova_libvirt nova_libvirt\\nnova_virtlogd nova_virtlogd\\n'",
        "Running container: nova_compute_init_log",
        "Skipping existing container: nova_compute_init_log",
        "Running container: create_haproxy_wrapper",
        "Skipping existing container: create_haproxy_wrapper"
    ]
}

Removing the container manually on the affected nodes allows this to run.

Revision history for this message
Brent Eagles (beagles) wrote :

Forgot to note this is being seen on stein but I suspect other branches are similarily affected.

Changed in tripleo:
milestone: none → ussuri-2
importance: Undecided → Critical
status: New → Triaged
tags: added: train-backport-potential
tags: added: stein-backport-potential
Revision history for this message
Brent Eagles (beagles) wrote :
tags: added: queens-backport-potential
tags: added: alert
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/703855

Changed in tripleo:
assignee: nobody → Brent Eagles (beagles)
status: Triaged → In Progress
tags: added: idempotency
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/704402

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/703855
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=714e1b5d3120b1babfec01c15bf508cc9241ceb2
Submitter: Zuul
Branch: master

commit 714e1b5d3120b1babfec01c15bf508cc9241ceb2
Author: Brent Eagles <email address hidden>
Date: Wed Jan 22 15:16:12 2020 -0330

    Add DeployIdentifier to extra config containers

    Certain config containers might need to be replaced and re-run
    regardless of whether configuration changes on update and upgrade.
    Adding the DeployIdentifier to the env will ensure that they are.

    Change-Id: I150212ebac3fed471ffb4e7ed7b6eb6c7af3fad9
    Closes-Bug: #1860571

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/707393

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Brent Eagles (<email address hidden>) on branch: master
Review: https://review.opendev.org/707393
Reason: mistakenly proposed to master - afaict this isn't a problem here because of systemd wrappers.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/707395

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (stable/train)

Change abandoned by Brent Eagles (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/707395
Reason: rendered unnecessary by https://review.opendev.org/#/c/705642/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/704402
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=83d4e84a47e951e2c13e669c9cf99aad588c0007
Submitter: Zuul
Branch: stable/train

commit 83d4e84a47e951e2c13e669c9cf99aad588c0007
Author: Brent Eagles <email address hidden>
Date: Wed Jan 22 15:16:12 2020 -0330

    Add DeployIdentifier to extra config containers

    Certain config containers might need to be replaced and re-run
    regardless of whether configuration changes on update and upgrade.
    Adding the DeployIdentifier to the env will ensure that they are.

    Conflicts:
      deployment/mistral/mistral-api-container-puppet.yaml

    Note: this cherry-pick is missing changes to
    ovn-metadata-container-puppet.yaml that were made in the original
    patch because the systemd-tripleo-wrappers was backported, making this
    particular change unnecessary.

    Change-Id: I150212ebac3fed471ffb4e7ed7b6eb6c7af3fad9
    Closes-Bug: #1860571
    (cherry picked from commit 714e1b5d3120b1babfec01c15bf508cc9241ceb2)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 12.1.0

This issue was fixed in the openstack/tripleo-heat-templates 12.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.4.0

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.