some containers are not idempotent and is restarted every deployment

Bug #1871734 reported by Alex Schultz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Incomplete
High
Unassigned

Bug Description

When rerunning the standalone, the memcache container is removed and recreated every deployment. This seems like it might actually be related to the way we define the command for the container. There are additional containers like the swift, nova, cinder_volume and horizon containers that also are recreated/restarted every deployment.

[root@standalone ~]# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a5bf3341c347 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-nova-compute:current-tripleo kolla_start 23 minutes ago Up 23 minutes ago nova_compute
6769098ffb3b standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-swift-proxy-server:current-tripleo kolla_start 26 minutes ago Up 26 minutes ago swift_proxy
e1ebca01e3c3 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-swift-object:current-tripleo kolla_start 29 minutes ago Up 29 minutes ago swift_rsync
794abed57f71 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-swift-object:current-tripleo kolla_start 29 minutes ago Up 29 minutes ago swift_object_updater
9973c227cacc standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-swift-object:current-tripleo kolla_start 30 minutes ago Up 30 minutes ago swift_object_server
030e0b826118 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-swift-proxy-server:current-tripleo kolla_start 30 minutes ago Up 30 minutes ago swift_object_expirer
ff737da012f3 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-swift-container:current-tripleo kolla_start 30 minutes ago Up 30 minutes ago swift_container_updater
781722e4e60c standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-swift-container:current-tripleo kolla_start 30 minutes ago Up 30 minutes ago swift_container_server
9eec9278aa0f standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-swift-account:current-tripleo kolla_start 30 minutes ago Up 30 minutes ago swift_account_server
969503ae3ac8 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-swift-account:current-tripleo kolla_start 32 minutes ago Up 32 minutes ago swift_account_reaper
795a945c8cc6 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-nova-compute:current-tripleo kolla_start 33 minutes ago Up 33 minutes ago nova_migration_target
1e7e3103b62b standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-cinder-volume:current-tripleo kolla_start 34 minutes ago Up 33 minutes ago cinder_volume
f1835e132f62 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-iscsid:current-tripleo kolla_start 39 minutes ago Up 39 minutes ago iscsid
73306e52cf02 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-nova-libvirt:current-tripleo kolla_start 40 minutes ago Up 39 minutes ago nova_libvirt
844bdee678fb standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-nova-libvirt:current-tripleo kolla_start 41 minutes ago Up 40 minutes ago nova_virtlogd
d7a251e458b8 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-horizon:current-tripleo kolla_start 41 minutes ago Up 41 minutes ago horizon
79e00e4ecf50 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-memcached:current-tripleo kolla_start About an hour ago Up 47 minutes ago memcached
6e3a3008d5d8 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-rabbitmq:current-tripleo kolla_start 4 hours ago Up 4 hours ago rabbitmq
1cb12e0b10a3 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-nova-api:current-tripleo kolla_start 4 hours ago Up 4 hours ago nova_api_cron
9a228fb31099 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-ovn-northd:current-tripleo kolla_start 4 hours ago Up 4 hours ago ovn_northd
eb5409318963 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-nova-api:current-tripleo kolla_start 4 hours ago Up 4 hours ago nova_metadata
e57d584f973a standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-nova-api:current-tripleo kolla_start 4 hours ago Up 4 hours ago nova_api
e075689d0131 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-glance-api:current-tripleo kolla_start 4 hours ago Up 4 hours ago glance_api
9c35e19a8e2b standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-placement-api:current-tripleo kolla_start 5 hours ago Up 5 hours ago placement_api
41013394211b standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-neutron-metadata-agent-ovn:current-tripleo kolla_start 5 hours ago Up 5 hours ago ovn_metadata_agent
b73815cf1654 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-ovn-controller:current-tripleo kolla_start 5 hours ago Up 5 hours ago ovn_controller
0956be0c29fc standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-ovn-sb-db-server:current-tripleo kolla_start 5 hours ago Up 5 hours ago ovn_south_db_server
c85dda8077e3 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-ovn-nb-db-server:current-tripleo kolla_start 5 hours ago Up 5 hours ago ovn_north_db_server
e95b9c7b76f4 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-nova-novncproxy:current-tripleo kolla_start 5 hours ago Up 5 hours ago nova_vnc_proxy
8639be2d2246 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-nova-scheduler:current-tripleo kolla_start 5 hours ago Up 5 hours ago nova_scheduler
9d39d7e79494 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-nova-conductor:current-tripleo kolla_start 5 hours ago Up 5 hours ago nova_conductor
c9dccc26c2b6 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-neutron-server:current-tripleo kolla_start 5 hours ago Up 5 hours ago neutron_api
9d4fae818118 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-cron:current-tripleo kolla_start 5 hours ago Up 5 hours ago logrotate_crond
41377328e6aa standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-cinder-scheduler:current-tripleo kolla_start 5 hours ago Up 5 hours ago cinder_scheduler
119bed4b6e08 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-cinder-api:current-tripleo kolla_start 5 hours ago Up 5 hours ago cinder_api_cron
36696fb18540 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-cinder-api:current-tripleo kolla_start 5 hours ago Up 5 hours ago cinder_api
c4be1f63d648 standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-keystone:current-tripleo kolla_start 5 hours ago Up 5 hours ago keystone
e81c8e946f0c standalone.ctlplane.localdomain:8787/tripleomaster/centos-binary-mariadb:current-tripleo kolla_start 5 hours ago Up 5 hours ago mysql

Revision history for this message
Michele Baldessari (michele) wrote :

FWIW I brought up the swift ones some time ago: https://bugzilla.redhat.com/show_bug.cgi?id=1746113

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Related change https://review.opendev.org/#/c/713415/ for the swift containers, needs testing

tags: added: idempotency
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/718565
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=71e9b806b4765ebe5474c26621ef8ddcec3f5039
Submitter: Zuul
Branch: master

commit 71e9b806b4765ebe5474c26621ef8ddcec3f5039
Author: Alex Schultz <email address hidden>
Date: Wed Apr 8 16:08:30 2020 -0600

    Use kolla tools for memcached configuration

    This change updates the memcached service to leverage the kolla
    configuration to configure and launch the memcached service. This is the
    same pattern we use for the other services.

    Related-Bug: #1871734
    Change-Id: Ib0bfc3e9009357f0aed28c0a10b927c5ca2e53a7

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/719056

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/719056
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=cf3c03ebde7fa09e84e0754cde90fb453239b6c1
Submitter: Zuul
Branch: stable/train

commit cf3c03ebde7fa09e84e0754cde90fb453239b6c1
Author: Alex Schultz <email address hidden>
Date: Wed Apr 8 16:08:30 2020 -0600

    Use kolla tools for memcached configuration

    This change updates the memcached service to leverage the kolla
    configuration to configure and launch the memcached service. This is the
    same pattern we use for the other services.

    Related-Bug: #1871734
    Change-Id: Ib0bfc3e9009357f0aed28c0a10b927c5ca2e53a7
    (cherry picked from commit 71e9b806b4765ebe5474c26621ef8ddcec3f5039)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/719671

tags: added: queens-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.opendev.org/719671
Reason: clearing gate, need to land https://review.opendev.org/#/c/720132/

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :
summary: - memcached is not idempotent and is restarted every deployment
+ some containers are not idempotent and is restarted every deployment
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/719671
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f8eb0a8fa04d96660a6f03e4187a1f246f0694f0
Submitter: Zuul
Branch: master

commit f8eb0a8fa04d96660a6f03e4187a1f246f0694f0
Author: Alex Schultz <email address hidden>
Date: Mon Apr 13 13:12:27 2020 -0600

    Exclude /etc/puppet from config generation

    When we run the container puppet process, we nuke /etc/puppet in the
    container and sync it in from the system. Since this is not part of the
    actual configuration, we're getting changes to the hieradata which is
    causing unnecessary container restarts.

    Change-Id: I9e770a2e3342fbd9617ae4a59c7e617f84e9233d
    Related-Bug: #1871734

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/722786

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/722786
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c2e22af3e9267dad55b01a214d560afa779ebc4c
Submitter: Zuul
Branch: stable/train

commit c2e22af3e9267dad55b01a214d560afa779ebc4c
Author: Alex Schultz <email address hidden>
Date: Mon Apr 13 13:12:27 2020 -0600

    Exclude /etc/puppet from config generation

    When we run the container puppet process, we nuke /etc/puppet in the
    container and sync it in from the system. Since this is not part of the
    actual configuration, we're getting changes to the hieradata which is
    causing unnecessary container restarts.

    Change-Id: I9e770a2e3342fbd9617ae4a59c7e617f84e9233d
    Related-Bug: #1871734
    (cherry picked from commit f8eb0a8fa04d96660a6f03e4187a1f246f0694f0)

wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc1 → ussuri-rc3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Changed in tripleo:
milestone: victoria-1 → victoria-3
Changed in tripleo:
milestone: victoria-3 → wallaby-1
Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Changed in tripleo:
milestone: wallaby-3 → wallaby-rc1
Changed in tripleo:
milestone: wallaby-rc1 → xena-1
Revision history for this message
Marios Andreou (marios-b) wrote :

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

Changed in tripleo:
milestone: xena-1 → none
status: Triaged → Incomplete
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.