Unexpected container restart inside container create roles

[Bjoern]

After running the add_host.yml playbook to add new compute nodes we noticed that all containers got restarted, especially the cinder_volumes. This was particular unexpected that we didn't expect and container change for the existing installation, certainly not a sudden reboot. As a result we lost access to all iscsi volumes and manually restored those.
Looking at the container_common and container_setup roles, the restart seemed to be triggered after applying the following changes to the container/hosts:

TASK: [container_setup | Create Required local monitoring directories]
  - /openstack/backup/<container name> changed
  - /openstack/log/<container name> changed

TASK: [container_setup | Basic Inner Container Setup ]
  - /etc/network/interfaces.d/management.cfg changed

My ask is to investigate under which circumstances the /etc/network/interfaces.d/management.cfg could have been rewritten, and if we can introduce a no-container-restart flag to be added to certain playbooks to rule any issue out, like add_host.yml for example.

[Kevin Carter (kevin-carter)]

I don't know that this is a change we need to make as the issue only pertains to Icehouse and Juno which are both EOL. In my opinion the fix to this stability issue is to upgrade to Kilo and beyond.

[Christopher H. Laco (claco)]

Was the add_host play run with --limit against only the new node? Are we saying that running that play then restarted all containers on all of the other hosts as well?

Do this restart every time always?

TASK: [container_setup | Basic Inner Container Setup ]
  - /etc/network/interfaces.d/management.cfg changed

What I'm fishing for here is, is there any indication why adding a host would cause updates to Basic Inner Container Setup to run.

[Bjoern (bjoern-t)]

Our current workaround is to use --limit, just for adding hosts.
But I'm more afraid of situations where people run playbooks just to rewrite config files.
The restart did only happen once, when we did restart the playbooks, at least in a test environment it did not restart the container.
Not knowing why files like /etc/network/interfaces.d/management.cfg got updated, they should never after the container got created, at least running the playbooks of the same release), make me believe we need a fail safe to suppress restarts when rerunning playbooks in a not 1st install scenario.

[Bjoern (bjoern-t)]

FYI, we did not use --limit which caused this issue. New instructions add the --limit

[Bjoern (bjoern-t)]

This is also an issue for Kilo see https://github.com/openstack/openstack-ansible/blob/kilo/playbooks/roles/lxc_container_create/tasks/container_create.yml#L91-L103 when we upgrade from Juno to Kilo. The nic config gets converted from the LXC config file to <nic>.ini and triggers a container restart