[upgrade] Rsync docker container after Fuel upgrade uses old puppet manifests while generating xinetd configuration

[Artem Panchenko]

We have a bug in Fuel 5.0.x https://bugs.launchpad.net/fuel/+bug/1322577 which is fixed in 5.1. But after Fuel upgrade from 5.0.1 to 5.1 the issue isn't gone because new 'rsync' container still has old configuration for xinetd:

[root@fuel-lab-cz5558 ~]# dockerctl shell rsync cat /etc/xinetd.d/rsync
service rsync
{
        disable = no
        socket_type = stream
        wait = no
        user = root
        server = /usr/bin/rsync
        server_args = --daemon
        log_on_failure += USERID
}

This happens becuase the folder with puppet manifests inside 'rsync' container is mounted from host system and '/etc/puppet/modules' directory contains old (5.0.1 in my case) files:

[root@fuel-lab-cz5558 ~]# dockerctl list -l | grep rsync
rsync fuel/rsync_5.1 Running fuel-core-5.1-rsync

[root@fuel-lab-cz5558 ~]# dockerctl shell rsync grep puppet /proc/mounts
/dev/mapper/os-root /etc/puppet ext4 ro,relatime,barrier=1,stripe=128,data=ordered 0 0

[root@fuel-lab-cz5558 ~]# ll /etc/puppet/
total 28
drwxr-xr-x 4 root root 4096 вер 17 18:29 2014.1.1-5.0.2
drwxr-xr-x 4 root root 4096 вер 17 18:28 2014.1.1-5.1
-rw-r--r--. 1 root root 4133 січ 6 2014 auth.conf
-rw-r--r--. 1 root root 0 жов 16 08:41 hiera.yaml
drwxr-xr-x. 2 root root 4096 жов 16 08:41 manifests
drwxr-xr-x. 62 root root 4096 жов 16 08:41 modules
-rw-r--r--. 1 root root 853 січ 6 2014 puppet.conf

This issue can be workarounded in the following way:

[root@fuel-lab-cz5558 ~]# dockerctl shell rsync sed -e 's/puppet\/modules/puppet\/2014.1.1-5.1\/modules/' -e 's/apply/apply --modulepath \/etc\/puppet\/2014.1.1-5.1\/modules/' -i /usr/local/bin/start.sh
[root@fuel-lab-cz5558 ~]# dockerctl restart rsync

[Matthew Mosesohn (raytrac3r)]

This only affects 5.1 because 6.0 uses host manifests directly and no longer relies on the puppet container.
This should be fixed in the fuel-upgrade scripts because all containers read /etc/puppet/modules and /etc/puppet/manifests directly. It's possible there was a bug in the upgrade here and the symlink wasn't set up right.

[Bogdan Dobrelya (bogdando)]

According to #1, This should be fixed in the fuel-upgrade scripts

[Evgeniy L (rustyrobot)]

>> This should be fixed in the fuel-upgrade scripts because all containers read /etc/puppet/modules and /etc/puppet/manifests directly. It's possible there was a bug in the upgrade here and the symlink wasn't set up right.

We should not rewrite `/etc/puppet/modules` and `/etc/puppet/manifests` with some symlinks, because this manifests are used during the deployment of previous versions of environments.

What we can do is to make symlink to /etc/puppet/latest from the latest version of manifests, and fix start.sh in the container to run manifests and modules from this directory.

[Evgeniy L (rustyrobot)]

The right way to solve the problem is not to mix openstack manifests and manifests for fuel master node.
We should keep separate directories for them, it make sense because the release cycle for openstack release and master can be different.
And we can have problems if we don't make this separation.

[Evgeniy L (rustyrobot)]

Another way to solve the problem is to fix Dockerfile for rsync [1], not to copy manifests and modules in /etc/puppet, use another directory and fix start.sh to run manifests and modules from this new directory, in this case manifests/modules will not be overridden by manifests/modules from the host system. And container will always have original puppet scripts without this "symlink magic" which we will have to handle somehow in case of rollbacks.

[1] https://github.com/stackforge/fuel-main/blob/master/docker/rsync/Dockerfile

[Matthew Mosesohn (raytrac3r)]

/etc/puppet/modules is already a symlink. Why do you want to make this more complicated? It's not even clear how the /etc/puppet directory should look like before, during, and after host upgrade.

We can move to host puppet manifests for 5.1.1 if it can be passed as an exception and if it helps, but proposing /etc/puppet/modules-HOST_ONLY directory doesn't seem to gain any benefits. All it does is add extra lines of upgrade logic outside of fuel_upgrade which already manages this task.

[Evgeniy L (rustyrobot)]

http://irclog.perlgeek.de/fuel-dev/2014-11-18#i_9679058

[Ihor Kalnytskyi (ikalnytskyi)]

+1 to Evgeniy. The container MUST be self-sufficient as much as possible. That's from docker best practices and it makes sence, because "less external factors" means "less points of failures".

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/135295

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/135947

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/135947
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=34dd49e31791da777faf088d313a23edd44124b3
Submitter: Jenkins
Branch: stable/5.1

commit 34dd49e31791da777faf088d313a23edd44124b3
Author: Matthew Mosesohn <email address hidden>
Date: Tue Nov 18 17:13:31 2014 +0400

    Create dedicated puppet volume for rsync container

    New host volume /etc/puppet maps to rsync container /puppet,
    freeing up /etc/puppet:/etc/puppet mapping, which may be enabled
    for some desired testing function.
    This also has a desired benefit to use bundled manifests when
    deploying rsync container, but still serve arbitrary host puppet
    contents via rsync.

    Change-Id: I6e7f29679a7d475a6c49dd6e3d498c14c943408f
    Closes-Bug: #1382531

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/135295
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=bfe13c7aa589b57cb584e85c08ec39ad0b39ec7b
Submitter: Jenkins
Branch: master

commit bfe13c7aa589b57cb584e85c08ec39ad0b39ec7b
Author: Matthew Mosesohn <email address hidden>
Date: Tue Nov 18 17:13:31 2014 +0400

    Create dedicated puppet volume for rsync container

    New host volume /etc/puppet maps to rsync container /puppet,
    freeing up /etc/puppet:/etc/puppet mapping, which may be enabled
    for some desired testing function.
    This also has a desired benefit to use bundled manifests when
    deploying rsync container, but still serve arbitrary host puppet
    contents via rsync.

    Change-Id: I6e7f29679a7d475a6c49dd6e3d498c14c943408f
    Closes-Bug: #1382531

[Matthew Mosesohn (raytrac3r)]

This commit was reverted and now it is blocked on either of these two bugs for a solution:
https://bugs.launchpad.net/fuel/+bug/1395047
https://bugs.launchpad.net/fuel/+bug/1395008

1395047 will provide a complete solution, but 1395008 is a partial workaround only.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/136725

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/5.1
Review: https://review.openstack.org/136731

[Bogdan Dobrelya (bogdando)]

related https://bugs.launchpad.net/fuel/+bug/1395658

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/136725
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=e3c385db9f5ae3b73117dc47238659a4d3bfac0b
Submitter: Jenkins
Branch: master

commit e3c385db9f5ae3b73117dc47238659a4d3bfac0b
Author: Matthew Mosesohn <email address hidden>
Date: Mon Nov 24 12:58:28 2014 +0400

    Remove puppet run from static containers

    Additional puppet runs have no impact on deployment
    of rsync and rsyslog containers because they do not
    change their configuration based on how a user deploys
    their environment.

    Change-Id: Ic5aac99620a8b6e24c5c5c2be9d246186df08a6c
    Closes-Bug: #1395658
    Related-Bug: #1382531

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/136731
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=ace6f48b84a2bf60898de7188688e2ee0acf259e
Submitter: Jenkins
Branch: stable/5.1

commit ace6f48b84a2bf60898de7188688e2ee0acf259e
Author: Matthew Mosesohn <email address hidden>
Date: Mon Nov 24 12:58:28 2014 +0400

    Remove puppet run from static containers

    Additional puppet runs have no impact on deployment
    of rsync and rsyslog containers because they do not
    change their configuration based on how a user deploys
    their environment.

    Change-Id: Ic5aac99620a8b6e24c5c5c2be9d246186df08a6c
    Closes-Bug: #1395658
    Related-Bug: #1382531