[upgrade] storage volumes don't get created on upgrade

[Artem Panchenko]

After Fuel (master node) upgrade from 5.0.1 to 5.1 'dockerctl revert ${container_name}' command doesn't work and its execution leads to container crash:

[root@fuel-lab-cz5558 ~]# dockerctl revert rsync
Stopping rsync...
fuel-core-5.1-rsync
Stopping rsync...
fuel-core-5.1-rsync
fuel-core-5.1-rsync
c15743763d9fcee70247ee5ee23c2be53f0e21d8bb0d675666e0002744e00246
[debug] commands.go:1905 End of CmdRun(), Waiting for hijack to finish.
2014/10/17 11:20:18 Error: Cannot start container c15743763d9fcee70247ee5ee23c2be53f0e21d8bb0d675666e0002744e00246: Container fuel-storage-5.1-puppet not found. Impossible to
 mount its volumes

[root@fuel-lab-cz5558 ~]# dockerctl start rsync
Error: Cannot start container fuel-core-5.1-rsync: Container fuel-storage-5.1-puppet not found. Impossible to mount its volumes
2014/10/17 11:29:48 Error: failed to start one or more containers
...

[root@fuel-lab-cz5558 ~]# docker ps -a | grep fuel-storage
3911e3e7c987 storage/puppet:latest /bin/sh -c '/bin/ech 26 hours ago Exited (0) 26 hours ago fuel-storage-5.0.1-puppet
7c53d8ea7e25 storage/repo:latest /bin/sh -c '/bin/ech 26 hours ago Exited (0) 26 hours ago fuel-storage-5.0.1-repo

I guess it happens because we don't install new (5.1) storage containers during Fuel upgrade, so docker is unable to re-build other new containers.

[Matthew Mosesohn (raytrac3r)]

The problem here is storage containers don't get made. They are all defined in the config.py for fuel_upgrade, but not a single one gets made and there are no references in fuel_upgrade.log for them. This should affect 5.1.1 and 6.0 as well when upgrading from 5.1.

[Matthew Mosesohn (raytrac3r)]

http://paste.openstack.org/show/T48poRSLuVtD0a2W6XmK/

[Evgeniy L (rustyrobot)]

>> The problem here is storage containers don't get made. They are all defined in the config.py for fuel_upgrade, but not a single one gets made

It's not correct, the problem is, they have different names.

https://github.com/stackforge/fuel-web/blob/master/fuel_upgrade_system/fuel_upgrade/fuel_upgrade/config.py#L556-L622

My suggestion for solving the problem is

Change prefix to "fuel-core-${VERSION}-storage-"
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/docker/templates/dockerctl_config.erb#L41

Then, in upgrade system we will rename this volumes.
Here is a list of volume containers which we have

fuel-core-5.1-volume_repos
fuel-core-5.1-volume_logs
fuel-core-5.1-volume_ssh_keys
fuel-core-5.1-volume_dump
fuel-core-5.1-volume_fuel_configs
fuel-core-5.1-volume_puppet_manifests
fuel-core-5.1-volume_upgrade_directory

We'll change them to

fuel-core-5.1-storage-repo
fuel-core-5.1-storage-log
fuel-core-5.1-storage-puppet
# This volumes you don't create for some reasons
fuel-core-5.1-storage-fuel-configs
fuel-core-5.1-storage-ssh-keys
fuel-core-5.1-storage-dump
# This volume is required for upgrade only
fuel-core-5.1-storage-upgrade-directory

By the way in terms of docker these are volume containers, not "storages" and I would prefer to rename them to

fuel-core-5.1-volume-repo
fuel-core-5.1-volume-log
fuel-core-5.1-volume-puppet
fuel-core-5.1-volume-fuel-configs
fuel-core-5.1-volume-ssh-keys
fuel-core-5.1-volume-dump
fuel-core-5.1-volume-upgrade-directory

Also in dockerctl I can see, that some of the volumes are mounted directly into the containers, without volume containers, what is the point of making the same things in the different ways?

[Matthew Mosesohn (raytrac3r)]

fuel config, ssh keys, dump, and log are host volume mounts. There's no container here. I should add here that overlapping container mounts are simply impossible and can't be worked around in any way.

Renaming containers to fix a bug is not an appropriate solution and should be in a blueprint. All we're looking at here is trying to ensure the following containers are present after upgrade:
fuel-core-5.1-storage-repo
fuel-core-5.1-storage-log
fuel-core-5.1-storage-puppet

But we can remove repo because it is actually deprecated in 5.1 and it's now a host volume. See the only 2 storage container mounts are here:
https://github.com/stackforge/fuel-library/blob/stable/5.1/deployment/puppet/docker/templates/dockerctl_config.erb#L127-L129

Some of the others you mentioned exist by those names only in fuel_upgrade. There are host mounts here:
https://github.com/stackforge/fuel-library/blob/stable/5.1/deployment/puppet/docker/templates/dockerctl_config.erb#L87-L92

Correct me if I'm wrong, but what's missing in fuel_upgrade and what's breaking the upgrade process isn't the naming convention of these containers. It's because fuel_upgrade doesn't create the new storage containers (which obvious by the bug description). It could be done by the following steps:
1) Include dockerfiles for storage containers (missing in upgrade tarball)
2) Build these containers before starting 6.0 containers by running dockerctl build storage or:
source_path=$UPG_ROOT/docker/source/
release="6.0"
for storage_container in $source_path/storage*; do
  docker build -t storage/$(basename $storage_container)_$release $storage_container
done

Lastly to address your last question, the reason for direct volume mounts is because the limitations of LXC directory resharing, symlinking, and then sharing again to a dependent container (such as host:/var/www/nailgun -> repo:/repo -> nailgun:/var/www/nailgun) led to previous upgrade bugs. Direct mounts proved to be a better alternative than trying to manage symbolic links to work around LXC limitations. We currently are using no storage volume containers to actually be repositories of data. At the end of the day, these are just host volume mounts and that is all. Trying to force the other host volumes into containers will create more bugs. Maybe some of this will be fixed when we move to Docker 1.x with libcontainer, but that remains to be seen.

[Matthew Mosesohn (raytrac3r)]

Correction, logs go through a middle storage container. I misspoke in my first sentence in #5

[Matthew Mosesohn (raytrac3r)]

Actually, let's expand on this further. Serious user issues encountered in 5.1 such as full disk on master node ( https://bugs.launchpad.net/fuel/+bug/1383741 ) cannot be fixed if a user upgraded from 5.0.x to 5.1 because it's impossible to use dockerctl to repair an existing environment because of the changed names of storage containers.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/137051

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/137058

[Bogdan Dobrelya (bogdando)]

If we used direct mounts, what would have blocked upgrade-rollback from 6.0 to 5.1(.1) due to "fuel_upgrade doesn't create the new storage containers (which obvious by the bug description)", right?

[Matthew Mosesohn (raytrac3r)]

Evgeniy and I had a discussion on this and we came to the following approach. Rather than trying to synchronize storage container names and worry about its implications with upgrade, rollback, and redeploying individual containers, the best solution is to simply drop all the storage containers and move to direct host mounts. They don't add any value and only serve to confuse devels because there are some host mounts and then some storage containers, plus strange middle logic.

My patch is already on review and is being tested. We're still waiting on a fuel_upgrade patch which brings parity to new deployment vs fuel_upgrade.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/137146

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/137051
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=7fb96b7c56429256d66bcdbe442f205c1f68c9de
Submitter: Jenkins
Branch: master

commit 7fb96b7c56429256d66bcdbe442f205c1f68c9de
Author: Matthew Mosesohn <email address hidden>
Date: Tue Nov 25 16:05:53 2014 +0400

    Remove all storage containers and use host mounts

    Middle storage containers add complexity and additional
    logic to deployment, upgrades, and downgrades. Removing
    them allows host mounts to function better without
    workarounds with path hacks.

    Additionally, most host mounts operated without a middle
    container already. This brings the remaining two (log and
    puppet) into line with the others.

    Change-Id: I9db2dcc45210affafe62ebe9509a58829fc62ea6
    Partial-Bug: #1382527

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/137058
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=00233178a8197435755ad4b6a090430fa18df3e4
Submitter: Jenkins
Branch: stable/5.1

commit 00233178a8197435755ad4b6a090430fa18df3e4
Author: Matthew Mosesohn <email address hidden>
Date: Tue Nov 25 16:05:53 2014 +0400

    Remove all storage containers and use host mounts

    Middle storage containers add complexity and additional
    logic to deployment, upgrades, and downgrades. Removing
    them allows host mounts to function better without
    workarounds with path hacks.

    Additionally, most host mounts operated without a middle
    container already. This brings the remaining two (log and
    puppet) into line with the others.

    Change-Id: I9db2dcc45210affafe62ebe9509a58829fc62ea6
    Partial-Bug: #1382527

[Matthew Mosesohn (raytrac3r)]

This is partially resolved, but https://review.openstack.org/#/c/137146/ is still open

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/137146
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=b5171a5797b46303b3916abe946766a35d0a3f16
Submitter: Jenkins
Branch: master

commit b5171a5797b46303b3916abe946766a35d0a3f16
Author: Igor Kalnitsky <email address hidden>
Date: Tue Nov 25 17:31:50 2014 +0200

    Replace volume containers with direct mountings

    Fuel storage containers are a bit useless in our architecture and lead
    us to the problems when after upgrade we have storage containers with
    not expected names. In order to be consistent with dockerctl and prevent
    desync with dockerctl it was decided to remove them.

    Change-Id: Id3b614ce5f8cd5284bc387d1384d0c9928f84981
    Related-Bug: #1382527

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/5.1
Review: https://review.openstack.org/137380

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/137380
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=500e36d08a45dbb389bf2bd97673d9bff48ee84d
Submitter: Jenkins
Branch: stable/5.1

commit 500e36d08a45dbb389bf2bd97673d9bff48ee84d
Author: Igor Kalnitsky <email address hidden>
Date: Tue Nov 25 17:31:50 2014 +0200

    Replace volume containers with direct mountings

    Fuel storage containers are a bit useless in our architecture and lead
    us to the problems when after upgrade we have storage containers with
    not expected names. In order to be consistent with dockerctl and prevent
    desync with dockerctl it was decided to remove them.

    (cherry picked from commit b5171a5797b46303b3916abe946766a35d0a3f16)

    Change-Id: Id3b614ce5f8cd5284bc387d1384d0c9928f84981
    Related-Bug: #1382527

[Andrey Sledzinskiy (asledzinskiy)]

verified on
{

    "build_id": "2014-12-03_01-07-36",
    "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
    "build_number": "48",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d",
    "production": "docker",
    "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404",
    "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1.1",
    "release_versions": {
        "2014.1.3-5.1.1": {
            "VERSION": {
                "build_id": "2014-12-03_01-07-36",
                "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
                "build_number": "48",
                "api": "1.0",
                "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d",
                "production": "docker",
                "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404",
                "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "5.1.1",
                "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"
            }
        },
        "2014.1.1-5.1": {
            "VERSION": {
                "build_id": "2014-09-17_21-40-34",
                "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
                "build_number": "11",
                "api": "1.0",
                "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d",
                "production": "docker",
                "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd",
                "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "5.1",
                "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"
            }
        }
    },
    "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"

}

[Andrey Sledzinskiy (asledzinskiy)]

verified on {

    "build_id": "2014-12-17_03-38-09",
    "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4",
    "build_number": "50",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "dc621c96e30f0dd284998e975f3b800afe9fc082",
    "production": "docker",
    "fuelmain_sha": "fb89876812320464ebad60e7ca34b433d6b1d815",
    "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91",
    "feature_groups": [
        "mirantis"
    ],
    "release": "6.0",
    "release_versions": {
        "2014.1.3-5.1.1": {
            "VERSION": {
                "build_id": "2014-12-03_01-07-36",
                "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
                "build_number": "48",
                "api": "1.0",
                "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d",
                "production": "docker",
                "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404",
                "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "5.1.1",
                "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"
            }
        },
        "2014.2-6.0": {
            "VERSION": {
                "build_id": "2014-12-17_03-38-09",
                "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4",
                "build_number": "50",
                "api": "1.0",
                "nailgun_sha": "dc621c96e30f0dd284998e975f3b800afe9fc082",
                "production": "docker",
                "fuelmain_sha": "fb89876812320464ebad60e7ca34b433d6b1d815",
                "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "6.0",
                "fuellib_sha": "484f53f586c084cc38c88615b0594d9c48882c5a"
            }
        }
    },
    "fuellib_sha": "484f53f586c084cc38c88615b0594d9c48882c5a"

}