Patching env from 5.1 to 5.1.1 fails on Swift

[Claude Durocher]

{"build_id": "2014-12-03_01-07-36", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "48", "auth_required": true, "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis", "experimental"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-12-03_01-07-36", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "48", "api": "1.0", "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker", "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404", "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"}}, "2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-17_21-40-34", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"}

[Operating System]
    Ubuntu 12.04

[Reference Architecture]
    HA with Ceph
    1 Zabbix node
    3 controller nodes
    3 compute nodes
    3 Ceph nodes

[Network model ]
    Nova-network

[Related Projects installed]
    None

Steps to reproduce
------------------------------

0. deploy OpenStack with Fuel 5.1 (Make sure to use Ceph for Cinder and Glance.)
1. install 5.1.1 upgrade file on Fuel server
2. enable experimental option
3. in Fuel ui, start upgrade from 5.1 to 5.1.1
4. Zabbix server upgrades OK
5. Upgrade fails on 1st controller node with error :

(/Stage[main]/Swift::Storage::Container/Swift::Storage::Generic[container]/Service[swift-container-replicator]) Failed to call refresh: Could not restart Service[swift-container-replicator]
...

Expected result
--------------------------

Upgrade successful

Actual result
---------------------

Error log on Logs tab:
http://pastebin.com/Fq78EiRT

Workaround
----------------------
None found : retry upgrade fails the same way

[Vladimir Kuklin (vkuklin)]

Hi, Claude, unfortunately we cannot mark this bug as critical as openstack patching is still an experimental feature.

[Claude Durocher (claude-d)]

Vladimir, I understand it's an experimental feature. Do you think I would get better results with CentOS?

PS I've been able to reproduce the problem on a second similar setup with Ubuntu so it's reproducible.

[Bartłomiej Piotrowski (bpiotrowski)]

I'm unable to reproduce the issue.

[Claude Durocher (claude-d)]

Make sure to use Ceph for Cinder and Glance.

I think the problem is due to Glance using Ceph and having Swift not installed in this configuration.

[Aleksandr Didenko (adidenko)]

Could you please provide diagnostic snapshot? Without it we won't be able to investigate the issue.

[Bogdan Dobrelya (bogdando)]

Please provide a diagnostic logs snapshot, we should have it in order to investigate the issue

[Bogdan Dobrelya (bogdando)]

It would be nice if QA team will try to reproduce the bug as well

[Claude Durocher (claude-d)]

Sent a private message to bogdando with location of logs snapshot.

[Bogdan Dobrelya (bogdando)]

We need more reproducers for the issue.
The storage configuration should be as the following:
storage:
  iser: false
  images_ceph: true
  objects_ceph: false
  metadata:
    weight: 60
    label: Storage
  volumes_vmdk: false
  volumes_lvm: false
  volumes_ceph: true
  osd_pool_size: "2"
  pg_num: 1024
  ephemeral_ceph: false

[Claude Durocher (claude-d)]

What is the meaning of objects_ceph?

We use ceph for storing instances and images...
Le 2015-01-16 04:15, "Bogdan Dobrelya" <email address hidden> a écrit :

> We need more reproducers for the issue.
> The storage configuration should be as the following:
> storage:
> iser: false
> images_ceph: true
> objects_ceph: false
> metadata:
> weight: 60
> label: Storage
> volumes_vmdk: false
> volumes_lvm: false
> volumes_ceph: true
> osd_pool_size: "2"
> pg_num: 1024
> ephemeral_ceph: false
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1401989
>
> Title:
> Patching env from 5.1 to 5.1.1 fails on Swift
>
> Status in Fuel: OpenStack installer that works:
> Incomplete
>
> Bug description:
> {"build_id": "2014-12-03_01-07-36", "ostf_sha":
> "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "48",
> "auth_required": true, "api": "1.0", "nailgun_sha":
> "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker",
> "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404",
> "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce",
> "feature_groups": ["mirantis", "experimental"], "release": "5.1.1",
> "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id":
> "2014-12-03_01-07-36", "ostf_sha":
> "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "48",
> "api": "1.0", "nailgun_sha":
> "500e36d08a45dbb389bf2bd97673d9bff48ee84d", "production": "docker",
> "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404",
> "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce",
> "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha":
> "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"}}, "2014.1.1-5.1":
> {"VERSION": {"build_id": "2014-09-17_21-40-34", "ostf_sha":
> "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "11",
> "api": "1.0", "nailgun_sha":
> "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker",
> "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd",
> "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
> "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha":
> "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha":
> "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"}
>
> [Operating System]
> Ubuntu 12.04
>
> [Reference Architecture]
> HA with Ceph
> 1 Zabbix node
> 3 controller nodes
> 3 compute nodes
> 3 Ceph nodes
>
> [Network model ]
> Nova-network
>
> [Related Projects installed]
> None
>
> Steps to reproduce
> ------------------------------
>
> 0. deploy OpenStack with Fuel 5.1 (Make sure to use Ceph for Cinder and
> Glance.)
> 1. install 5.1.1 upgrade file on Fuel server
> 2. enable experimental option
> 3. in Fuel ui, start upgrade from 5.1 to 5.1.1
> 4. Zabbix server upgrades OK
> 5. Upgrade fails on 1st controller node with error :
>
>
> (/Stage[main]/Swift::Storage::Container/Swift::Storage::Generic[container]/Service[swift-container-replicator])
> Failed to call refresh: Could not restart
> Service[swift-container-replicator]...

[Bogdan Dobrelya (bogdando)]

Then Swift was chosen as an Object Storage, the objects_ceph parameter would be false.

[Stanislaw Bogatkin (sbogatkin)]

We don't support that type of patching anymore. Closed.