Cinder-volume service isn't started after reboot on Ubuntu

[Artem Panchenko]

After restart of all nodes in cluster cinder-volume services on compute/storage nodes do not start automatically, because of special override file:

root@node-1:~# cat /etc/init/cinder-volume.override
manual

Steps to reproduce:

1. Create new cluster: Ubuntu, HA, NeutronGre
2. Add 1 controller and 2 compute+cinder nodes
3. Deploy changes. Everything works fine after deployment, cluster passes health checks.
4. Shutdown all nodes in cluster (/sbin/shutdown -Ph now)
5. Start all nodes simultaneously.

Expected result:

- all nodes are booted correctly, cluster passes health checks

Actual:

- cinder-volume services are down

Here is an output of cinder service-list command on controller:

+------------------+--------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+--------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | node-2 | nova | enabled | up | 2015-02-13T09:42:52.000000 | None |
| cinder-volume | node-1 | nova | enabled | down | 2015-02-13T06:32:46.000000 | None |
| cinder-volume | node-3 | nova | enabled | down | 2015-02-13T06:32:54.000000 | None |
+------------------+--------+------+---------+-------+----------------------------+-----------------+

On compute/cinder cinder-volume services nodes are down after restart:

root@node-1:~# /etc/init.d/cinder-volume status
Rather than invoking init scripts through /etc/init.d, use the service(8)
utility, e.g. service cinder-volume status

Since the script you are attempting to invoke has been converted to an
Upstart job, you may also use the status(8) utility, e.g. status cinder-volume
cinder-volume stop/waiting

After 'cinder-volume' services restart they begin to work fine. If remove '/etc/init/cinder-volume.override' file cinder-volume also begin to start on boot. Here is the parts of puppet.log from cinder node(node-1):

http://paste.openstack.org/show/172884/

I guess cinder-volume.override was created by puppet during deployment, but wasn't removed after that.

Diagnostic snapshot is attached. Fuel version info: http://paste.openstack.org/show/172789/

[Bogdan Dobrelya (bogdando)]

The solution is to add post-deployment task to purge all override files Fuel created during the deployment

[Bogdan Dobrelya (bogdando)]

For 6.0 release the same job should be done by astute post-deploy hook

[Aleksandr Didenko (adidenko)]

There is no need to add hooks. Such override files are handled via tweaks::ubuntu_service_override resource which has strict dependency:

File[$file_name] -> Package <| name == $package_name |> -> Exec[$exec_name]

File[$file_name] - creates override file
Exec[$exec_name] - removes override file

But I see no dependency between File and Package in the snapshot logs for 'cinder-volume'. This is possible only if Package <| name == $package_name |> resource is not found in the puppet catalogue.

We have tweaks::ubuntu_service_override for 'cinder-volume' in openstack::compute class but not in openstack::cinder.

So here's the root cause.

node-1 is multirole, deployment steps were:

1) puppet apply for compute role, evaluate openstack::compute class which has tweaks::ubuntu_service_override { 'cinder-volume':} in it. But it does not have Package['cinder-volume']. So remove exec evaluated before file was created:

2015-02-13T06:22:53.939259+00:00 info: (/Stage[main]/Openstack::Compute/Tweaks::Ubuntu_service_override[cinder-volume]/Exec[remove_cinder-volume_override]) Evaluated in 0.07 seconds
2015-02-13T06:22:54.021568+00:00 info: (/Stage[main]/Openstack::Compute/Tweaks::Ubuntu_service_override[cinder-volume]/File[create_cinder-volume_override]) Evaluated in 0.01 seconds

2) puppet apply for cinder role, evaluate openstack::cinder class which does not have tweaks::ubuntu_service_override { 'cinder-volume':} in it. So it just installed a package and /etc/init/cinder-volume.override was already there since previous deployment step (compute role).

Solution: move cinder-volume tweaks::ubuntu_service_override into appropriate class (from openstack::compute to openstack::cinder). We can also add direct dependency into tweaks::ubuntu_service_override definition:

File[$file_name] -> Package <| name == $package_name |> -> Exec[$exec_name]
File[$file_name] -> Exec[$exec_name]

just to make sure we remove override file even if package is not found.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/155854

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/156257

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/155854
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=8994a8ad5e9767f1a7f84cf95b387e71bb39c6cf
Submitter: Jenkins
Branch: master

commit 8994a8ad5e9767f1a7f84cf95b387e71bb39c6cf
Author: Aleksandr Didenko <email address hidden>
Date: Fri Feb 13 20:24:17 2015 +0200

    Fix ubuntu_service_override ordering

    Fixes resource ordering in tweaks::ubuntu_service_override to make
    sure we do not leave any 'override' files.
    Replaces tabs with spaces in ubuntu_service_override.pp
    Added tweaks::ubuntu_service_override for cinder-volume in
    openstack::cinder class.

    Change-Id: I036cb65153029bb1a09be5eea2ee5bb4cbe97ae9
    Closes-bug: #1421595

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/156557

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/156257
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b8fbdd51230eb6cfcb2338e93af5b3afa3016643
Submitter: Jenkins
Branch: stable/6.0

commit b8fbdd51230eb6cfcb2338e93af5b3afa3016643
Author: Aleksandr Didenko <email address hidden>
Date: Fri Feb 13 20:24:17 2015 +0200

    Fix ubuntu_service_override ordering

    Fixes resource ordering in tweaks::ubuntu_service_override to make
    sure we do not leave any 'override' files.
    Replaces tabs with spaces in ubuntu_service_override.pp
    Added tweaks::ubuntu_service_override for cinder-volume in
    openstack::cinder class.

    Change-Id: I036cb65153029bb1a09be5eea2ee5bb4cbe97ae9
    Closes-bug: #1421595
    (cherry picked from commit 8994a8ad5e9767f1a7f84cf95b387e71bb39c6cf)

[Anastasia Palkina (apalkina)]

Verified on ISO #145

"build_id": "2015-02-25_22-54-44", "ostf_sha": "1a0b2c6618fac098473c2ed5a9af11d3a886a3bb", "build_number": "145", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-02-25_22-54-44", "ostf_sha": "1a0b2c6618fac098473c2ed5a9af11d3a886a3bb", "build_number": "145", "api": "1.0", "nailgun_sha": "54b0302665b7c02009b89f693170c26437276e76", "production": "docker", "python-fuelclient_sha": "5657dbf06fddb74adb61e9668eb579a1c57d8af8", "astute_sha": "6d6ad68e0cde286d74ac7d52e21da4fc8dcbe9ab", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "87223c15bb734bb668050850a6b43aa4291349b7", "fuellib_sha": "8384b8ca4db84794fb21e287202f05e31f78841c"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "54b0302665b7c02009b89f693170c26437276e76", "production": "docker", "python-fuelclient_sha": "5657dbf06fddb74adb61e9668eb579a1c57d8af8", "astute_sha": "6d6ad68e0cde286d74ac7d52e21da4fc8dcbe9ab", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "87223c15bb734bb668050850a6b43aa4291349b7", "fuellib_sha": "8384b8ca4db84794fb21e287202f05e31f78841c"

root@node-3:~# cinder service-list
+------------------+--------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+--------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | node-3 | nova | enabled | up | 2015-02-27T09:41:48.000000 | None |
| cinder-volume | node-4 | nova | enabled | up | 2015-02-27T09:41:54.000000 | None |
| cinder-volume | node-6 | nova | enabled | up | 2015-02-27T09:41:44.000000 | None |
+------------------+--------+------+---------+-------+----------------------------+-----------------+

[OpenStack Infra (hudson-openstack)]

Change abandoned by Aleksandr Didenko (<email address hidden>) on branch: stable/5.1
Review: https://review.openstack.org/156557

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/167628

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/167628
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=1734f63b4cee2ad1656a9dc0a62105487cb43845
Submitter: Jenkins
Branch: stable/5.1

commit 1734f63b4cee2ad1656a9dc0a62105487cb43845
Author: Aleksandr Didenko <email address hidden>
Date: Wed Mar 25 16:59:06 2015 +0200

    Fix ubuntu_service_override ordering

    Fixes resource ordering in tweaks::ubuntu_service_override to make
    sure we do not leave any 'override' files.
    Replaces tabs with spaces in ubuntu_service_override.pp
    Added tweaks::ubuntu_service_override for cinder-volume in
    openstack::cinder class.

    Change-Id: I036cb65153029bb1a09be5eea2ee5bb4cbe97ae9
    Closes-bug: #1421595

[JohnsonYi (yichengli)]

Hello Aleksandr,

I also faced the same problem, but for both nova and glance api service not started,
 /etc/init/nova-api.override
 /etc/init/glance-api.override
 /etc/init/glance-registry.override
after removed the above override files, these services was started after node reboot, it was auto-created and refreshed after deployment.

But I still have the following override files,
-rw-r--r-- 1 root root 6 Jul 2 16:09 cinder-backup.override
-rw-r--r-- 1 root root 29 Jun 8 17:33 cinder-volume.override
-rw-r--r-- 1 root root 6 Jul 2 16:08 corosync.override
-rw-r--r-- 1 root root 6 Jun 8 17:10 haproxy.override
-rw-r--r-- 1 root root 7 Jun 8 17:16 heat-engine.override
-rw-r--r-- 1 root root 6 Jul 2 16:09 murano_api.override
-rw-r--r-- 1 root root 6 Jul 2 16:09 murano_engine.override
-rw-r--r-- 1 root root 6 Jul 2 16:09 nova-spicehtml5proxy.override
-rw-r--r-- 1 root root 6 Jul 2 16:09 nova-spiceproxy.override
-rw-r--r-- 1 root root 6 Jul 2 16:09 sahara-api.override

Should I removed all of them?

[JohnsonYi (yichengli)]

My openstack environment has 3 controller node, 2 compute node, 3 ceph node with fuel 6.0. F.Y.I

[JohnsonYi (yichengli)]

The timestamp for nova-api.override is Jul 2, but I can't find any puppet logs during Jul 2.
-rw-r--r-- 1 root root 6 Jul 2 16:10 nova-api.override