lvm is reporting problems, not sending heartbeat. Service will appear "down"

[Shivashankar C R]

Hi,

I have deployed Openstack-ansbile (Pike) in my test environment on Ubuntu 16.

I restarted Controller(it is Deployment node as well) and compute nodes.

After this, i am seeing cinder-volume service being down and lvm is reporting errors on the compute node's cinder logs.

Even "cinder-volumes" virtual group is missing from the vgs output.

On Controller:
---------------

root@ubt-controller1:~# cinder service-list
+------------------+--------------------------------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+--------------------------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | infra1-cinder-scheduler-container-bac085be | nova | enabled | up | 2018-05-18T09:45:36.000000 | - |
| cinder-volume | ubuntu-compute1@lvm | nova | enabled | down | 2018-05-18T06:24:10.000000 | - |
+------------------+--------------------------------------------+------+---------+-------+----------------------------+-----------------+

On Compute:
-----------
logs in compute1-cinder.log are as below:

root@ubuntu-compute1:/openstack/log/compute1-cinder# tail -f cinder-volume.log
2018-05-18 17:42:43.218 5983 ERROR cinder.service [-] Manager for service cinder-volume ubuntu-compute1@lvm is reporting problems, not sending heartbeat. Service will appear "down".
2018-05-18 17:42:53.219 5983 ERROR cinder.service [-] Manager for service cinder-volume ubuntu-compute1@lvm is reporting problems, not sending heartbeat. Service will appear "down".
2018-05-18 17:43:00.019 5983 WARNING cinder.volume.manager [req-b3e6811e-675f-41ce-a780-7b7e55d8b3f6 - - - - -] Update driver status failed: (config name lvm) is uninitialized.
2018-05-18 17:43:03.219 5983 ERROR cinder.service [-] Manager for service cinder-volume ubuntu-compute1@lvm is reporting problems, not sending heartbeat. Service will appear "down".
2018-05-18 17:43:13.220 5983 ERROR cinder.service [-] Manager for service cinder-volume ubuntu-compute1@lvm is reporting problems, not sending heartbeat. Service will appear "down".

Can anyone please suggest me a solution for this??

I am very new to Openstack, can anyone please respond in detail steps about this issue?

Regards,
Shivashankar

[Mohammed Naser (mnaser)]

Hey there,

It looks like you're running into an issue which could be a config problem, but also could be a bug. Do you have output of your cinder-volume logs? That would be helpful in determining the issue

Thanks!

[Shivashankar C R (shivashankarcr)]

Hi,

I got this issue resolved.

I saw lvm2-lvmetad service was disabled but it was active.

-----------------------
root@ubuntu-Compute1:~#
root@ubuntu-Compute1:~# systemctl status lvm2-lvmetad
● lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/lib/systemd/system/lvm2-lvmetad.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-06-05 15:09:32 +08; 2h 10min ago
     Docs: man:lvmetad(8)
 Main PID: 1023 (lvmetad)
   CGroup: /system.slice/lvm2-lvmetad.service
           └─1023 /sbin/lvmetad -f

Jun 05 15:09:32 ubuntu-Compute1 systemd[1]: Started LVM2 metadata daemon.
-----------------------

Executed these series of commands and lvm2 was enabled and things started working.

 systemctl enable lvm2-lvmetad.service
 systemctl enable lvm2-lvmetad.socket
 systemctl start lvm2-lvmetad.service
 systemctl start lvm2-lvmetad.socket

 systemctl status lvm2-lvmetad

-----------------------
root@ubuntu-Compute1:~# systemctl status lvm2-lvmetad
● lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/lib/systemd/system/lvm2-lvmetad.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-06-05 17:29:38 +08; 15h ago
     Docs: man:lvmetad(8)
 Main PID: 1024 (lvmetad)
    Tasks: 1
   Memory: 576.0K
      CPU: 13ms
   CGroup: /system.slice/lvm2-lvmetad.service
           └─1024 /sbin/lvmetad -f
-----------------------
Thanks

[Jonathan Rosser (jrosser)]

I am seeing similar after re-running the setup-openstack playbooks after a rolling replacement upgrade of the control plane from xenial->bionic, with OSA rocky deployment.

The playbooks failed for cinder at this step https://github.com/openstack/openstack-ansible-os_cinder/blob/f9aa4c5a30f54e23665cad9c3396c34d3b8a3287/tasks/cinder_db_setup.yml#L42-L56

The resulting logs for cinder-volume

May 22 10:19:11 infra1.os.mist.rd.bbc.co.uk cinder-volume[7979]: 2019-05-22 10:19:11.768 7979 WARNING cinder.volume.manager [req-43f92ce9-631a-4239-949a-4b84b69c4e01 - - - - -] Update driver status failed: (config name rbd_hdd) is uninitialized.
May 22 10:19:16 infra1.os.mist.rd.bbc.co.uk cinder-volume[7979]: 2019-05-22 10:19:16.750 7979 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_hdd is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:16 infra1.os.mist.rd.bbc.co.uk cinder-volume[7988]: 2019-05-22 10:19:16.912 7988 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_nvme is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:26 infra1.os.mist.rd.bbc.co.uk cinder-volume[7979]: 2019-05-22 10:19:26.751 7979 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_hdd is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:26 infra1.os.mist.rd.bbc.co.uk cinder-volume[7988]: 2019-05-22 10:19:26.912 7988 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_nvme is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:36 infra1.os.mist.rd.bbc.co.uk cinder-volume[7979]: 2019-05-22 10:19:36.751 7979 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_hdd is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:36 infra1.os.mist.rd.bbc.co.uk cinder-volume[7988]: 2019-05-22 10:19:36.913 7988 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_nvme is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:36 infra1.os.mist.rd.bbc.co.uk cinder-volume[7988]: 2019-05-22 10:19:36.917 7988 WARNING cinder.volume.manager [req-c5e58acc-465d-4f87-b7ec-5547b879261e - - - - -] Update driver status failed: (config name rbd_nvme) is uninitialized.
May 22 10:19:46 infra1.os.mist.rd.bbc.co.uk cinder-volume[7979]: 2019-05-22 10:19:46.751 7979 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_hdd is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:46 infra1.os.mist.rd.bbc.co.uk cinder-volume[7988]: 2019-05-22 10:19:46.913 7988 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_nvme is reporting problems, not sending heartbeat. Service will appear "down".

Restarting the cinder-volume service on the infra host brings the service back correctly, unfortunately there does not appear to be anything more useful in the log to determine the root cause.

[Bram Nagtegals (turtleeeeeeeeeeeeeee)]

I also got a similar issue.
It could potentially be caused by rerunning a playbook.
Restarting doesn't fix it for me, someone has a suggestion?

Jul 08 19:27:51 aio1 cinder-volume[25193]: 2019-07-08 19:27:51.616 25193 DEBUG oslo_concurrency.processutils [req-eabd3210-b1db-42ed-91b6-9f8f3d945a34 - - - - -] u'sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --version' failed. Not Retrying. execute /openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/oslo_concurrency/processutils.py:457
Jul 08 19:27:51 aio1 cinder-volume[25193]: 2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager [req-eabd3210-b1db-42ed-91b6-9f8f3d945a34 - - - - -] Failed to initialize driver.: OSError: [Errno 2] No such file or directory
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager Traceback (most recent call last):
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager File "/openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/cinder/volume/manager.py", line 469, in _init_host
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager self.driver.check_for_setup_error()
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager File "/openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 292, in check_for_setup_error
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager if volutils.supports_thin_provisioning():
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager File "/openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/cinder/volume/utils.py", line 644, in supports_thin_provisioning
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager utils.get_root_helper())
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager File "/openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 225, in supports_thin_provisioning
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager return LVM.get_lvm_version(root_helper) >= (2, 2, 95)
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager File "/openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 202, in get_lvm_version
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager run_as_root=True)
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager File "/openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 391, in execute
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager env=env_variables)
                                           2019-07-0...