Install and configure a storage node in Installation Guide

[Sanusi David]

By following the guide stated in this url (http://docs.openstack.org/newton/install-guide-ubuntu/cinder-storage-install.html) to install the cinder service on a separate block storage node for the newton release, the cinder-volume service does not start. NTP is running on the storage node and syncronized with the controller. Attempts to create a volume generates an error.
Find below logs to possibly help in resolving this issue:

From /var/log/syslog :

Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.202 3963 WARNING oslo_reports.guru_meditation_report [-] Guru mediation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.374 3963 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.463 3963 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.531 3963 WARNING py.warnings [req-ec2d3a89-0f1f-40d3-984b-0984e07dff99 - - - - -] /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py:241: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
Nov 24 14:31:05 localhost cinder-volume[3963]: exception.NotSupportedWarning
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.727 3963 INFO cinder.rpc [req-ec2d3a89-0f1f-40d3-984b-0984e07dff99 - - - - -] Automatically selected cinder-scheduler objects version 1.11 as minimum service version.
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume [req-ec2d3a89-0f1f-40d3-984b-0984e07dff99 - - - - -] Volume service cinder@lvm failed to start.
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume Traceback (most recent call last):
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 81, in main
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume binary='cinder-volume')
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 268, in create
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume service_name=service_name)
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 150, in __init__
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume *args, **kwargs)
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 235, in __init__
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume *args, **kwargs)
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/manager.py", line 156, in __init__
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume self.scheduler_rpcapi = scheduler_rpcapi.SchedulerAPI()
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/rpc.py", line 188, in __init__
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume serializer = base.CinderObjectSerializer(obj_version_cap)
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/objects/base.py", line 412, in __init__
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume raise exception.CappedVersionUnknown(version=version_cap)
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume CappedVersionUnknown: Unrecoverable Error: Versioned Objects in DB are capped to unknown version 1.11.
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume
Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.732 3963 ERROR cinder.cmd.volume [req-ec2d3a89-0f1f-40d3-984b-0984e07dff99 - - - - -] No volume service(s) started successfully, terminating.
Nov 24 14:31:05 localhost systemd[1]: cinder-volume.service: Main process exited, code=exited, status=1/FAILURE
Nov 24 14:31:05 localhost systemd[1]: cinder-volume.service: Unit entered failed state.
Nov 24 14:31:05 localhost systemd[1]: cinder-volume.service: Failed with result 'exit-code'.
Nov 24 14:31:06 localhost systemd[1]: cinder-volume.service: Service hold-off time over, scheduling restart.
Nov 24 14:31:06 localhost systemd[1]: Stopped OpenStack Cinder Volume.
Nov 24 14:31:06 localhost systemd[1]: cinder-volume.service: Start request repeated too quickly.
Nov 24 14:31:06 localhost systemd[1]: Failed to start OpenStack Cinder Volume.

From /var/log/cinder/cinder-volume.log

2016-11-24 14:31:05.202 3963 WARNING oslo_reports.guru_meditation_report [-] Guru mediation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-11-24 14:31:05.374 3963 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
2016-11-24 14:31:05.463 3963 INFO root [-] Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
2016-11-24 14:31:05.531 3963 WARNING py.warnings [req-ec2d3a89-0f1f-40d3-984b-0984e07dff99 - - - - -] /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py:241: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
  exception.NotSupportedWarning

2016-11-24 14:31:05.727 3963 INFO cinder.rpc [req-ec2d3a89-0f1f-40d3-984b-0984e07dff99 - - - - -] Automatically selected cinder-scheduler objects version 1.11 as minimum service version.
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume [req-ec2d3a89-0f1f-40d3-984b-0984e07dff99 - - - - -] Volume service cinder@lvm failed to start.
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume Traceback (most recent call last):
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 81, in main
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume binary='cinder-volume')
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 268, in create
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume service_name=service_name)
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 150, in __init__
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume *args, **kwargs)
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 235, in __init__
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume *args, **kwargs)
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/manager.py", line 156, in __init__
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume self.scheduler_rpcapi = scheduler_rpcapi.SchedulerAPI()
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/rpc.py", line 188, in __init__
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume serializer = base.CinderObjectSerializer(obj_version_cap)
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/objects/base.py", line 412, in __init__
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume raise exception.CappedVersionUnknown(version=version_cap)
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume CappedVersionUnknown: Unrecoverable Error: Versioned Objects in DB are capped to unknown version 1.11.
2016-11-24 14:31:05.728 3963 ERROR cinder.cmd.volume
2016-11-24 14:31:05.732 3963 ERROR cinder.cmd.volume [req-ec2d3a89-0f1f-40d3-984b-0984e07dff99 - - - - -] No volume service(s) started successfully, terminating.

Kindly publish an update to this issue.

-----------------------------------
Release: 0.1 on 2016-11-22 10:55
SHA: 464565b42fc90a51358080b7b3dbfe985a4d0d70
Source: http://git.openstack.org/cgit/openstack/openstack-manuals/tree/doc/install-guide/source/cinder-storage-install.rst
URL: http://docs.openstack.org/newton/install-guide-ubuntu/cinder-storage-install.html

[Joseph Robinson (joseph-r-email)]

Hi Sanusi, There's one similar bug that might have a solution - https://bugs.launchpad.net/openstack-manuals/+bug/1582133 - This one is for RHEL 7.2 and installing python-keystonemiddleware resolved the issue in that instance.

[Sanusi David (davidbolt)]

Hi Joseph, Thanks for responding. It actually turns out that python-keystonemiddleware is already installed on my cinder node running Ubuntu 16.04 and it's up to date. So the cinder-volume service cannot start still. But there is a particular error that stands out in the log that I think might be worth taking a look at:

root@storage:~#cat /var/log/cinder/cinder-volume log
...
CappedVersionUnknown: Unrecoverable Error: Versioned Objects in DB are capped to unknown version 1.11
...

[Alexandra Settle (alexandra-settle)]

Sanusi,

Were you able to replicate this on other versions than Newton? (Mitaka, especially)

Cheers,

Alex

[Sanusi David (davidbolt)]

No I haven't tried this on any other versions. I might try this in my spare
time.

On Thu, Dec 29, 2016 at 11:58 AM, Alexandra Settle <email address hidden>
wrote:

> Sanusi,
>
> Were you able to replicate this on other versions than Newton? (Mitaka,
> especially)
>
> Cheers,
>
> Alex
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1644562
>
> Title:
> Install and configure a storage node in Installation Guide
>
> Status in openstack-manuals:
> New
>
> Bug description:
> By following the guide stated in this url (http://docs.openstack.org/
> newton/install-guide-ubuntu/cinder-storage-install.html) to install the
> cinder service on a separate block storage node for the newton release, the
> cinder-volume service does not start. NTP is running on the storage node
> and syncronized with the controller. Attempts to create a volume generates
> an error.
> Find below logs to possibly help in resolving this issue:
>
> From /var/log/syslog :
>
> Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.202
> 3963 WARNING oslo_reports.guru_meditation_report [-] Guru mediation now
> registers SIGUSR1 and SIGUSR2 by default for backward compatibility.
> SIGUSR1 will no longer be registered in a future release, so please use
> SIGUSR2 to generate reports.
> Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.374
> 3963 INFO root [-] Generating grammar tables from
> /usr/lib/python2.7/lib2to3/Grammar.txt
> Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.463
> 3963 INFO root [-] Generating grammar tables from
> /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.531
> 3963 WARNING py.warnings [req-ec2d3a89-0f1f-40d3-984b-0984e07dff99 - - -
> - -] /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py:241:
> NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported
> Nov 24 14:31:05 localhost cinder-volume[3963]:
> exception.NotSupportedWarning
> Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.727
> 3963 INFO cinder.rpc [req-ec2d3a89-0f1f-40d3-984b-0984e07dff99 - - - - -]
> Automatically selected cinder-scheduler objects version 1.11 as minimum
> service version.
> Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728
> 3963 ERROR cinder.cmd.volume [req-ec2d3a89-0f1f-40d3-984b-0984e07dff99 -
> - - - -] Volume service cinder@lvm failed to start.
> Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728
> 3963 ERROR cinder.cmd.volume Traceback (most recent call last):
> Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728
> 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-
> packages/cinder/cmd/volume.py", line 81, in main
> Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728
> 3963 ERROR cinder.cmd.volume binary='cinder-volume')
> Nov 24 14:31:05 localhost cinder-volume[3963]: 2016-11-24 14:31:05.728
> 3963 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/service.py",
> line 268, in create
> ...

[Alexandra Settle (alexandra-settle)]

Thanks Sanusi, let us know what your findings are!

[Alexandra Settle (alexandra-settle)]

Sanusi - I checked in with the cinder team and their responses were:

Basically this error happens if there are old, dead records in services, that reported older versions of either RPC APIs or objects. Or - the code isn't checked out from exactly the same git ref and versions differ.
- http://docs.openstack.org/developer/cinder/upgrade.html#rpc-version-pinning
- 5th point here: http://docs.openstack.org/developer/cinder/upgrade.html#plan-your-upgrade