Can't run Cinder-Volume (Newton) on Xenial

[GonzO]

After following the instructions here http://docs.openstack.org/newton/install-guide-ubuntu/cinder.html to build an Openstack cluster, the cinder-volume service refuses to start.

The error I'm getting in /var/log/cinder/cinder-volume.log:

2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume [req-5cc49afd-ffc3-4900-a399-da24a2aace0e - - - - -] Volume service mtv-compute-01@lvm failed to start.
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume Traceback (most recent call last):
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 100, in main
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume cluster=cluster)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 382, in create
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume cluster=cluster)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 150, in __init__
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume self.is_upgrading_to_n = self.is_svc_upgrading_to_n(binary)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 223, in is_svc_upgrading_to_n
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume return rpc_api.determine_obj_version_cap() == '1.3'
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/rpc.py", line 233, in determine_obj_version_cap
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume cinder.context.get_admin_context(), cls.BINARY)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 188, in get_minimum_obj_version
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume binary)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 161, in _get_minimum_version
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume services = ServiceList.get_all_by_binary(context, binary)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/objects/service.py", line 216, in get_all_by_binary
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume disabled=disabled)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/db/api.py", line 127, in service_get_all
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume return IMPL.service_get_all(context, backend_match_level, **filters)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 178, in wrapper
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume return f(*args, **kwargs)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 504, in service_get_all
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume return [] if not query else query.all()
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume return list(self)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume return self._execute_and_instances(context)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume result = conn.execute(querycontext.statement, self._params)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume return meth(self, multiparams, params)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume return connection._execute_clauseelement(self, multiparams, params)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume compiled_sql, distilled_params
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume context)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume util.raise_from_cause(newraise, exc_info)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume reraise(type(exception), exception, tb=exc_tb)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume context)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume cursor.execute(statement, parameters)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 158, in execute
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume result = self._query(query)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/pymysql/cursors.py", line 308, in _query
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume conn.query(q)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 820, in query
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1002, in _read_query_result
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume result.read()
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1285, in read
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume first_packet = self.connection._read_packet()
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 966, in _read_packet
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume packet.check_error()
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 394, in check_error
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume err.raise_mysql_exception(self._data)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume _check_mysql_exception(errinfo)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 115, in _check_mysql_exception
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume raise InternalError(errno, errorvalue)
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume DBError: (pymysql.err.InternalError) (1054, u"Unknown column 'services.cluster_name' in 'field list'") [SQL: u'SELECT services.created_at AS services_created_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.cluster_name AS services_cluster_name, services.host AS services_host, services.`binary` AS services_binary, services.updated_at AS services_updated_at, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen \nFROM services \nWHERE services.deleted = false AND services.`binary` = %(binary_1)s'] [parameters: {u'binary_1': 'cinder-volume'}]
2017-02-09 22:26:21.337 15214 ERROR cinder.cmd.volume
2017-02-09 22:26:21.341 15214 ERROR cinder.cmd.volume [req-5cc49afd-ffc3-4900-a399-da24a2aace0e - - - - -] No volume service(s) started successfully, terminating.

The proposed package didn't help any.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: cinder-volume 2:9.1.0-0ubuntu1~cloud0 [origin: Canonical]
ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35
Uname: Linux 4.4.0-59-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
CrashDB:
 {
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
             }
Date: Thu Feb 9 22:27:31 2017
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: cinder
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.cinder.rootwrap.d.volume.filters: [deleted]

[GonzO (gonzo)]

Addendum: That column actually does exist on the DB.

[GonzO (gonzo)]

Please close this bug. After looking at it with fresh eyes today, I (finally) spotted a configuration error. This is on me.