Failed to create manila shares

[caty]

create manila-share fails, manila-shares showing these logs:

2016-08-17 08:11:16.309 18 WARNING manila.context [-] Arguments dropped when creating context: {u'read_only': False, u'domain': None, u'show_deleted': False, u'user_identity': u'3f5ad05c1bb542d1a6f4e1564748376d b5c540dd99ee4ac38a32dac40ed7d019 - - -', u'project_domain': None, u'resource_uuid': None, u'user_domain': None}.
2016-08-17 08:11:16.860 18 WARNING manila.share.drivers.service_instance [req-38144cf4-2ac4-4229-afed-946b7337a140 3f5ad05c1bb542d1a6f4e1564748376d b5c540dd99ee4ac38a32dac40ed7d019 - - -] No key path is available. May be non-existent key path is provided. Check path_to_private_key (current value /home/stack/.ssh/id_rsa) and path_to_public_key (current value /home/stack/.ssh/id_rsa.pub) in manila configuration file.
2016-08-17 08:11:18.431 18 WARNING manila.network.linux.interface [req-38144cf4-2ac4-4229-afed-946b7337a140 3f5ad05c1bb542d1a6f4e1564748376d b5c540dd99ee4ac38a32dac40ed7d019 - - -] Device tapc3261090-49 already exists.
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters [req-38144cf4-2ac4-4229-afed-946b7337a140 3f5ad05c1bb542d1a6f4e1564748376d b5c540dd99ee4ac38a32dac40ed7d019 - - -] DBAPIError exception wrapped from (pymysql.err.IntegrityError) (1048, u"Column 'value' cannot be null") [SQL: u'INSERT INTO share_server_backend_details (created_at, updated_at, deleted_at, deleted, `key`, value, share_server_id) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(key)s, %(value)s, %(share_server_id)s)'] [parameters: {'deleted': 'False', 'created_at': datetime.datetime(2016, 8, 17, 8, 11, 44, 880720), 'updated_at': None, 'value': None, 'share_server_id': '5e8a0dbe-ea04-4315-8e80-eaf72739ff66', 'key': 'subnet_id', 'deleted_at': None}]
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters context)
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/cursors.py", line 158, in execute
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query)
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/cursors.py", line 308, in _query
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q)
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 820, in query
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 1002, in _read_query_result
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters result.read()
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 1285, in read
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet()
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 966, in _read_packet
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters packet.check_error()
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 394, in check_error
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data)
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters _check_mysql_exception(errinfo)
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/err.py", line 112, in _check_mysql_exception
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters raise errorclass(errno, errorvalue)
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters IntegrityError: (1048, u"Column 'value' cannot be null")
2016-08-17 08:11:44.884 18 ERROR oslo_db.sqlalchemy.exc_filters
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters [req-38144cf4-2ac4-4229-afed-946b7337a140 3f5ad05c1bb542d1a6f4e1564748376d b5c540dd99ee4ac38a32dac40ed7d019 - - -] DBAPIError exception wrapped from (pymysql.err.IntegrityError) (1048, u"Column 'value' cannot be null") [SQL: u'INSERT INTO share_server_backend_details (created_at, updated_at, deleted_at, deleted, `key`, value, share_server_id) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(key)s, %(value)s, %(share_server_id)s)'] [parameters: {'deleted': 'False', 'created_at': datetime.datetime(2016, 8, 17, 8, 11, 44, 942163), 'updated_at': None, 'value': None, 'share_server_id': '5e8a0dbe-ea04-4315-8e80-eaf72739ff66', 'key': 'router_id', 'deleted_at': None}]
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters context)
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/cursors.py", line 158, in execute
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query)
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/cursors.py", line 308, in _query
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q)
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 820, in query
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 1002, in _read_query_result
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters result.read()
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 1285, in read
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet()
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 966, in _read_packet
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters packet.check_error()
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/connections.py", line 394, in check_error
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data)
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters _check_mysql_exception(errinfo)
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/pymysql/err.py", line 112, in _check_mysql_exception
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters raise errorclass(errno, errorvalue)
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters IntegrityError: (1048, u"Column 'value' cannot be null")
2016-08-17 08:11:44.943 18 ERROR oslo_db.sqlalchemy.exc_filters
2016-08-17 08:11:45.152 18 ERROR manila.share.manager [req-38144cf4-2ac4-4229-afed-946b7337a140 3f5ad05c1bb542d1a6f4e1564748376d b5c540dd99ee4ac38a32dac40ed7d019 - - -] Creation of share instance 82d7e199-6ee6-46b5-8e3a-95c2a43339cd failed: failed to get share server.
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher [req-38144cf4-2ac4-4229-afed-946b7337a140 3f5ad05c1bb542d1a6f4e1564748376d b5c540dd99ee4ac38a32dac40ed7d019 - - -] Exception during message handling: Instance 988e7ee1-77da-4ffe-be2c-f6a206dfc6bc failed to reach active state in 300 seconds. Current status: ERROR.
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/share/manager.py", line 146, in wrapped
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher return f(self, *args, **kwargs)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/utils.py", line 616, in wrapper
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher return func(self, *args, **kwargs)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/share/manager.py", line 986, in create_share_instance
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher {'status': constants.STATUS_ERROR}
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/share/manager.py", line 976, in create_share_instance
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher consistency_group=consistency_group_ref
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/share/manager.py", line 456, in _provide_share_server_for_share
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher return _provide_share_server_for_share()
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/share/manager.py", line 448, in _provide_share_server_for_share
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher context, compatible_share_server)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/share/manager.py", line 2390, in _setup_server
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher self.driver.deallocate_network(context, share_server['id'])
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/share/manager.py", line 2344, in _setup_server
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher network_info, metadata=metadata)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/share/driver.py", line 625, in setup_server
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher return self._setup_server(*args, **kwargs)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/share/drivers/generic.py", line 903, in _setup_server
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher self.admin_context, network_info)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/share/drivers/service_instance.py", line 421, in set_up_service_instance
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher context, instance_name, network_info)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/share/drivers/service_instance.py", line 557, in _create_service_instance
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher self.max_time_to_build_instance)
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/manila/share/drivers/service_instance.py", line 681, in wait_for_instance_to_be_active
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher instance_status=instance_status))
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher ServiceInstanceException: Instance 988e7ee1-77da-4ffe-be2c-f6a206dfc6bc failed to reach active state in 300 seconds. Current status: ERROR.
2016-08-17 08:11:45.199 18 ERROR oslo_messaging.rpc.dispatcher

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/366247
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=3091f94a0779b279fc8c4c754aa4adab74ad38f4
Submitter: Jenkins
Branch: master

commit 3091f94a0779b279fc8c4c754aa4adab74ad38f4
Author: Mauricio Lima <email address hidden>
Date: Tue Sep 6 12:30:08 2016 -0400

    Fix manila_share container isn't up after deploy

    Change-Id: Ia5ece5f02841aebc1bdb0950b3f827e1cee359e5
    Closes-Bug: #1620721
    Closes-Bug: #1614033

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/369034

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mauricio Lima (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/369034
Reason: More tests are needed for confirm the necessity for this backport.