keystone reports errors saying backends.sql module is not found

[Vedamurthy Joshi]

R3.0 2704 Ubuntu 14.04 Kilo multi-node HA setup

On this HA setup, verify_openstack task failed. It was seen that keystone.log throws these errors saying backends.sql module was not found.

2016-01-27 19:59:31:768256: [root@10.204.216.58] Executing task 'verify_openstack'
2016-01-27 19:59:31:768622: [root@10.204.216.58] sudo: service keystone status
2016-01-27 19:59:31:768880: [root@10.204.216.58] out: keystone RUNNING pid 14528, uptime 0:08:50
2016-01-27 19:59:31:872590: [root@10.204.216.58] out:
2016-01-27 19:59:31:872860:
2016-01-27 19:59:31:873805: [root@10.204.216.58] sudo: source /etc/contrail/openstackrc; keystone tenant-list
2016-01-27 19:59:31:874023: [root@10.204.216.58] out: Authorization Failed: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-91282241-cd0a-49d9-99b0-1ce401cf0a22)
2016-01-27 19:59:32:609619: [root@10.204.216.58] out:
2016-01-27 19:59:32:625146:

2016-01-27 19:59:32:625751: Warning: sudo() received nonzero return code 1 while executing 'source /etc/contrail/openstackrc; keystone tenant-list'!
2016-01-27 19:59:32:625751:

=======
keystone.log:

2016-01-27 20:00:24.817 14654 TRACE keystone.common.wsgi
2016-01-27 20:00:55.956 14656 ERROR keystone.common.wsgi [-] No module named backends.sql
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi Traceback (most recent call last):
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 238, in __call__
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi result = method(context, **params)
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/controllers.py", line 145, in authenticate
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi auth_token_data, roles_ref=roles_ref, catalog_ref=catalog_ref)
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/provider.py", line 333, in issue_v2_token
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi self._create_token(token_id, data)
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/provider.py", line 190, in _create_token
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi self._persistence.get_token(token_id)
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/provider.py", line 164, in _persistence
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi self._persistence_manager = persistence.PersistenceManager()
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/dependency.py", line 193, in wrapper
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi self.__wrapped_init__(*args, **kwargs)
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/token/persistence/core.py", line 50, in __init__
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi super(PersistenceManager, self).__init__(CONF.token.driver)
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 70, in __init__
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi self.driver = importutils.import_object(driver_name)
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 38, in import_object
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi return import_class(import_str)(*args, **kwargs)
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 27, in import_class
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi __import__(mod_str)
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi ImportError: No module named backends.sql
2016-01-27 20:00:55.956 14656 TRACE keystone.common.wsgi
2016-01-27 20:02:02.003 17983 CRITICAL keystone [-] ImportError: No module named backends.sql
2016-01-27 20:02:02.003 17983 TRACE keystone Traceback (most recent call last):
2016-01-27 20:02:02.003 17983 TRACE keystone File "/usr/bin/keystone-manage", line 44, in <module>
2016-01-27 20:02:02.003 17983 TRACE keystone cli.main(argv=sys.argv, config_files=config_files)
2016-01-27 20:02:02.003 17983 TRACE keystone File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 585, in main
2016-01-27 20:02:02.003 17983 TRACE keystone CONF.command.cmd_class.main()
2016-01-27 20:02:02.003 17983 TRACE keystone File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 242, in main
2016-01-27 20:02:02.003 17983 TRACE keystone token_manager = token.persistence.PersistenceManager()
2016-01-27 20:02:02.003 17983 TRACE keystone File "/usr/lib/python2.7/dist-packages/keystone/common/dependency.py", line 193, in wrapper
2016-01-27 20:02:02.003 17983 TRACE keystone self.__wrapped_init__(*args, **kwargs)
2016-01-27 20:02:02.003 17983 TRACE keystone File "/usr/lib/python2.7/dist-packages/keystone/token/persistence/core.py", line 50, in __init__
2016-01-27 20:02:02.003 17983 TRACE keystone super(PersistenceManager, self).__init__(CONF.token.driver)
2016-01-27 20:02:02.003 17983 TRACE keystone File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 70, in __init__
2016-01-27 20:02:02.003 17983 TRACE keystone self.driver = importutils.import_object(driver_name)
2016-01-27 20:02:02.003 17983 TRACE keystone File "/usr/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 38, in import_object
2016-01-27 20:02:02.003 17983 TRACE keystone return import_class(import_str)(*args, **kwargs)
2016-01-27 20:02:02.003 17983 TRACE keystone File "/usr/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 27, in import_class
2016-01-27 20:02:02.003 17983 TRACE keystone __import__(mod_str)
2016-01-27 20:02:02.003 17983 TRACE keystone ImportError: No module named backends.sql
2016-01-27 20:02:02.003 17983 TRACE keystone
====================

env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6, host7],
    'cfgm': [host1,host2,host3],
    'openstack': [host1,host2,host3],
    'control': [host1,host2,host3],
    'compute': [host4,host5, host6, host7],
    'collector': [host1,host2,host3],
    'webui': [host1],
    'database': [host1,host2,host3],
    'toragent': [host6, host7],
    'tsn': [host6, host7],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodec1', 'nodec2', 'nodec3', 'nodek1', 'nodek2', 'nodek3', 'nodeg11']
}