Cinder is unable to connect to the database

[Ryan Hallisey]

2015-08-05 21:54:20.341 1 CRITICAL cinder [-] DBConnectionError: (OperationalError) (2002, "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)") None None
2015-08-05 21:54:20.341 1 ERROR cinder Traceback (most recent call last):
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/bin/cinder-volume", line 10, in <module>
2015-08-05 21:54:20.341 1 ERROR cinder sys.exit(main())
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 81, in main
2015-08-05 21:54:20.341 1 ERROR cinder server = service.Service.create(binary='cinder-volume')
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/service.py", line 249, in create
2015-08-05 21:54:20.341 1 ERROR cinder service_name=service_name)
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/service.py", line 129, in __init__
2015-08-05 21:54:20.341 1 ERROR cinder *args, **kwargs)
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 200, in __init__
2015-08-05 21:54:20.341 1 ERROR cinder *args, **kwargs)
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/manager.py", line 143, in __init__
2015-08-05 21:54:20.341 1 ERROR cinder super(SchedulerDependentManager, self).__init__(host, db_driver)
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/manager.py", line 85, in __init__
2015-08-05 21:54:20.341 1 ERROR cinder super(Manager, self).__init__(db_driver)
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/db/base.py", line 42, in __init__
2015-08-05 21:54:20.341 1 ERROR cinder self.db.dispose_engine()
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 84, in dispose_engine
2015-08-05 21:54:20.341 1 ERROR cinder if 'sqlite' not in IMPL.get_engine().name:
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 86, in get_engine
2015-08-05 21:54:20.341 1 ERROR cinder facade = _create_facade_lazily()
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 73, in _create_facade_lazily
2015-08-05 21:54:20.341 1 ERROR cinder **dict(CONF.database)
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 930, in __init__
2015-08-05 21:54:20.341 1 ERROR cinder slave_connection=slave_connection)
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 339, in _start
2015-08-05 21:54:20.341 1 ERROR cinder engine_args, maker_args)
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 355, in _setup_for_connection
2015-08-05 21:54:20.341 1 ERROR cinder sql_connection=sql_connection, **engine_kwargs)
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 152, in create_engine
2015-08-05 21:54:20.341 1 ERROR cinder test_conn = _test_connection(engine, max_retries, retry_interval)
2015-08-05 21:54:20.341 1 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 331, in _test_connection
2015-08-05 21:54:20.341 1 ERROR cinder six.reraise(type(de_ref), de_ref)
2015-08-05 21:54:20.341 1 ERROR cinder File "<string>", line 2, in reraise
2015-08-05 21:54:20.341 1 ERROR cinder DBConnectionError: (OperationalError) (2002, "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)") None None
2015-08-05 21:54:20.341 1 ERROR cinder
INFO - This container has already been configured; Refusing to copy new configs
No handlers could be found for logger "oslo_config.cfg"

[Ryan Hallisey (rthall14)]

This was using the ansible-cinder patch. It's possible that was the cause of it.

[Sam Yaple (s8m)]

Reading that stacktrace shows that the cinder conf for database is clearly not getting read properly. It is even trying to talk to the database via a socket. If i had to guess this is not pulling in the config properly for a bootstrap.

DBConnectionError: (OperationalError) (2002, "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)") None None

An additional question is why are we filing bugs against code that is not merged yet? Shouldn't this type of thing be a -1 on the patchset?

[Ryan Hallisey (rthall14)]

Good point. I wasn't sure it was the cause, but at the very least it will make coolsvap aware of the issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/212157

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/212157
Committed: https://git.openstack.org/cgit/stackforge/kolla/commit/?id=759cc41647072061574eccd2dcaaa88c168620c2
Submitter: Jenkins
Branch: master

commit 759cc41647072061574eccd2dcaaa88c168620c2
Author: rthallisey <email address hidden>
Date: Wed Aug 12 15:22:01 2015 -0400

    All the Cinder services need to reference the correct config

    Every service except cinder-api was looking for the file
    '/opt/kolla/cinder/cinder-<service>.conf' when the config file in place was named
    'cinder.conf'. Change the rest of the services to look for cinder.conf or
    the config file will never be copied over to into '/etc/cinder/'.

    Change-Id: I1cd59f1da09e6f3bc7e7b3a4f45f9da2b8247479
    Closes-Bug: #1481928
    Closes-Bug: #1484248