Instance fail to be built due to the error "No sql_connection parameter is established"

[KC Bi]

Recently, below error is being hit frequently on Dell EqualLogic and Dell Storage Center iSCSI CI setups:

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/test.py", line 103, in wrapper
        return f(self, *func_args, **func_kwargs)
      File "tempest/scenario/test_volume_boot_pattern.py", line 105, in test_volume_boot_pattern
        security_group=security_group)
      File "tempest/scenario/test_volume_boot_pattern.py", line 70, in _boot_instance_from_resource
        return self.create_server(image_id='', **create_kwargs)
      File "tempest/scenario/manager.py", line 203, in create_server
        image_id=image_id, **kwargs)
      File "tempest/common/compute.py", line 219, in create_test_server
        server['id'])
      File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
        self.force_reraise()
      File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
        six.reraise(self.type_, self.value, self.tb)
      File "tempest/common/compute.py", line 188, in create_test_server
        clients.servers_client, server['id'], wait_until)
      File "tempest/common/waiters.py", line 76, in wait_for_server_status
        server_id=server_id)
    tempest.exceptions.BuildErrorException: Server aac418e5-82eb-497e-9405-8fa1c511b4aa failed to build and is in ERROR status
    Details: {u'code': 500, u'message': u'No sql_connection parameter is established', u'created': u'2017-08-04T11:09:46Z'}

These 2 x setups work fine previously, and nothing has been changed physically. Some OpenStack code must contribute to this.

10 x times test have been finished today with several cases which tend to fail easily. 6 out of 10 runs failed with 1 or 2 cases with above mentioned issue. Please refer to the attached log for details.

[Jared Rice Sr. (arisebank)]

Same issue here.

[Jesse Wu (jessewu1)]

Our Cinder CI was in a similar situation as the original poster.
All runs had started failing with no changes made on our side.
What was a bit different was that all new build runs had failed on different tempest tests,
though all of the tests pretty much had the same error stack trace from the "create_server" line down to the "No sql_connection parameter is established" error.

Grepping the Server uuid in various other logs, I found that there was also this recurring error:
"libvirtError: operation failed: domain 'instance-0000001a' already exists with uuid e7a7695d-9617-4a98-84ef-b9a11dc97f8d"

After removing the already existing instances (`virsh undefine instance-0000001a`), the builds started succeeding again.
(Ref: https://serverfault.com/questions/362796/libvirterror-that-domain-already-exists-but-virsh-list-doesnt-show-it/362835#362835)

Not sure if the lingering virsh instance was the root cause of the "No sql_connection parameter is established" error,
but removing the virsh instance had made the error disappear in our case.
Hope this may be of use to someone.

[KC Bi (bikecheng)]

Below configuration can be added to ease the issue, just FYI:

[database]
db_max_retries = 20
retry_interval = 10

[Dr. Jens Harbott (j-harbott)]

This looks like an issue in the CI setup, if you can reproduce this with a fresh devstack installation, please include the steps to reproduce and a full stack.sh log.

[Launchpad Janitor (janitor)]

[Expired for devstack because there has been no activity for 60 days.]