functional mysql tests fail the majority of the time: "Detach replicas before deleting replica source

[Adam Gandelman]

We've been trying to push bot-proposed requirement syncs to trove stable/juno in preparation for a release, but can't seem to get a successful merge. The gate-trove-functional-dsvm-mysql fails >50% of the time.

Tests fail with the first error at:

2015-04-10 06:48:18.116 | 2015-04-10 06:48:18.104 | Wait until the instance is created and set up as slave.
2015-04-10 06:56:00.459 | 2015-04-10 06:56:00.446 | test_slave_created FAIL
2015-04-10 06:56:00.460 | 2015-04-10 06:56:00.447 | VerifySlave
2015-04-10 06:56:00.461 | 2015-04-10 06:56:00.449 | test_correctly_started_replication SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.463 | 2015-04-10 06:56:00.450 | test_backup_deleted SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.464 | 2015-04-10 06:56:00.452 | test_slave_is_read_only SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.466 | 2015-04-10 06:56:00.453 | test_create_db_on_master SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.467 | 2015-04-10 06:56:00.455 | test_database_replicated_on_slave SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.469 | 2015-04-10 06:56:00.456 | test_existing_db_exists_on_slave SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.470 | 2015-04-10 06:56:00.458 | test_slave_user_exists SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.472 | 2015-04-10 06:56:00.459 | TestInstanceListing
2015-04-10 06:56:00.474 | 2015-04-10 06:56:00.461 | Test replication information in instance listing.
2015-04-10 06:56:00.475 | 2015-04-10 06:56:00.462 | test_get_master_instance SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.476 | 2015-04-10 06:56:00.464 | test_get_slave_instance SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.479 | 2015-04-10 06:56:00.466 | DetachReplica
2015-04-10 06:56:00.480 | 2015-04-10 06:56:00.467 | delete_before_detach_replica SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.482 | 2015-04-10 06:56:00.469 | test_detach_replica SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.483 | 2015-04-10 06:56:00.470 | test_slave_is_not_read_only SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.484 | 2015-04-10 06:56:00.472 | test_slave_user_removed SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.486 | 2015-04-10 06:56:00.473 | DeleteSlaveInstance
2015-04-10 06:56:00.487 | 2015-04-10 06:56:00.475 | test_delete_slave_instance SKIP: Failure in <function test_slave_created at 0x7fc896dd5b90>
2015-04-10 06:56:00.489 | 2015-04-10 06:56:00.476 | DeleteInstance
2015-04-10 06:56:00.490 | 2015-04-10 06:56:00.478 | Delete the created instance.
2015-04-10 06:56:00.573 | 2015-04-10 06:56:00.560 | test_delete ERROR
2015-04-10 06:56:00.574 | 2015-04-10 06:56:00.561 | test_volume_is_deleted SKIP: Cannot test volume is deleted from db.
2015-04-10 06:56:00.576 | 2015-04-10 06:56:00.563 | AfterDeleteChecks
2015-04-10 06:56:00.944 | 2015-04-10 06:56:00.931 | test_instance_delete_event_sent ERROR
2015-04-10 06:56:01.032 | 2015-04-10 06:56:01.019 | test_instance_status_deleted_in_db FAIL
2015-04-10 06:56:01.033 | 2015-04-10 06:56:01.021 |
2015-04-10 06:56:01.035 | 2015-04-10 06:56:01.022 | ======================================================================
2015-04-10 06:56:01.037 | 2015-04-10 06:56:01.024 | ERROR: proboscis.case.MethodTest (test_delete)
2015-04-10 06:56:01.038 | 2015-04-10 06:56:01.026 | ----------------------------------------------------------------------
2015-04-10 06:56:01.040 | 2015-04-10 06:56:01.027 | Traceback (most recent call last):
2015-04-10 06:56:01.042 | 2015-04-10 06:56:01.029 | File "/usr/local/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
2015-04-10 06:56:01.043 | 2015-04-10 06:56:01.030 | compatability.capture_type_error(s_func)
2015-04-10 06:56:01.044 | 2015-04-10 06:56:01.032 | File "/usr/local/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
2015-04-10 06:56:01.046 | 2015-04-10 06:56:01.033 | func()
2015-04-10 06:56:01.047 | 2015-04-10 06:56:01.035 | File "/usr/local/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
2015-04-10 06:56:01.049 | 2015-04-10 06:56:01.036 | func(test_case.state.get_state())
2015-04-10 06:56:01.050 | 2015-04-10 06:56:01.038 | File "/opt/stack/new/trove/trove/tests/api/instances.py", line 1235, in test_delete
2015-04-10 06:56:01.052 | 2015-04-10 06:56:01.039 | dbaas.instances.delete(instance_info.id)
2015-04-10 06:56:01.054 | 2015-04-10 06:56:01.041 | File "/usr/local/lib/python2.7/dist-packages/troveclient/v1/instances.py", line 153, in delete
2015-04-10 06:56:01.055 | 2015-04-10 06:56:01.042 | resp, body = self.api.client.delete(url)
2015-04-10 06:56:01.057 | 2015-04-10 06:56:01.044 | File "/usr/local/lib/python2.7/dist-packages/troveclient/compat/client.py", line 249, in delete
2015-04-10 06:56:01.058 | 2015-04-10 06:56:01.046 | return self._cs_request(url, 'DELETE', **kwargs)
2015-04-10 06:56:01.060 | 2015-04-10 06:56:01.047 | File "/usr/local/lib/python2.7/dist-packages/troveclient/compat/client.py", line 231, in _cs_request
2015-04-10 06:56:01.061 | 2015-04-10 06:56:01.049 | return request()
2015-04-10 06:56:01.063 | 2015-04-10 06:56:01.050 | File "/usr/local/lib/python2.7/dist-packages/troveclient/compat/client.py", line 221, in request
2015-04-10 06:56:01.064 | 2015-04-10 06:56:01.052 | **kwargs)
2015-04-10 06:56:01.066 | 2015-04-10 06:56:01.053 | File "/usr/local/lib/python2.7/dist-packages/troveclient/compat/client.py", line 209, in _time_request
2015-04-10 06:56:01.067 | 2015-04-10 06:56:01.055 | resp, body = self.request(url, method, **kwargs)
2015-04-10 06:56:01.069 | 2015-04-10 06:56:01.056 | File "/usr/local/lib/python2.7/dist-packages/troveclient/compat/client.py", line 187, in request
2015-04-10 06:56:01.070 | 2015-04-10 06:56:01.058 | raise exceptions.from_response(resp, body)
2015-04-10 06:56:01.072 | 2015-04-10 06:56:01.059 | Forbidden: Detach replicas before deleting replica source. (HTTP 403)
2015-04-10 06:56:01.073 | 2015-04-10 06:56:01.060 | -------------------- >> begin captured logging << --------------------
2015-04-10 06:56:01.075 | 2015-04-10 06:56:01.062 | troveclient.compat.client: DEBUG: REQ: curl -i http://127.0.0.1:8779/v1.0/1cde25ce6a8d474aaa53eb418a53e632/instances/a0998e14-6142-4e80-b8c1-1c3be03c4755 DELETE -H "X-Auth-Project-Id: alt_demo" -H "User-Agent: python-troveclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: b7d58a9168b646188c61ba639073094c"
2015-04-10 06:56:01.076 | 2015-04-10 06:56:01.063 |
2015-04-10 06:56:01.077 | 2015-04-10 06:56:01.065 | troveclient.compat.client: DEBUG: RESP:{'date': 'Fri, 10 Apr 2015 06:56:00 GMT', 'status': '403', 'content-length': '90', 'content-type': 'application/json; charset=UTF-8'} {"forbidden": {"message": "Detach replicas before deleting replica source.", "code": 403}}
2015-04-10 06:56:01.079 | 2015-04-10 06:56:01.066 |
2015-04-10 06:56:01.080 | 2015-04-10 06:56:01.068 | --------------------- >> end captured logging << ---------------------

See: https://review.openstack.org/#/c/171643/

or

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRGV0YWNoIHJlcGxpY2FzIGJlZm9yZSBkZWxldGluZyByZXBsaWNhIHNvdXJjZS5cIiBBTkQgdGFnczpcImNvbnNvbGVcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiYWxsIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyODY0OTIxNTkzNH0=

At 1h20 min per test run, this makes a failed check+verify almost 3 hours of wasted time. This is currently blocking the 2014.2.3 release at midnight :|

[Morgan Jones (6-morgan)]

The issue indicated is a red herring - the test is failing because it can't create a slave - the detach is merely a side effect of the failure.

Unfortunately, the tests are now passing in the indicated changeset and the linked log doesn't appear to be available.

[Launchpad Janitor (janitor)]

[Expired for Trove because there has been no activity for 60 days.]