500 error returned in List Records in a Domain

[kenichiro matsuda]

Ongoing repeat the Create Record and the Delete a record,
there is a case where 500 error returned in the List Records in a Domain
that ran in these in between.

* reproduce the bug:
  Ongoing repeat the following APIs.
    1. Create Record
      - POST /v1/domains/(uuid: domain_id)/records
        - Use a different "name" each time.
    2. List Records in a Domain
      - GET /v1/domains/(uuid: domain_id)/records
    3. Delete a record
      - DELETE /v1/domains/(uuid: domain_id)/records/(uuid: id)

* Actual results:
  There is a case where 500 error returned in the "2. List Records in a Domain".

* Expected results:
  Return 200 in all APIs.

* Additional note:
  A "KeyError" exception raised in the designate-api.

[designate-api log]
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware Traceback (most recent call last):
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware return request.get_response(self.application)
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware resp = self.call_func(req, *args, **self.kwargs)
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware return self.func(req, *args, **kwargs)
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/designate/api/middleware.py", line 221, in __call__
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware return request.get_response(self.application)
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/webob/request.py", line 1317, in send
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware application, catch_exc_info=False)
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware app_iter = application(self.environ, start_response)
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware return self.wsgi_app(environ, start_response)
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware response = self.make_response(self.handle_exception(e))
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware reraise(exc_type, exc_value, tb)
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware response = self.full_dispatch_request()
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware rv = self.handle_user_exception(e)
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware reraise(exc_type, exc_value, tb)
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware rv = self.dispatch_request()
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware return self.view_functions[rule.endpoint](**req.view_args)
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/designate/api/v1/records.py", line 165, in get_records
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware records = [_inner(r) for r in records]
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware File "/home/ossuser/openstack/designate/.venv/lib/python2.7/site-packages/designate/api/v1/records.py", line 162, in _inner
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware recordset = recordsets[record['recordset_id']]
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware KeyError: u'48ef11a6-4148-431a-8955-7d70694e2f1b'
2015-08-24 00:49:16.257 24625 TRACE designate.api.middleware
2015-08-24 00:49:16.260 24625 INFO eventlet.wsgi [req-44de24f5-a642-43fc-a948-d9ff2084bb35 noauth-user noauth-project - - -] 127.0.0.1 - - [24/Aug/2015 00:49:16] "GET /v1/domains/a3bbb48b-451c-401c-9390-ab8194ae722c/records HTTP/1.1" 500 307 0.304923

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/226313

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/226313
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=eea5c1e8ad55d7ec17bf4695ba63efbc0e170784
Submitter: Jenkins
Branch: master

commit eea5c1e8ad55d7ec17bf4695ba63efbc0e170784
Author: Kiall Mac Innes <email address hidden>
Date: Tue Sep 22 14:05:01 2015 +0100

    Fix race condition in V1 Records API

    When listing records in the V1 API, two calls to central
    are made - once for records once for recordsets - if the
    content of either changes between the two calls, a failure
    can occour. Instead, we already have all the data we need
    so use it from the recordsets rather than make two calls

    Change-Id: Ic43d7e8c0d60267ec4788b63ed5f7e88e7bbe625
    Closes-Bug: 1487934

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/227433

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/227433
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=21de1d5351582bf1854ecf2e9b7d6e09e5819248
Submitter: Jenkins
Branch: stable/kilo

commit 21de1d5351582bf1854ecf2e9b7d6e09e5819248
Author: Kiall Mac Innes <email address hidden>
Date: Tue Sep 22 14:05:01 2015 +0100

    Fix race condition in V1 Records API

    When listing records in the V1 API, two calls to central
    are made - once for records once for recordsets - if the
    content of either changes between the two calls, a failure
    can occour. Instead, we already have all the data we need
    so use it from the recordsets rather than make two calls

    Change-Id: Ic43d7e8c0d60267ec4788b63ed5f7e88e7bbe625
    Closes-Bug: 1487934
    (cherry picked from commit eea5c1e8ad55d7ec17bf4695ba63efbc0e170784)