DevStack gates occasionally failing during tempest tests

[Kiall Mac Innes]

See http://logs.openstack.org/68/197968/2/gate/gate-designate-dsvm-powerdns/7a73fbb/

No obvious reason why.

2015-07-06 11:51:01.687 | 2015-07-06 11:51:01.669 |
2015-07-06 11:51:01.688 | 2015-07-06 11:51:01.671 | ==============================
2015-07-06 11:51:01.690 | 2015-07-06 11:51:01.672 | Failed 1 tests - output below:
2015-07-06 11:51:01.691 | 2015-07-06 11:51:01.674 | ==============================
2015-07-06 11:51:01.693 | 2015-07-06 11:51:01.675 |
2015-07-06 11:51:01.694 | 2015-07-06 11:51:01.677 | v2.test_recordset.RecordsetTest.test_crud_recordset_A
2015-07-06 11:51:01.708 | 2015-07-06 11:51:01.678 | -----------------------------------------------------
2015-07-06 11:51:01.724 | 2015-07-06 11:51:01.680 |
2015-07-06 11:51:01.725 | 2015-07-06 11:51:01.681 | Captured traceback:
2015-07-06 11:51:01.725 | 2015-07-06 11:51:01.682 | ~~~~~~~~~~~~~~~~~~~
2015-07-06 11:51:01.725 | 2015-07-06 11:51:01.684 | Traceback (most recent call last):
2015-07-06 11:51:01.725 | 2015-07-06 11:51:01.685 | File "/opt/stack/new/designate/functionaltests/common/utils.py", line 25, in new_method
2015-07-06 11:51:01.725 | 2015-07-06 11:51:01.686 | return f(self, *args, **kwargs)
2015-07-06 11:51:01.726 | 2015-07-06 11:51:01.688 | File "/opt/stack/new/designate/functionaltests/api/v2/test_recordset.py", line 63, in test_crud_recordset
2015-07-06 11:51:01.726 | 2015-07-06 11:51:01.689 | self.zone.id, recordset_id)
2015-07-06 11:51:01.726 | 2015-07-06 11:51:01.690 | File "/opt/stack/new/designate/functionaltests/api/v2/clients/recordset_client.py", line 61, in wait_for_recordset
2015-07-06 11:51:01.726 | 2015-07-06 11:51:01.692 | lambda: self.is_recordset_active(zone_id, recordset_id))
2015-07-06 11:51:01.726 | 2015-07-06 11:51:01.693 | File "/opt/stack/new/designate/functionaltests/common/utils.py", line 94, in wait_for_condition
2015-07-06 11:51:01.727 | 2015-07-06 11:51:01.695 | raise Exception("Timed out after {0} seconds".format(timeout))
2015-07-06 11:51:01.727 | 2015-07-06 11:51:01.696 | Exception: Timed out after 40 seconds

[Kiall Mac Innes (kiall)]

Cause looks to be:

2015-07-14 13:43:39.276 ERROR designate.service [req-39b841da-1ccd-4786-91a8-d61456d00a9b - - - - -] Unhandled exception while processing request from 192.168.27.100:14244
2015-07-14 13:43:39.276 TRACE designate.service Traceback (most recent call last):
2015-07-14 13:43:39.276 TRACE designate.service File "/opt/stack/designate/designate/service.py", line 372, in _dns_handle
2015-07-14 13:43:39.276 TRACE designate.service self._dns_sock_udp.sendto(response, addr)
2015-07-14 13:43:39.276 TRACE designate.service File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 382, in sendto
2015-07-14 13:43:39.276 TRACE designate.service self._trampoline(self.fd, write=True)
2015-07-14 13:43:39.276 TRACE designate.service File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 201, in _trampoline
2015-07-14 13:43:39.276 TRACE designate.service mark_as_closed=self._mark_as_closed)
2015-07-14 13:43:39.276 TRACE designate.service File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/__init__.py", line 160, in trampoline
2015-07-14 13:43:39.276 TRACE designate.service listener = hub.add(hub.WRITE, fileno, current.switch, current.throw, mark_as_closed)
2015-07-14 13:43:39.276 TRACE designate.service File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/epolls.py", line 49, in add
2015-07-14 13:43:39.276 TRACE designate.service listener = BaseHub.add(self, evtype, fileno, cb, tb, mac)
2015-07-14 13:43:39.276 TRACE designate.service File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in add
2015-07-14 13:43:39.276 TRACE designate.service evtype, fileno, evtype, cb, bucket[fileno]))
2015-07-14 13:43:39.276 TRACE designate.service RuntimeError: Second simultaneous write on fileno 8 detected. Unless you really know what you're doing, make sure that only one greenthread can write any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False) - MY THREAD=<built-in method switch of GreenThread object at 0x7f71e04af370>; THAT THREAD=FdListener('write', 8, <built-in method switch of GreenThread object at 0x7f71e04af550>, <built-in method throw of GreenThread object at 0x7f71e04af550>)
2015-07-14 13:43:39.276 TRACE designate.service

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/201563

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/202004

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/201563
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=3b70b72085825bf4a59bd90aa8e1d1b048934d03
Submitter: Jenkins
Branch: master

commit 3b70b72085825bf4a59bd90aa8e1d1b048934d03
Author: Kiall Mac Innes <email address hidden>
Date: Tue Jul 14 14:45:53 2015 +0100

    Allow multiple reader/writers to UDP DNS Sockets

    Eventlet would complain loudly about multiple readers/writers
    to the UDP DNS socket. As DNS over UDP is single datagram, we
    can safely ignore this.

    Change-Id: I06c5854de4ea7f5ca48fa6e24efbe8873afc7a8b
    Closes-Bug: 1471789

[OpenStack Infra (hudson-openstack)]

Change abandoned by Graham Hayes (<email address hidden>) on branch: master
Review: https://review.openstack.org/202004
Reason: As