Designate Agent With BIND 9 Driver Crashes

[Ron Rickard]

I have a configuration with the Designate Central, Sink, and API running on one node and the Designate Agent running on another node with a BIND 9 backend. When the Designate Agent is started without the Designate Central running, the Designate Agent crashes after a short time with the following errors:

2013-12-04 15:04:07 INFO [designate.central.rpcapi] find_domains: Calling central's find_domains.
2013-12-04 15:04:07 DEBUG [designate.openstack.common.rpc.amqp] Making synchronous call on central ...
2013-12-04 15:04:07 DEBUG [designate.openstack.common.rpc.amqp] MSG_ID is 28dfc6007d51486c96025aee773a8608
2013-12-04 15:04:07 DEBUG [designate.openstack.common.rpc.amqp] UNIQUE_ID is 24e96bc51c2a41c9885b5f7b2c6ee7cc.
2013-12-04 15:04:07 DEBUG [designate.openstack.common.rpc.amqp] Pool creating new connection
2013-12-04 15:04:07 INFO [designate.openstack.common.rpc.common] Connected to AMQP server on 192.168.56.35:5672
2013-12-04 15:05:07 DEBUG [designate.openstack.common.rpc.common] Timed out waiting for RPC response: timed out
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
    readers.get(fileno, noop).cb(fileno)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/designate/designate/openstack/common/service.py", line 65, in run_service
    service.start()
  File "/opt/stack/designate/designate/agent/service.py", line 40, in start
    self.manager.start()
  File "/opt/stack/designate/designate/backend/impl_bind9.py", line 45, in start
    domains = self.central_service.find_domains(self.admin_context)
  File "/opt/stack/designate/designate/central/rpcapi.py", line 181, in find_domains
    return self.call(context, msg)
  File "/opt/stack/designate/designate/openstack/common/rpc/proxy.py", line 80, in call
    return rpc.call(context, self._get_topic(topic), msg, timeout)
  File "/opt/stack/designate/designate/openstack/common/rpc/__init__.py", line 140, in call
    return _get_impl().call(CONF, context, topic, msg, timeout)
  File "/opt/stack/designate/designate/openstack/common/rpc/impl_kombu.py", line 798, in call
    rpc_amqp.get_connection_pool(conf, Connection))
  File "/opt/stack/designate/designate/openstack/common/rpc/amqp.py", line 612, in call
    rv = list(rv)
  File "/opt/stack/designate/designate/openstack/common/rpc/amqp.py", line 554, in __iter__
    self.done()
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/opt/stack/designate/designate/openstack/common/rpc/amqp.py", line 551, in __iter__
    self._iterator.next()
  File "/opt/stack/designate/designate/openstack/common/rpc/impl_kombu.py", line 648, in iterconsume
    yield self.ensure(_error_callback, _consume)
  File "/opt/stack/designate/designate/openstack/common/rpc/impl_kombu.py", line 566, in ensure
    error_callback(e)
  File "/opt/stack/designate/designate/openstack/common/rpc/impl_kombu.py", line 629, in _error_callback
    raise rpc_common.Timeout()
Timeout: Timeout while waiting on RPC response.
Removing descriptor: 5
2013-12-04 15:05:07 ERROR [designate.openstack.common.threadgroup] Timeout while waiting on RPC response.
Traceback (most recent call last):
  File "/opt/stack/designate/designate/openstack/common/threadgroup.py", line 110, in wait
    x.wait()
  File "/opt/stack/designate/designate/openstack/common/threadgroup.py", line 49, in wait
    return self.thread.wait()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
    return self._exit_event.wait()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
    return hubs.get_hub().switch()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
    return self.greenlet.switch()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/designate/designate/openstack/common/service.py", line 65, in run_service
    service.start()
  File "/opt/stack/designate/designate/agent/service.py", line 40, in start
    self.manager.start()
  File "/opt/stack/designate/designate/backend/impl_bind9.py", line 45, in start
    domains = self.central_service.find_domains(self.admin_context)
  File "/opt/stack/designate/designate/central/rpcapi.py", line 181, in find_domains
    return self.call(context, msg)
  File "/opt/stack/designate/designate/openstack/common/rpc/proxy.py", line 80, in call
    return rpc.call(context, self._get_topic(topic), msg, timeout)
  File "/opt/stack/designate/designate/openstack/common/rpc/__init__.py", line 140, in call
    return _get_impl().call(CONF, context, topic, msg, timeout)
  File "/opt/stack/designate/designate/openstack/common/rpc/impl_kombu.py", line 798, in call
    rpc_amqp.get_connection_pool(conf, Connection))
  File "/opt/stack/designate/designate/openstack/common/rpc/amqp.py", line 612, in call
    rv = list(rv)
  File "/opt/stack/designate/designate/openstack/common/rpc/amqp.py", line 554, in __iter__
    self.done()
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/opt/stack/designate/designate/openstack/common/rpc/amqp.py", line 551, in __iter__
    self._iterator.next()
  File "/opt/stack/designate/designate/openstack/common/rpc/impl_kombu.py", line 648, in iterconsume
    yield self.ensure(_error_callback, _consume)
  File "/opt/stack/designate/designate/openstack/common/rpc/impl_kombu.py", line 566, in ensure
    error_callback(e)
  File "/opt/stack/designate/designate/openstack/common/rpc/impl_kombu.py", line 629, in _error_callback
    raise rpc_common.Timeout()
Timeout: Timeout while waiting on RPC response.

The reason the Designate Agent is dying is because the BIND 9 backend driver (designate/backend/impl_bind9.py) is attempting to contact the Designate Central to retrieve the list of domains so it can do a 'rndc reload' on them. This can be fixed by either not retrieving the list of domains and doing the 'rndc reloads' or by ignoring the exception when the list cannot be retrieved from Designate Central.

[Ron Rickard (rjrjr)]

Pool manager and mini-dns addresses this problem.