Intermittent test failures in TestDNSConfigModicfications: "dig" returns 9

[Jeroen T. Vermeulen]

This failure can affect different tests in TestDNSConfigModifications. Going by the source code for "dig," a return value of 9 happens when connection to the DNS server either hits an error of any kind, or times out. The actual error is printed to stdout, not to stderr, which may explain why the test failures don't seem to say much about the failure.

An example follows:

======================================================================
ERROR: maasserver.tests.test_dns.TestDNSConfigModifications.test_delete_node_updates_zone
----------------------------------------------------------------------
_StringException: named.log: {{{
info: additionally listening on IPv4 interface lo, 127.0.0.1#53313
error: could not listen on UDP socket: address in use
error: creating IPv4 interface lo failed; interface ignored
info: managed-keys-zone: loaded serial 0
notice: all zones loaded
notice: running
info: received control channel command 'reload'
info: loading configuration from '/tmp/tmp1BZlap/named.conf'
info: reading built-in trusted keys from file '/etc/bind/bind.keys'
info: using default UDP/IPv4 port range: [1024, 65535]
info: using default UDP/IPv6 port range: [1024, 65535]
info: listening on IPv4 interface lo, 127.0.0.1#53313
error: could not listen on UDP socket: address in use
error: creating IPv4 interface lo failed; interface ignored
warning: not listening on any interfaces
info: sizing zone task pool based on 0 zones
notice: command channel listening on 127.0.0.1#55782
info: additionally listening on IPv4 interface lo, 127.0.0.1#53313
error: could not listen on UDP socket: address in use
error: creating IPv4 interface lo failed; interface ignored
info: reloading configuration succeeded
info: reloading zones succeeded
notice: all zones loaded
notice: running
info: received control channel command 'reload'
info: loading configuration from '/tmp/tmp1BZlap/named.conf'
info: reading built-in trusted keys from file '/etc/bind/bind.keys'
info: using default UDP/IPv4 port range: [1024, 65535]
info: using default UDP/IPv6 port range: [1024, 65535]
info: listening on IPv4 interface lo, 127.0.0.1#53313
error: could not listen on UDP socket: address in use
error: creating IPv4 interface lo failed; interface ignored
warning: not listening on any interfaces
info: sizing zone task pool based on 2 zones
info: additionally listening on IPv4 interface lo, 127.0.0.1#53313
error: could not listen on UDP socket: address in use
error: creating IPv4 interface lo failed; interface ignored
info: reloading configuration succeeded
info: reloading zones succeeded
info: zone 0.168.192.in-addr.arpa/IN: loaded serial 25
info: zone nodegroup-ULhlsQ/IN: loaded serial 25
notice: all zones loaded
notice: running
info: received control channel command 'reload nodegroup-ULhlsQ'
info: zone nodegroup-ULhlsQ/IN: loaded serial 26
info: received control channel command 'reload 0.168.192.in-addr.arpa'
info: zone 0.168.192.in-addr.arpa/IN: loaded serial 26
info: received control channel command 'reload nodegroup-ULhlsQ'
info: zone nodegroup-ULhlsQ/IN: loaded serial 27
info: received control channel command 'reload 0.168.192.in-addr.arpa'
info: zone 0.168.192.in-addr.arpa/IN: loaded serial 27
}}}

Traceback (most recent call last):
  File "/home/jtv/launchpad/maas/maas/src/maasserver/tests/test_dns.py", line 399, in test_delete_node_updates_zone
    self.assertEqual([''], self.dig_resolve(fqdn))
  File "/home/jtv/launchpad/maas/maas/src/maasserver/tests/test_dns.py", line 217, in dig_resolve
    commands=[fqdn, '+short']).split('\n')
  File "/home/jtv/launchpad/maas/maas/src/provisioningserver/testing/tests/test_bindfixture.py", line 60, in dig_call
    return check_output(cmd).strip()
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '[u'dig', u'+time=10', u'+tries=5', u'@127.0.0.1', u'-p', u'53313', u'nNyu7Bu7X3Uq6vT1Xini.nodegroup-ULhlsQ', u'+short']' returned non-zero exit status 9

.

.

Another example:

======================================================================
ERROR: maasserver.tests.test_dns.TestDNSConfigModifications.test_changing_interface_management_updates_DNS_zone
----------------------------------------------------------------------
_StringException: named.log: {{{
info: additionally listening on IPv4 interface lo, 127.0.0.1#58739
error: could not listen on UDP socket: address in use
error: creating IPv4 interface lo failed; interface ignored
info: managed-keys-zone: loaded serial 0
notice: all zones loaded
notice: running
info: received control channel command 'reload'
info: loading configuration from '/tmp/tmpLkvbJb/named.conf'
info: reading built-in trusted keys from file '/etc/bind/bind.keys'
info: using default UDP/IPv4 port range: [1024, 65535]
info: using default UDP/IPv6 port range: [1024, 65535]
info: listening on IPv4 interface lo, 127.0.0.1#58739
error: could not listen on UDP socket: address in use
error: creating IPv4 interface lo failed; interface ignored
warning: not listening on any interfaces
info: sizing zone task pool based on 0 zones
notice: command channel listening on 127.0.0.1#44984
info: additionally listening on IPv4 interface lo, 127.0.0.1#58739
error: could not listen on UDP socket: address in use
error: creating IPv4 interface lo failed; interface ignored
info: reloading configuration succeeded
info: reloading zones succeeded
notice: all zones loaded
notice: running
info: received control channel command 'reload'
info: loading configuration from '/tmp/tmpLkvbJb/named.conf'
info: reading built-in trusted keys from file '/etc/bind/bind.keys'
info: using default UDP/IPv4 port range: [1024, 65535]
info: using default UDP/IPv6 port range: [1024, 65535]
info: listening on IPv4 interface lo, 127.0.0.1#58739
error: could not listen on UDP socket: address in use
error: creating IPv4 interface lo failed; interface ignored
warning: not listening on any interfaces
info: sizing zone task pool based on 2 zones
info: additionally listening on IPv4 interface lo, 127.0.0.1#58739
error: could not listen on UDP socket: address in use
error: creating IPv4 interface lo failed; interface ignored
info: reloading configuration succeeded
info: reloading zones succeeded
info: zone 7.168.192.in-addr.arpa/IN: loaded serial 22
info: zone nodegroup-8oQukC/IN: loaded serial 22
notice: all zones loaded
notice: running
info: received control channel command 'reload'
info: loading configuration from '/tmp/tmpLkvbJb/named.conf'
info: reading built-in trusted keys from file '/etc/bind/bind.keys'
info: using default UDP/IPv4 port range: [1024, 65535]
info: using default UDP/IPv6 port range: [1024, 65535]
info: listening on IPv4 interface lo, 127.0.0.1#58739
error: could not listen on UDP socket: address in use
error: creating IPv4 interface lo failed; interface ignored
warning: not listening on any interfaces
info: sizing zone task pool based on 0 zones
info: zone 7.168.192.in-addr.arpa/IN: (master) removed
info: zone nodegroup-8oQukC/IN: (master) removed
info: additionally listening on IPv4 interface lo, 127.0.0.1#58739
error: could not listen on UDP socket: address in use
error: creating IPv4 interface lo failed; interface ignored
info: reloading configuration succeeded
info: reloading zones succeeded
notice: all zones loaded
notice: running
}}}

Traceback (most recent call last):
  File "/tmp/tarmac/branch.WHiXIj/src/maasserver/tests/test_dns.py", line 373, in test_changing_interface_management_updates_DNS_zone
    self.assertEqual([''], self.dig_resolve(generated_hostname(ip)))
  File "/tmp/tarmac/branch.WHiXIj/src/maasserver/tests/test_dns.py", line 217, in dig_resolve
    commands=[fqdn, '+short']).split('\n')
  File "/tmp/tarmac/branch.WHiXIj/src/provisioningserver/testing/tests/test_bindfixture.py", line 60, in dig_call
    return check_output(cmd).strip()
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '[u'dig', u'+time=10', u'+tries=5', u'@127.0.0.1', u'-p', u'58739', u'192-168-7-76', u'+short']' returned non-zero exit status 9