Spurious test failure: rndc hits “address in use” error

[Jeroen T. Vermeulen]

Thanks to improved logging, we now get decent output when rndc fails during test runs. As it turns out, the write_dns_config() task restarts the name server but finds the server's port in use:

warning: /tmp/tmp7Xz17g/named.conf.rndc.maas:7: couldn't add command channel 127.0.0.1#56314: address in use

Abridged callback:

src/maasserver/tests/test_dns.py, line 370, in test_changing_interface_management_updates_DNS_zone
    management=NODEGROUPINTERFACE_MANAGEMENT.DHCP_AND_DNS)
src/maasserver/testing/factory.py, line 260, in make_node_group
    maas_url=maas_url, **interface_settings)

…. (save() call chain leading to post-save signal) ….

src/maasserver/dns_connect.py, line 52, in dns_post_save_NodeGroupInterface
    add_zone(instance.nodegroup)
src/maasserver/dns.py, line 284, in add_zone
    zones=zones_to_write, callback=write_dns_config_subtask)

…. (celery ‘delay’ call chain) …

src/provisioningserver/tasks.py, line 92, in wrapper
    func(*args, **kwargs)
src/provisioningserver/tasks.py, line 278, in write_dns_zone_config
    callback.delay()

…. (celery ‘delay’ call chain) …

src/provisioningserver/tasks.py, line 92, in wrapper
    func(*args, **kwargs)
src/provisioningserver/tasks.py, line 261, in write_dns_config
    callback.delay()

…. (celery ‘delay’ call chain) …

src/provisioningserver/tasks.py, line 92, in wrapper
    func(*args, **kwargs)
src/provisioningserver/tasks.py, line 211, in rndc_command
    execute_rndc_command(arguments)
src/provisioningserver/dns/config.py, line 153, in execute_rndc_command
    call_and_check(rndc_cmd, stdout=devnull)
src/provisioningserver/utils.py, line 121, in call_and_check
    return subprocess.check_call(command, *args, **kwargs)

ExternalProcessError: Command `rndc -c /tmp/tmp7Xz17g/rndc.conf.maas reconfig` returned non-zero exit status 1:
None

Download link for full test output: https://code.launchpad.net/~blake-rouse/maas/sm15k-restapi-v2/+merge/211136/comments/498825