rndc_command: Command `rndc -c /etc/bind/maas/rndc.conf.maas reload` returned non-zero exit status 1 after dist-upgrade

[Diogo Matsubara]

I was testing a MAAS upgrade from cloud-tools (1.4+bzr1693+dfsg-0ubuntu2.3~ctools0) to cloud-tools-next (1.5+bzr2252-0ubuntu1~ctools0) and after the upgrade I got the following traceback in /var/log/maas/celery-region.log

[2014-06-06 17:21:03,047: ERROR/PoolWorker-3] rndc_command: Command `rndc -c /etc/bind/maas/rndc.conf.maas reload` returned non-zero exit status 1:
None
[2014-06-06 17:21:03,050: ERROR/MainProcess] Task provisioningserver.tasks.rndc_command[05daa257-720e-4397-b9fc-0000613a02e2] raised exception: UnpickleableExceptionWrapper('provisioningserver.utils', 'ExternalProcessError', (), 'ExternalProcessError()')
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/execute/trace.py", line 47, in trace
    return cls(states.SUCCESS, retval=fun(*args, **kwargs))
  File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 247, in __call__
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/app/__init__.py", line 175, in run
    return fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/tasks.py", line 93, in wrapper
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/tasks.py", line 216, in rndc_command
    exc=exc, countdown=RNDC_COMMAND_RETRY_DELAY)
  File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 535, in retry
    self.name, options["task_id"], args, kwargs))
ExternalProcessError: ExternalProcessError()
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/execute/trace.py", line 47, in trace
    return cls(states.SUCCESS, retval=fun(*args, **kwargs))
  File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 247, in __call__
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/app/__init__.py", line 175, in run
    return fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/tasks.py", line 93, in wrapper
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/tasks.py", line 216, in rndc_command
    exc=exc, countdown=RNDC_COMMAND_RETRY_DELAY)
  File "/usr/lib/python2.7/dist-packages/celery/app/task/__init__.py", line 535, in retry
    self.name, options["task_id"], args, kwargs))
ExternalProcessError: ExternalProcessError()

My setup was a bit odd because I had a custom /etc/bind/named.conf.options file with the DNS forwarder. Now on version 1.5 I can set that forwarder through MAAS itself and once I did it I got this in the same log file:

[2014-06-06 18:05:18,434: INFO/MainProcess] Got task from broker: provisioningserver.tasks.write_full_dns_config[fe25e1c5-70f1-42bf-a4fb-8574d26c3f0c]
[2014-06-06 18:05:18,555: INFO/MainProcess] Got task from broker: provisioningserver.tasks.rndc_command[f8692e0d-d9ec-4659-9bf7-2486789b1340]

So it seems things are now working once I set up the upstream_dns option through MAAS. I'm logging this here anyway in case others run into a similar problem. The rndc_command failure doesn't seem related to the previous config I had and it's possible that when I changed the upstream_dns config, MAAS triggered a new task that made the error go away, but I'm not sure.

[Julian Edwards (julian-edwards)]

I think what happened here is that because you already had the options file, when maas tried to configured a forwarders option it clashed with your existing one.

Have you still got the bind logs from this startup to prove this?

[Diogo Matsubara (matsubara)]

I don't have the logs from the specific instance when I reported but I did a similar upgrade today and got the following logs:

/var/log/maas/celery-region.log
/var/log/syslog
/var/log/apt/term.log
/var/log/apt/history.log

I don't have any specific log for bind in /var/log, since it seems to log to syslog. You probably want to look at around 21:57 when the dist-upgrade started. You can see in celery-region.log the same error described originally in this bug report. It seems to be fairly easy to reproduce the error in the MAAS lab.

[Julian Edwards (julian-edwards)]

So are you saying that if the upstream_dns config is not set, it will have this problem on an upgrade? Does it still do it without your custom blob of config?

[Julian Edwards (julian-edwards)]

Nobody else has seen this, so downgrading for now