maas-dns failed to install completely on Trusty

[Jeff Lane ]

Installed maas packages on a fresh trusty install like so:

$ sudo apt-get install maas maas-region-controller maas-cluster-controller maas-dhcp maas-dns

Installation proceeded successfully until maas-dns. maas-dns errored out:

Setting up maas (1.4+bzr1789+dfsg-0ubuntu1) ...
Processing triggers for libc-bin ...
Errors were encountered while processing:
 maas-dns
E: Sub-process /usr/bin/dpkg returned an error code (1)

After apt-get exited, I re-ran the comand and this time the configuration of maas-dns seems to have successfully completed and exited, but there was a failure when trying to restart bind9:

bladernr@critical-maas:~$ sudo apt-get install maas maas-region-controller maas-cluster-controller maas-dns maas-dhcp
[sudo] password for bladernr:
Reading package lists... Done
Building dependency tree
Reading state information... Done
maas is already the newest version.
maas-cluster-controller is already the newest version.
maas-dhcp is already the newest version.
maas-dns is already the newest version.
maas-region-controller is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up maas-dns (1.4+bzr1789+dfsg-0ubuntu1) ...
 * Stopping domain name service... bind9 waiting for pid 22203 to die
                                                                         [ OK ]
 * Starting domain name service... bind9 [fail]
invoke-rc.d: initscript bind9, action "restart" failed.

Looking in the apt term.log, I noticed this traceback the first time I tried installing maas-dns:
Setting up maas-dns (1.4+bzr1789+dfsg-0ubuntu1) ...
Traceback (most recent call last):
  File "/usr/bin/django-admin", line 5, in <module>
    management.execute_from_command_line()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python2.7/dist-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/python2.7/dist-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/dist-packages/maasserver/management/commands/set_up_dns.py", line 54, in handle
    upstream_dns = Config.objects.get_config("upstream_dns")
  File "/usr/lib/python2.7/dist-packages/maasserver/models/config.py", line 93, in get_config
    return self.get(name=name).value
  File "/usr/lib/python2.7/dist-packages/django/db/models/manager.py", line 151, in get
    return self.get_queryset().get(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 301, in get
    num = len(clone)
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 77, in __len__
    self._fetch_all()
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 854, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 710, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 780, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 159, in cursor
    cursor = util.CursorWrapper(self._cursor(), self)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 129, in _cursor
    self.ensure_connection()
  File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 124, in ensure_connection
    self.connect()
  File "/usr/lib/python2.7/dist-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 124, in ensure_connection
    self.connect()
  File "/usr/lib/python2.7/dist-packages/django/db/backends/__init__.py", line 112, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 116, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/lib/python2.7/dist-packages/psycopg2/__init__.py", line 179, in connect
    connection_factory=connection_factory, async=async)
django.db.utils.OperationalError: FATAL: password authentication failed for user "maas"
FATAL: password authentication failed for user "maas"

dpkg: error processing maas-dns (--configure):
 subprocess installed post-installation script returned error exit status 1

and this was term.log for the retry:
Setting up maas (1.4+bzr1789+dfsg-0ubuntu1) ...
Processing triggers for libc-bin ...
Errors were encountered while processing:
 maas-dns
Log ended: 2014-01-07 11:10:08

Log started: 2014-01-07 11:10:39
Setting up maas-dns (1.4+bzr1789+dfsg-0ubuntu1) ...
 * Stopping domain name service... bind9 ESC[80G waiting for pid 22203 to die
^MESC[74G[ OK ]
 * Starting domain name service... bind9 ESC[80G ^MESC[74G[ESC[31mfailESC[39;49m]
invoke-rc.d: initscript bind9, action "restart" failed.
Log ended: 2014-01-07 11:10:42

Additionall, manually attempting to restart bind9 results in the following:
bladernr@critical-maas:~$ sudo service bind9 restart
 * Stopping domain name service... bind9 rndc: connect failed: 127.0.0.1#953: connection refused
                                                                         [ OK ]
 * Starting domain name service... bind9 [fail]

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: maas-dns 1.4+bzr1789+dfsg-0ubuntu1
ProcVersionSignature: Ubuntu 3.12.0-7.15-generic 3.12.4
Uname: Linux 3.12.0-7-generic x86_64
ApportVersion: 2.12.7-0ubuntu3
Architecture: amd64
CurrentDesktop: Unity
Date: Tue Jan 7 11:12:16 2014
InstallationDate: Installed on 2014-01-07 (0 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Alpha amd64 (20140106)
PackageArchitecture: all
SourcePackage: maas
UpgradeStatus: No upgrade log present (probably fresh install)

[Jeff Lane  (bladernr)]

Looked into syslog when manually restarting bind9 and it seems to hang on the maas zone file:
Jan 7 11:28:16 critical-maas named[29464]: starting BIND 9.9.3-rpz2+rl.13214.22-P2-Ubuntu-1:9.9.3.dfsg.P2-4ubuntu2 -u bind
Jan 7 11:28:16 critical-maas named[29464]: built with '--prefix=/usr' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc/bind' '--localstatedir=/var' '--enable-threads' '--enable-largefile' '--with-libtool' '--enable-shared' '--enable-static' '--with-openssl=/usr' '--with-gssapi=/usr' '--with-gnu-ld' '--with-geoip=/usr' '--with-atf=no' '--enable-ipv6' '--enable-filter-aaaa' 'CFLAGS=-fno-strict-aliasing -DDIG_SIGCHASE -O2'
Jan 7 11:28:16 critical-maas named[29464]: ----------------------------------------------------
Jan 7 11:28:16 critical-maas named[29464]: BIND 9 is maintained by Internet Systems Consortium,
Jan 7 11:28:16 critical-maas named[29464]: Inc. (ISC), a non-profit 501(c)(3) public-benefit
Jan 7 11:28:16 critical-maas named[29464]: corporation. Support and training for BIND 9 are
Jan 7 11:28:16 critical-maas named[29464]: available at https://www.isc.org/support
Jan 7 11:28:16 critical-maas named[29464]: ----------------------------------------------------
Jan 7 11:28:16 critical-maas named[29464]: adjusted limit on open files from 4096 to 1048576
Jan 7 11:28:16 critical-maas named[29464]: found 4 CPUs, using 4 worker threads
Jan 7 11:28:16 critical-maas named[29464]: using 4 UDP listeners per interface
Jan 7 11:28:16 critical-maas named[29464]: using up to 4096 sockets
Jan 7 11:28:16 critical-maas named[29464]: loading configuration from '/etc/bind/named.conf'
Jan 7 11:28:16 critical-maas named[29464]: /etc/bind/named.conf.local:9: open: /etc/bind/maas/named.conf.maas: file not found
Jan 7 11:28:16 critical-maas named[29464]: loading configuration: file not found
Jan 7 11:28:16 critical-maas named[29464]: exiting (due to fatal error)

[Kent Baxley (kentb)]

I hit this as well. To get around it (at least for me), I ran:

sudo apt-get purge maas-dns

sudo apt-get install maas-dns

...then maas-dns started working again.

[Serge Hallyn (serge-hallyn)]

Same result for me.

[Jeff Lane  (bladernr)]

 https://bugs.launchpad.net/maas/+bug/1035998/comments/2

Reading that comment, it suggests that the named.conf.maas file should be empty by design.... so I created one like so:

sudo touch /etc/named/maas/named.conf.maas

and restarted bind and apache.

Then, I deleted my node and retried and it successfully commissioned and declared.

From there, I was able to access it via the UI and click "Commission Node" and the node turned on and successfully booted the commissioning image. (It was failing before because DNS was never started, causing the commissioning image to hang)

[Mario Splivalo (mariosplivalo)]

Hi, Jeff.

Is this still the issue? I just tried deploying maas with ubuntu trusty, installing just 'maas' package, which then install everything that was needed (incuding maas-dns). The maas version in trusty is 1.5, and I had no issues with installing.

Then I added maas stable ppa, and upgraded to maas 1.7, also no issues.

To be complete, I trashed my test env, created clean one, added maas stable ppa and installed maas from there - also no issues.

(I did all my tests in trusty lxc container).

I see you were testing with 1.4+bzr1789+dfsg-0ubuntu1, where maas-dns is currenlty on 1.5.

[Jeff Lane  (bladernr)]

On Fri, May 29, 2015 at 7:32 AM, Mario Splivalo
<email address hidden> wrote:
> Hi, Jeff.
>
> Is this still the issue? I just tried deploying maas with ubuntu trusty,
> installing just 'maas' package, which then install everything that was
> needed (incuding maas-dns). The maas version in trusty is 1.5, and I
> had no issues with installing.
>
> Then I added maas stable ppa, and upgraded to maas 1.7, also no issues.
>
> To be complete, I trashed my test env, created clean one, added maas
> stable ppa and installed maas from there - also no issues.
>
> (I did all my tests in trusty lxc container).
>
> I see you were testing with 1.4+bzr1789+dfsg-0ubuntu1, where maas-dns is
> currenlty on 1.5.

Hi Mario,

No, not for some time. Indeed this was a 1.4 issue that was resolved
in 1.5. And to be honest, I don't even bother with anything less than
1.8 these days. Cert instructs all test centers to install from the
maas/stable ppa, so they get 1.8 out of the box.

[Mike Pontillo (mpontillo)]

As Jeff said, I believe this is fixed in MAAS 1.7.

I worked on a related race condition having to do with migrating MAAS-managed BIND variables to the MAAS database.

If you are installing MAAS on a machine which will *not* host the MAAS database, there is still somewhat of a race condition, in that we cannot fully migrate the BIND configuration to the MAAS database unless the database is up and running.

On MAAS 1.7, if you hit this situation, you can run:

sudo maas-region-admin edit_named_options --migrate-conflicting-options

This will rewrite the BIND configuration to ensure that MAAS does not duplicate any BIND options (which would result in the symptom above: BIND failing to start).