[SRU] upgrade of MAAS removes local config for bind and breaks DNS

[James Troup]

On an Ubuntu 14.04 host, I recently upgraded MAAS from
1.5.2+bzr2282-0ubuntu0.2 to 1.5.4+bzr2294-0ubuntu1.2. This is a
production environment which uses MAAS to control both DNS and DHCP.

During the upgrade MAAS silently (without prompting) rewrote
/etc/bind/named.conf.options. However, in the process, it dropped
some of our local config and broke DNS resolution for all clients in
this environment. Specifically, it removed and did not re-add a
forwarders entry.

I've attached 3 files: the original named.conf.options, the version
left after the MAAS upgrade and an artifical diff between the two to
show what's actually changed (i.e. ignoring MAAS rearranging things,
messing up whitespace and removing comments).

[Impact]
In 1.7+, MAAS started managing DNS forwarders as part of its DNS management feature. However, if a user in 1.5 would manually modify their configuration file to add forwaders, after an upgrade to 1.7+, DNS would en up broken. This adds the ability for MAAS to upgrade the DNS configuration correctly and migrate the manually configuration to the MAAS managed configuration.

[Test Case]

Without the fix:
1. Install maas 1.5
2. Configure MAAS to manage DNS/DHCP
3. Manually edit /etc/bind/named.conf.options and add forwaders, allow-cache, allow-recursion and allow-cache-query options.
4. Upgrade to MAAS 1.7 with proposed fix.
5. MAAS will migrate the DNS forwarders config into MAAS' DB.
6. MAAS won't write allow-cache, allow-recursion, allow-cache-query.

[Regression Potential]
Minimal. This has been tested extensively in various environments, both on fresh installs and upgrades.

[Raphaël Badin (rvb)]

The culprit is the fix for https://bugs.launchpad.net/maas/+bug/1275649.

It creates a `edit_named_options` Django command that is run in postinst: that command is there to edit the named.conf.options file so that it includes the maas forwarders config. This command is supposed to parse the existing named.conf.options file and only add the required stanza but I suppose that this didn't work in your case.

[Raphaël Badin (rvb)]

> This command is supposed to parse the existing named.conf.options file and only add the required stanza but I suppose that this
> didn't work in your case.

Well, more precisely, it seems the `edit_named_options` assumes it can clobber the 'forwarders' option.

[Tom Haddon (mthaddon)]

Confirmed this is still an issue going from 1.5.4+bzr2294-0ubuntu1.3 to 1.7.5+bzr3369-0ubuntu1~14.04.1 (in trusty-proposed).

[Christian Reis (kiko)]

Confirming, though if this looks too risky for 1.7 we may just have to release note it.

[Barry Price (barryprice)]

These is a second issue here - the upgrades also adds the include mentioned in James's attachment:

include "/etc/bind/maas/named.conf.options.inside.maas";

This file, in my case, contained three lines:

allow-query { any; };
allow-recursion { trusted; };
allow-query-cache { trusted; };

Since our original config already set values for both allow-query and allow-query-cache, bind9 refused to start post-upgrade until I either commented out the duplicate line in the include file, or in our own config.

[James Troup (elmo)]

This just broke ProdStack 4½ and all production services in it (including jujucharms, git.launchpad.net, SSO etc.).

[Mike Pontillo (mpontillo)]

What I'd like to do is address the issue with the forwarders and DNSSEC issues first, since the MAAS configuration already contains options to set these. (at least it does in 1.8; in 1.7, only forwarders are allowed to be set) My plan is to ensure that these values are properly migrated from the BIND config to MAAS settings, rather than being silently deleted.

Barry, may I ask what values you had in the original config for allow-query, allow-recursion, and allow-query-cache? (were they set to something other than what MAAS set them to? If so, we may need a separate bug to address this. If not, it might be safe to simply delete these values from the original config - like we do with forwarders and DNSSEC - so that BIND doesn't crash on startup.) We'll need to study these options to see what the pitfalls are here.

[Mike Pontillo (mpontillo)]

By the way, we're also planning on adding comments regarding the pitfalls of changing the now-MAAS-managed named.conf.options to the header, before we write out the generated configuration.

[Mike Pontillo (mpontillo)]

We just landed a series of changes (which should make it to 1.7.6 and 1.8.1) to:

 * Ignore the "allow-query", "allow-recursion", and "allow-query-cache" options when writing the MAAS DNS options file, if they are already present within named.conf.options.

 * Migrate the "forwarders" and "dnssec-validation" options from the user's configuration to the MAAS database, if they are present within named.conf.options during an upgrade or install of maas-region-controller. (note that if maas-region-controller-min is being used, the command to migrate these options may need to run by manually, since packaging does not assume that a database is on the same machine as the regiond in that case; that is, "maas-region-admin edit_named_options --migrate-conflicting-options".)

Sadly, we don't preserve the original comments inside named.conf.options, but that is a significant amount of additional effort.

Hopefully this will resolve the issue for everyone.

[Launchpad Janitor (janitor)]

This bug was fixed in the package maas - 1.7.6+bzr3376-0ubuntu1

---------------
maas (1.7.6+bzr3376-0ubuntu1) wily; urgency=medium

  * New upstream release 1.7.6 bzr3376:
    - Accept list of forwarders for upstream_dns rather than just
      one. (LP: #1470585)
    - Fix upgrade issue where it would remove custom DNS config,
      potentially breaking DNS. (LP: #1413388)

  [ Raphaël Badin ]
  * Drop dependency on python-iscpy: the code has been integrated into
    MAAS. (LP: #1413388).

  [ Andres Rodriguez ]
  * Refactor maas-dns upgrade code so it doesn't break local DNS config
    and it gets migrated (LP: #1413388)

 -- Andres Rodriguez <email address hidden> Fri, 03 Jul 2015 00:11:50 -0400

[Adam Conrad (adconrad)]

Hello James, or anyone else affected,

Accepted maas into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/maas/1.7.6+bzr3376-0ubuntu2~14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adam Conrad (adconrad)]

Hello James, or anyone else affected,

Accepted maas into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/maas/1.7.6+bzr3376-0ubuntu2~14.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adam Conrad (adconrad)]

Hello James, or anyone else affected,

Accepted maas into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/maas/1.7.6+bzr3376-0ubuntu2~15.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Mike Pontillo (mpontillo)]

I tested by installing bind9 on Trusty, then installing maas 1.7.6+bzr3376-0ubuntu2~14.04.1.

I verified that the forwarders were properly migrated to the MAAS configuration, as long as at least one cluster interface was set to be managing DNS.

The forwarders are migrated into the MAAS database, and values such as allow-query, allow-recursion, and allow-query-cache (which MAAS had previously always written into /etc/bind/maas/named.conf.options.inside.maas) are now only written if the values are not already present in /etc/bind/named.conf.options.

Hopefully this solves the issue for everyone.

[Andres Rodriguez (andreserl)]

I've tested this in trusty, utopic and vivid. Confirm it works as expected as per Mike above.

[Launchpad Janitor (janitor)]

This bug was fixed in the package maas - 1.7.6+bzr3376-0ubuntu2~14.04.1

---------------
maas (1.7.6+bzr3376-0ubuntu2~14.04.1) trusty; urgency=medium

  * debian/control: Make maas-dns a Dependy of maas-region-controller.
  * debian/maas-region-controller.postinst: Ensure DNS config migration is
    always run. (LP: #1413388)

 -- Andres Rodriguez <email address hidden> Fri, 10 Jul 2015 13:47:40 -0400

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for maas has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package maas - 1.7.6+bzr3376-0ubuntu2~15.04.1

---------------
maas (1.7.6+bzr3376-0ubuntu2~15.04.1) vivid; urgency=medium

  * debian/control: Make maas-dns a Dependy of maas-region-controller.
  * debian/maas-region-controller.postinst: Ensure DNS config migration is
    always run. (LP: #1413388)

 -- Andres Rodriguez <email address hidden> Fri, 10 Jul 2015 13:47:40 -0400

[David Lawson (deej)]

When upgrading from 1.5 to 1.7.6+bzr3376-0ubuntu2~14.04.1 using maas-region-controller-min it's not clear when we should be running "maas-region-admin edit_named_options --migrate-conflicting-options", should this be prior to the upgrade? As it was, I had to edit named.conf.options and remove the forwarders option manually, then reload rndc, running the migrate-conflicting-options command post-upgrade resulted in rndc failing to reload because forwarders was defined in multiple places.

[Junien F (axino)]

Also : would running "maas-region-admin edit_named_options --migrate-conflicting-options" without "forwarders" in named.conf.options remove the forwarder setting from the MAAS database ?

Thanks