curtin files are not being updated when the package updates, leading to stale config files that break deployments

[Jeff Lane ]

It seems that it's possible to somehow end up with stale curtin files for some reason. This was found on a failing Partner system recently.

Essentially, they were unable to deploy Xenial due to broken curtin files.

NOW< it's possible they just hadn't updated, HOWEVER, it's entirely possible also that apt didn't update them as they're considered config files.

We need to investiate the following scenario:

Older version w/ older curtin files updated to latest version: are curtin files automatically updated?
Older version with manually modified curtin files updated to latest version: are files automatically updated?

If not, it may be worth while doing one of two things:

add an "Update" to maniacs-setup so the following happens:
maniacs-setup -u checks the two versions (Installed and Candidate via 'apt-cache policy') and if they are wrong, it updates m-c-s automatically.
then runs -p to ensure the curtin files are updated

Another option to look at (and maybe do before messing with maniacs) is to see if there's a way to ensure/force the curtin files to update when apt updates m-c-s

[Jeff Lane  (bladernr)]

Yeah, so this looks more like a procedural problem...

To replicate this I did the following:

Fresh trusty install in a VM
in the VM:
* add the hardware-cert PPA only
* modify the cert ppa .list to change trusty to utopic (to get an older version of m-c-s)
* sudo apt-get update
* sudo apt-get maas-cert-server

eventually got it all installed, or at least what I needed.
Checked /etc/maas/preseeds and found that curtin_userdata and curtin_userdata_custom were default
ran maniacs-setup
* had to move the preseed bit to the front, as this older m-c-s version breaks here due to some DNS problem that isn't germane to this.
* verified that maniacs setup DID create the -orig files and copied our versions to /etc/maas/preseed

So now, I have trusty, with maas, and with our preseeds via m-c-s (version 2.1.0 utopic)
Next, I corrected /etc/maas/source.list.d/hardware-certification.list and changed utopic back to trusty.

I then also added the maas/stable ppa to also grab the latest MAAS.

I did an update and upgrade and got the latest maas (1.9.3) and the latest m-c-s (2.20 trusty)

Then I rechecked and noted that the preseeds are not updated.

This only occurs manually.

[Jeff Lane  (bladernr)]

Steps needed:

need to disconnect the preseed bit from m-c-s and move it into packaging so that on installation of m-c-s, the preseeds are installed at install time. Then every update will give updated preseeds.

Potential issues:
user-modified curtin files
** shouldn't be an issue, as this is not geared to production systems, but to specificlly configured test environments

Loss of originals
** packaging should insure that the originals are copied to -orig or something like that.

[Jeff Lane  (bladernr)]

Another option would be to have maniacs-setup -u executed after the package is configured. (or during configuration?)

Problem there is that we don't want that to happen on the first time.

maniacs-setup is for configuring things. packaging is for installing things in the package. Since the curtin files are in the package and being installed... they should be installed by the packaging, not an external script.

[Jeff Lane  (bladernr)]

Fixed with attached code changes. This was tested on the following scenarios:

Fresh install of this version of MCS on both Trusty and Xenial
Upgrade from 0.2.20 to this version of MCS on both Trusty and Xenial

In all tests, the files were copied and preserved as expected.

[Rod Smith (rodsmith)]

The problem with having the packaging install the preseed files is threefold:

* The files are owned by maas-region-api, so replacing them via packaging
  is likely to create a conflict. (That said, I'm not sure if dpkg/APT has
  some way to handle such issues, so there may be an official way to handle
  such issues.)
* The preseed files may be modified by the maniacs-setup script to point to
  the MAAS server rather than to ppa.launchpad.net, but only if the user has
  configured the MAAS server as a local mirror. Currently, maniacs-setup
  requires user interaction to verify this detail. This could be
  streamlined to not require user interaction, but care will need to be
  taken to not break something else in the process.
* If "maniacs-setup -p" were to be run as part of the post-install script
  for the maas-cert-server package, it would currently break on initial
  install because maniacs-setup would not have been run the first time,
  which is an effective requirement for running it with "-p". This could
  be handled by having the script plant a flag somewhere when it's done
  its initial run and having the post-install script, or maniacs-setup
  itself, look for this flag and refuse to run with options if it's not
  found.

The philosophy behind the current approach is that the preseed files are configuration files, and it's therefore legitimate for users to modify them. The maniacs-setup script helps automate that modification (and lots more). Running "maniacs-setup -p" in the maas-cert-server package's post-install script could in principle be done, but doing so safely and in a way that won't cause further breakage could be tricky, which is why I've never coded it to do so to date.

When we've released updated preseed files, we've sent out e-mails urging people to run "maniacs-setup -p" to install the updated files. In the long term, improvements to the way all this is handled are desirable, but this specific issue is part of a much bigger picture. (There have been suggestions to enable maniacs-setup to operate completely non-interactively, for instance, and to have it do a better job of network discovery.) IMHO, it's better to tackle all of these issues simultaneously, since they interact with each other, than to try to fix them piecemeal.

[Jeff Lane  (bladernr)]

On Thu, Jul 7, 2016 at 9:48 AM, Rod Smith <email address hidden> wrote:
> The problem with having the packaging install the preseed files is
> threefold:
>
> * The files are owned by maas-region-api, so replacing them via packaging
> is likely to create a conflict. (That said, I'm not sure if dpkg/APT has
> some way to handle such issues, so there may be an official way to handle
> such issues.)

How so? I mean, beyond the potential we already have there when
running maniacs-setup. There's no new behaviour here, just moving
behaviour from one place to another. And following your own comments
below, if the suggested path is:

upgrade/install m-c-s
run maniacs-setup which replaces them

how is that different than:
upgrade/install m-c-s which replaces them
run maniacs-setup which finalizes config?

I could see a concern if the files are updated in MAAS itself,
however, that seems to be a very low chance as those files have
remained more or less the same since maas 1.5. At that point, simply
copying the files again from /usr/share/maas-cert-server/preseeds
would fix the issue (granted currently that would be a manual step).

And in any case, we don't suggest re-running it after a MAAS upgrade
(that doesn't also include an MCS upgrade) that I'm aware of.

> * The preseed files may be modified by the maniacs-setup script to point to
> the MAAS server rather than to ppa.launchpad.net, but only if the user has
> configured the MAAS server as a local mirror. Currently, maniacs-setup
> requires user interaction to verify this detail. This could be
> streamlined to not require user interaction, but care will need to be
> taken to not break something else in the process.

And this still exists, and still has the manual check. If you run
maniacs-setup, it still does the following:
if [ $UPDATE_PRESEEDS == 1 ] ; then
        if [ $MIRRORED = 0 ] && [ -d "/srv/mirror/ppa.launchpad.net" ] ; then
            get_yn "* Configure preseeds to use local APT archives" "Y"
            if [ "$YN" == "Y" ] ; then
                MIRRORED=1
            fi
        fi

which triggers the substitution in the preseed files. That all still
works when you run maniacs setup, regardless of which point in time
the preseeds are copied over.

Additionally, what I have set up doesn't overwrite any files. It
preserves any current copy with a timestamped file name so you can
easily roll back.

That said, I'm also looking at a way to fix that so that dpkg/apt
actually prompt before it replaces the preseeds, which should
automatically create backups rather than manually doing so. I'll look
at that today.

> * If "maniacs-setup -p" were to be run as part of the post-install script
> for the maas-cert-server package, it would currently break on initial
> install because maniacs-setup would not have been run the first time,
> which is an effective requirement for running it with "-p". This could
> be handled by having the script plant a flag somewhere when it's done
> its initial run and having the post-install script, or maniacs-setup
> itself, look for this flag and refuse to run with options if it's not
> found.

That's not a bad idea. Curren...

[Rod Smith (rodsmith)]

Concerning file ownership, my comment was intended to refer to the possibility of putting the files in our package under /etc/maas/preseeds/curtin_userdata (etc.). Keeping them where they've traditionally been in our package and then copying and modifying them via a post-install script avoids the problem of conflicting package "ownership" of the files.

Your patch seems to have worked around the remaining two issues -- maybe they weren't as deep as I'd originally thought, or maybe you're just brilliant, Jeff. ;-) Still, we may need to be alert to possible new problems, should a corner case crop up that's not been tested.

Also, and more importantly, I think we'll still need some ongoing maintenance and updates to the maniacs-setup script and to the maas-cert-server package that may interact with this issue, as noted in my earlier reply.