Comment 6 for bug 1599171

Revision history for this message
Jeff Lane  (bladernr) wrote : Re: [Bug 1599171] Re: curtin files are not being updated when the package updates, leading to stale config files that break deployments

On Thu, Jul 7, 2016 at 9:48 AM, Rod Smith <email address hidden> wrote:
> The problem with having the packaging install the preseed files is
> threefold:
>
> * The files are owned by maas-region-api, so replacing them via packaging
> is likely to create a conflict. (That said, I'm not sure if dpkg/APT has
> some way to handle such issues, so there may be an official way to handle
> such issues.)

How so? I mean, beyond the potential we already have there when
running maniacs-setup. There's no new behaviour here, just moving
behaviour from one place to another. And following your own comments
below, if the suggested path is:

upgrade/install m-c-s
run maniacs-setup which replaces them

how is that different than:
upgrade/install m-c-s which replaces them
run maniacs-setup which finalizes config?

I could see a concern if the files are updated in MAAS itself,
however, that seems to be a very low chance as those files have
remained more or less the same since maas 1.5. At that point, simply
copying the files again from /usr/share/maas-cert-server/preseeds
would fix the issue (granted currently that would be a manual step).

And in any case, we don't suggest re-running it after a MAAS upgrade
(that doesn't also include an MCS upgrade) that I'm aware of.

> * The preseed files may be modified by the maniacs-setup script to point to
> the MAAS server rather than to ppa.launchpad.net, but only if the user has
> configured the MAAS server as a local mirror. Currently, maniacs-setup
> requires user interaction to verify this detail. This could be
> streamlined to not require user interaction, but care will need to be
> taken to not break something else in the process.

And this still exists, and still has the manual check. If you run
maniacs-setup, it still does the following:
if [ $UPDATE_PRESEEDS == 1 ] ; then
        if [ $MIRRORED = 0 ] && [ -d "/srv/mirror/ppa.launchpad.net" ] ; then
            get_yn "* Configure preseeds to use local APT archives" "Y"
            if [ "$YN" == "Y" ] ; then
                MIRRORED=1
            fi
        fi

which triggers the substitution in the preseed files. That all still
works when you run maniacs setup, regardless of which point in time
the preseeds are copied over.

Additionally, what I have set up doesn't overwrite any files. It
preserves any current copy with a timestamped file name so you can
easily roll back.

That said, I'm also looking at a way to fix that so that dpkg/apt
actually prompt before it replaces the preseeds, which should
automatically create backups rather than manually doing so. I'll look
at that today.

> * If "maniacs-setup -p" were to be run as part of the post-install script
> for the maas-cert-server package, it would currently break on initial
> install because maniacs-setup would not have been run the first time,
> which is an effective requirement for running it with "-p". This could
> be handled by having the script plant a flag somewhere when it's done
> its initial run and having the post-install script, or maniacs-setup
> itself, look for this flag and refuse to run with options if it's not
> found.

That's not a bad idea. Currently it does not run -p during postinst,
but it does prompt the user with the following:
echo "***********************************************************************"
echo
echo " Installation complete. Now run /usr/sbin/maniacs-setup if necessary"
echo " to complete or update the configuration of your new Certification"
echo " MAAS Server."
echo " NOTE: If you have previously configured a local apt mirror, you will"
echo " need to run 'sudo maniacs-setup --update-preseeds' to ensure your"
echo " nodes will be deployed with the correct archive data."
echo
echo " Usage: /usr/sbin/maniacs-setup [ --download-virtualization-image ]"
echo " [ --import-boot-resources ] [ --mirror-archives ]"
echo " [ --update-point-releases ] [ --update-preseeds ]"
echo
echo "***********************************************************************"

>
> The philosophy behind the current approach is that the preseed files are
> configuration files, and it's therefore legitimate for users to modify
> them. The maniacs-setup script helps automate that modification (and
> lots more). Running "maniacs-setup -p" in the maas-cert-server package's
> post-install script could in principle be done, but doing so safely and
> in a way that won't cause further breakage could be tricky, which is why
> I've never coded it to do so to date.

Yes, and that's why I didn't have postinst call it either, but rather
left a prompt to run it manually after any install or upgrade of MCS.
That said, as noted above, it may be even better to classify the
preseeds as config files so that dpkg automatically prompts the user
to choose whether to preserve, overwrite, or whatever. AIUI that
functionality is built in to dpkg and all you have to do to call it is
establish that a given file is a conffile.

> When we've released updated preseed files, we've sent out e-mails urging
> people to run "maniacs-setup -p" to install the updated files. In the
> long term, improvements to the way all this is handled are desirable,
> but this specific issue is part of a much bigger picture. (There have
> been suggestions to enable maniacs-setup to operate completely non-
> interactively, for instance, and to have it do a better job of network
> discovery.) IMHO, it's better to tackle all of these issues
> simultaneously, since they interact with each other, than to try to fix
> them piecemeal.

Fair enough, though of the three concerns you have, I think only one
really needs much more investigation and action, but I could be wrong,
so grain of salt and all that. FWIW, I have tested this on about 8
different VMs with both Trusty (1.7 upgrading to 1.9, and 1.9 fresh
install) and Xenial (2.0 fresh install) both with the new version as
the fresh install and upgrading the new version and didn't run into
issues (beyond the aforementioned replacement of modified preseed
files, but as I said, those were handled by making timestamped backups
so facilitate rolling them back).

I'll keep looking at the conf file angle, that's something I started
thinking about last night and looking into this morning.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1599171
>
> Title:
> curtin files are not being updated when the package updates, leading
> to stale config files that break deployments
>
> Status in maas-cert-server:
> In Progress
>
> Bug description:
> It seems that it's possible to somehow end up with stale curtin files
> for some reason. This was found on a failing Partner system recently.
>
> Essentially, they were unable to deploy Xenial due to broken curtin
> files.
>
> NOW< it's possible they just hadn't updated, HOWEVER, it's entirely
> possible also that apt didn't update them as they're considered config
> files.
>
> We need to investiate the following scenario:
>
> Older version w/ older curtin files updated to latest version: are curtin files automatically updated?
> Older version with manually modified curtin files updated to latest version: are files automatically updated?
>
> If not, it may be worth while doing one of two things:
>
> add an "Update" to maniacs-setup so the following happens:
> maniacs-setup -u checks the two versions (Installed and Candidate via 'apt-cache policy') and if they are wrong, it updates m-c-s automatically.
> then runs -p to ensure the curtin files are updated
>
> Another option to look at (and maybe do before messing with maniacs)
> is to see if there's a way to ensure/force the curtin files to update
> when apt updates m-c-s
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas-cert-server/+bug/1599171/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas-cert-server; status=In Progress; importance=Medium; <email address hidden>;
> Launchpad-Bug-Tags: hwcert-server
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: bladernr rodsmith
> Launchpad-Bug-Reporter: Jeff Lane (bladernr)
> Launchpad-Bug-Modifier: Rod Smith (rodsmith)
> Launchpad-Message-Rationale: Subscriber
> Launchpad-Message-For: bladernr

--
"Entropy isn't what it used to be."

Jeff Lane -
Server Certification Lead, Warrior Poet, Biker, Lover of Pie
Phone: 919-442-8649
Ubuntu Ham: W4KDH Freenode IRC: bladernr or bladernr_
gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417 C466 4ABD 3635 3A14 B2DD