RELNOTE: upgrade from 1.5.4 to 1.7.0 lost private image stream

[Jeff Lane ]

I had a maas server running 1.5.4 from trusty. I added the experiemental PPA and installed the trusty build of 1.7.0 just now. On first login to the dashboard, I was met with the following warnings:

Boot image import process not started. Nodes will not be able to provision without boot images. Visit the boot images page to start the import.
Third party drivers may be used when booting or installing nodes. These may be proprietary and closed-source. The installation of third party drivers can be disabled on the settings page.

Previously, this server had a working set of boot images and had been used to provision a test server that i use for development purposes and general tinkering.

Now, looking at /etc/maas there is no more boot-resources.yaml, it seems. Where are those settings located now?

Looking back at the dashboard, the phrase "boot images" in "Visit the boot images page..." points to this like:

critical-maas/images (critical-maas is the hostname)

but clicking on that link returns a 404 error because /images is not present. The Images tab, however, points to

http://critical-maas/MAAS/images/

which is the correct URL. But again, the boot images I had before are no longer present. I grepped through all the files in /etc/maas but can not seem to find the right place to specify my personal image stream.

[Christian Reis (kiko)]

The 404 error has just been fixed; it was bug 1379154.

[Jeff Lane  (bladernr)]

Ok, lets see if any of this helps... First, attaching a tarball with the logs I think cover the upgrade. You'll note that I went from 1.5 to 1.6 to 1.7 because I initially had the daily PPA but didn't realize until after that I needed the experimental PPA. If needed, I can restore the system to 1.5.4 and do a straight upgrade to 1.7.

As for the boot resources I had, those actually still exist in /var/lib/maas/boot-resources:

ubuntu@critical-maas:/var/lib/maas/boot-resources$ du -h
1.8G ./snapshot-20140728-094233/amd64/generic/trusty/release
2.5G ./snapshot-20140728-094233/amd64/generic/trusty
1.8G ./snapshot-20140728-094233/amd64/generic/saucy/release
1.8G ./snapshot-20140728-094233/amd64/generic/saucy
1.7G ./snapshot-20140728-094233/amd64/generic/precise/release
1.7G ./snapshot-20140728-094233/amd64/generic/precise
5.9G ./snapshot-20140728-094233/amd64/generic
5.9G ./snapshot-20140728-094233/amd64
1.7G ./snapshot-20140728-094233/armhf/generic/trusty/release
1.7G ./snapshot-20140728-094233/armhf/generic/trusty
1.7G ./snapshot-20140728-094233/armhf/generic/saucy/release
1.7G ./snapshot-20140728-094233/armhf/generic/saucy
3.4G ./snapshot-20140728-094233/armhf/generic
4.0K ./snapshot-20140728-094233/armhf/highbank/trusty/release
8.0K ./snapshot-20140728-094233/armhf/highbank/trusty
1.4G ./snapshot-20140728-094233/armhf/highbank/precise/release
1.4G ./snapshot-20140728-094233/armhf/highbank/precise
1.4G ./snapshot-20140728-094233/armhf/highbank
4.8G ./snapshot-20140728-094233/armhf
8.0K ./snapshot-20140728-094233/grub
1.8G ./snapshot-20140728-094233/i386/generic/trusty/release
1.8G ./snapshot-20140728-094233/i386/generic/trusty
1.8G ./snapshot-20140728-094233/i386/generic/saucy/release
1.8G ./snapshot-20140728-094233/i386/generic/saucy
1.7G ./snapshot-20140728-094233/i386/generic/precise/release
1.7G ./snapshot-20140728-094233/i386/generic/precise
5.1G ./snapshot-20140728-094233/i386/generic
5.1G ./snapshot-20140728-094233/i386
16G ./snapshot-20140728-094233
1.8G ./snapshot-20141009-100806/amd64/generic/trusty/release
1.8G ./snapshot-20141009-100806/amd64/generic/trusty
1.8G ./snapshot-20141009-100806/amd64/generic
1.8G ./snapshot-20141009-100806/amd64
1.8G ./snapshot-20141009-100806
4.0K ./snapshot-20141004-225956/amd64/generic/trusty/release
8.0K ./snapshot-20141004-225956/amd64/generic/trusty
4.0K ./snapshot-20141004-225956/amd64/generic/saucy/release
8.0K ./snapshot-20141004-225956/amd64/generic/saucy
4.0K ./snapshot-20141004-225956/amd64/generic/precise/release
8.0K ./snapshot-20141004-225956/amd64/generic/precise
28K ./snapshot-20141004-225956/amd64/generic
32K ./snapshot-20141004-225956/amd64
4.0K ./snapshot-20141004-225956/armhf/generic/trusty/release
8.0K ./snapshot-20141004-225956/armhf/generic/trusty
4.0K ./snapshot-20141004-225956/armhf/generic/saucy/release
8.0K ./snapshot-20141004-225956/armhf/generic/saucy
20K ./snapshot-20141004-225956/armhf/generic
4.0K ./snapshot-20141004-225956/armhf/highbank/trusty/release
8.0K ./snapshot-20141004-225956/armhf/highbank/trusty
4.0K ./snapshot-20141004-225956/armhf/highbank/precise/release
8.0K ./snapshot-20141004-225956/armhf/highbank/precise
20K ./snapshot-20141004-225956/armhf/highbank
4...

[Julian Edwards (julian-edwards)]

The images are now all stored in the database before getting pushed out to clusters. Blake, did you write a migration for this to save people from re-downloading images?

[Jeff Lane  (bladernr)]

OK... so turns out my backups have corrupted postgresDBs and as I have limited (read no) experience with postgres, I spent a fair amount of time this weekend rebuilding it from the ground up so I could re-test the upgrade from 1.5.4 to 1.7 directly, skipping the 1.6 version that I hit before.

SO first, I started with a fully updated trusty install on my NUC and installed the current version of MAAS from that archive:
maas 1.5.4+bzr2294-0ubuntu1.1
maas-cli 1.5.4+bzr2294-0ubuntu1.1
maas-cluster-controller 1.5.4+bzr2294-0ubuntu1.1
maas-common 1.5.4+bzr2294-0ubuntu1.1
maas-dhcp 1.5.4+bzr2294-0ubuntu1.1
maas-dns 1.5.4+bzr2294-0ubuntu1.1
maas-region-controller 1.5.4+bzr2294-0ubuntu1.1
maas-region-controller-min 1.5.4+bzr2294-0ubuntu1.1
python-django-maas 1.5.4+bzr2294-0ubuntu1.1
python-maas-client 1.5.4+bzr2294-0ubuntu1.1
python-maas-provisioningserver 1.5.4+bzr2294-0ubuntu1.1

and it has 6 images available (I only imported Trusty images):
2 trusty amd64 generic commissioning release Oct. 13, 2014, 1:16 p.m.
4 trusty amd64 generic install release Oct. 13, 2014, 1:16 p.m.
1 trusty amd64 generic xinstall release Oct. 13, 2014, 1:16 p.m.
3 trusty i386 generic commissioning release Oct. 13, 2014, 1:16 p.m.
6 trusty i386 generic install release Oct. 13, 2014, 1:16 p.m.
5 trusty i386 generic xinstall release Oct. 13, 2014, 1:16 p.m.

I confirmed this new installation of maas works by enlisting, commissioning and deploying my target node with trusty.

Next, I added the experimental PPA because it contains 1.7-beta6 builds for trusty (the daily PPA still has 1.6 builds for trusty), did an apt-get update and apt-get dist-upgrade.

During the upgrade, I was prompted for /etc/maas/maas_local_settings.py. I said Yes to overwrite the older file with the newer version from 1.7. As soon as the upgrade was completed, I checked the dashboard and found again that there are no boot resoureces available:

"Boot image import process not started. Nodes will not be able to provision without boot images. Visit the boot images page to start the import."

So going directly from 1.5.4 to 1.7.0~beta6+bzr3232 also removes all my boot resources, forcing me to download them once more.

[Jeff Lane  (bladernr)]

FWIW, this is the package versions now that I have upgraded:
maas 1.7.0~beta6+bzr3232-0ubuntu1~trusty1
maas-cli 1.7.0~beta6+bzr3232-0ubuntu1~trusty1
maas-cluster-controller 1.7.0~beta6+bzr3232-0ubuntu1~trusty1
maas-common 1.7.0~beta6+bzr3232-0ubuntu1~trusty1
maas-dhcp 1.7.0~beta6+bzr3232-0ubuntu1~trusty1
maas-dns 1.7.0~beta6+bzr3232-0ubuntu1~trusty1
maas-region-controller 1.7.0~beta6+bzr3232-0ubuntu1~trusty1
maas-region-controller-min 1.7.0~beta6+bzr3232-0ubuntu1~trusty1
python-django-maas 1.7.0~beta6+bzr3232-0ubuntu1~trusty1
python-maas-client 1.7.0~beta6+bzr3232-0ubuntu1~trusty1
python-maas-provisioningserver 1.7.0~beta6+bzr3232-0ubuntu1~trusty1

[Graham Binns (gmb)]

Confirmed this just now; there doesn't appear to be any migration from 1.5 or 1.6 to 1.7 as far as boot resources are concerned.

[Blake Rouse (blake-rouse)]

There is no migration from 1.5 to 1.7 for boot images. The data that is stored in the database is very different then what is located on the cluster, mainly the root-image.gz. The cluster converts the root-image.gz to the root-image and root-tgz files, but the region needs the original root-image.gz file placed in the database.

The database also requires information about the version of the set, which doesn't exist anywhere in the cluster.

The best thing to do for upgrades and for new installs is to just start importing the boot resources. The cluster will not remove its old boot resources, until the region contains boot resources.

[Graham Binns (gmb)]

On 14 October 2014 15:43, Blake Rouse <email address hidden> wrote:
>
> The best thing to do for upgrades and for new installs is to just start
> importing the boot resources. The cluster will not remove its old boot
> resources, until the region contains boot resources.

Okay. We need to make this clear in:

 1. The release notes (and blog posts, etc.; we can frame it as "you'll
    have much more control over boot images but you'll need to reimport
    those that your nodes need")
 2. The UI (to prevent the "why am I having to reimport boot images?!" bugs.
    although if (1) is done right, there shouldn't _be_ any such bugs,
    so we can probably save ourselves the time).

[Julian Edwards (julian-edwards)]

On Tuesday 14 Oct 2014 15:00:18 you wrote:
> On 14 October 2014 15:43, Blake Rouse <email address hidden> wrote:
> > The best thing to do for upgrades and for new installs is to just start
> > importing the boot resources. The cluster will not remove its old boot
> > resources, until the region contains boot resources.
>
> Okay. We need to make this clear in:
>
> 1. The release notes (and blog posts, etc.; we can frame it as "you'll
> have much more control over boot images but you'll need to reimport
> those that your nodes need")
> 2. The UI (to prevent the "why am I having to reimport boot images?!" bugs.
> although if (1) is done right, there shouldn't _be_ any such bugs, so we
> can probably save ourselves the time).

I think this is suboptimal, but I'll make some notes in the changelog for
1.7.0

[Christian Reis (kiko)]

The bug also mentioned that the private image stream setting was lost; is that actually the case, or is it just stored somewhere else?

[Blake Rouse (blake-rouse)]

It was lost because I believe in 1.5, the stream is stored in bootresources.yaml and not in the database.

[Andres Rodriguez (andreserl)]

So as per discussed with Blake, at the very least we should show a message telling the user to import boot-images again.

[Julian Edwards (julian-edwards)]

How are we going to cope with the loss of the private stream setting? I don't know enough about Blake's import work yet to come up with a sensible suggestion.

[Christian Reis (kiko)]

The first half of this bug, the message is bug 1379890. Should we change the title to reflect what remains, the lost of a private image setting?