bionic boot source selections disappeared, images never synced

[Jason Hobbs]

In maas 2.5.3, bionic boot sources are selected by default and usually sync on their own. In this deployment, the boot source selections seem to have disappeared, or at least, the images never synced.

We added a xenial boot source selection, and waited for images to finish syncing to the region, then all rack controllers to report boot-images as synced, but bionic was not listed:

2019-05-19-10:11:18 root DEBUG maas root boot-sources read
2019-05-19-10:11:18 root DEBUG maas root boot-source-selections read 1
2019-05-19-10:11:19 foundationcloudengine.layers.maaslayer DEBUG creating {'release': 'xenial', 'arches': ['amd64']}
2019-05-19-10:11:19 root DEBUG maas root boot-source-selections create 1 os=ubuntu release=xenial subarches=* labels=* arches=amd64
2019-05-19-10:11:20 root DEBUG maas root boot-resources import
2019-05-19-10:11:20 root DEBUG maas root boot-resources is-importing
2019-05-19-10:11:51 root DEBUG maas root boot-resources is-importing
2019-05-19-10:12:22 root DEBUG maas root boot-resources is-importing
2019-05-19-10:12:52 root DEBUG maas root boot-resources is-importing
2019-05-19-10:13:23 root DEBUG maas root boot-resources is-importing
2019-05-19-10:13:54 root DEBUG maas root boot-resources is-importing
2019-05-19-10:14:24 root DEBUG maas root boot-resources is-importing
2019-05-19-10:14:55 root DEBUG maas root boot-resources is-importing
2019-05-19-10:15:26 root DEBUG maas root boot-resources is-importing
2019-05-19-10:15:57 root DEBUG maas root boot-resources is-importing
2019-05-19-10:16:27 root DEBUG maas root boot-resources is-importing
2019-05-19-10:16:58 root DEBUG maas root boot-resources is-importing
2019-05-19-10:17:29 root DEBUG maas root boot-resources is-importing
2019-05-19-10:17:59 root DEBUG maas root boot-resources is-importing
2019-05-19-10:18:30 root DEBUG maas root boot-resources is-importing
2019-05-19-10:18:31 root DEBUG maas root rack-controllers read
2019-05-19-10:18:32 root DEBUG maas root rack-controller list-boot-images nqw8gx
2019-05-19-10:18:32 root DEBUG maas root rack-controller list-boot-images 8chesd
2019-05-19-10:18:33 root DEBUG maas root rack-controller list-boot-images 86477e
2019-05-19-10:18:34 root INFO set([u'synced', u'syncing'])
2019-05-19-10:19:04 root DEBUG maas root rack-controllers read
2019-05-19-10:19:05 root DEBUG maas root rack-controller list-boot-images nqw8gx
2019-05-19-10:19:05 root DEBUG maas root rack-controller list-boot-images 8chesd
2019-05-19-10:19:06 root DEBUG maas root rack-controller list-boot-images 86477e
2019-05-19-10:19:07 root INFO set([u'synced'])
2019-05-19-10:19:07 root INFO [{u'images': [{u'subarches': [u'ga-16.04', u'ga-16.04-lowlatency', u'generic'], u'name': u'ubuntu/xenial', u'architecture': u'amd64'}, {u'subarches': [u'generic'], u'name': u'bootloader/uefi', u'architecture': u'amd64'}, {u'subarches': [u'generic'], u'name': u'bootloader/uefi', u'architecture': u'arm64'}, {u'subarches': [u'generic'], u'name': u'bootloader/pxe', u'architecture': u'i386'}, {u'subarches': [u'generic'], u'name': u'bootloader/open-firmware', u'architecture': u'ppc64el'}], u'status': u'synced', u'connected': True}, {u'images': [{u'subarches': [u'ga-16.04', u'ga-16.04-lowlatency', u'generic'], u'name': u'ubuntu/xenial', u'architecture': u'amd64'}, {u'subarches': [u'generic'], u'name': u'bootloader/open-firmware', u'architecture': u'ppc64el'}, {u'subarches': [u'generic'], u'name': u'bootloader/pxe', u'architecture': u'i386'}, {u'subarches': [u'generic'], u'name': u'bootloader/uefi', u'architecture': u'arm64'}, {u'subarches': [u'generic'], u'name': u'bootloader/uefi', u'architecture': u'amd64'}], u'status': u'synced', u'connected': True}, {u'images': [{u'subarches': [u'ga-16.04', u'ga-16.04-lowlatency', u'generic'], u'name': u'ubuntu/xenial', u'architecture': u'amd64'}, {u'subarches': [u'generic'], u'name': u'bootloader/open-firmware', u'architecture': u'ppc64el'}, {u'subarches': [u'generic'], u'name': u'bootloader/pxe', u'architecture': u'i386'}, {u'subarches': [u'generic'], u'name': u'bootloader/uefi', u'architecture': u'arm64'}, {u'subarches': [u'generic'], u'name': u'bootloader/uefi', u'architecture': u'amd64'}], u'status': u'synced', u'connected': True}]

Here's the same sequence on a successful 2.5.3 run, where we see 18.04 images:
http://paste.ubuntu.com/p/JNrXtfHVyg/

This caused xenial to be used for commissioning, and eventually, the juju controller bootstrap to fail because bionic was missing.

[Andres Rodriguez (andreserl)]

Hi Jason,

Did this issue happen again? What I'm thinking it could have been is that images were being updated in images.maas.io and this wasn't on the MAAS side.

[Jason Hobbs (jason-hobbs)]

It doesn't matter if images were updated on maas.io - bionic should be there, and maas shouldn't tell me my images are synced if it's not there.

[Blake Rouse (blake-rouse)]

I think this might be a timing issue. With your automation do you wait until the bionic selection is made before your make the selection for xenial? If not that is probably the issue, the system checks if any selections are made before it adds the default installation ones. This is an upgrade artifact of how MAAS handles upgrades from previous versions (i know this is a fresh installation).

The issue would be that your automation came in to quickly and made a selection, then MAAS checks to set the selection (sees that it already has one), then does nothing. As that would be unexpected by the user that a selection they didn't want be placed in the system.

[Jason Hobbs (jason-hobbs)]

Blake - we read the boot source selections and then add the ones that are
missing. In this case, we just added xenial because bionic was already
there.

On Wed, May 22, 2019 at 9:01 AM Blake Rouse <email address hidden>
wrote:

> I think this might be a timing issue. With your automation do you wait
> until the bionic selection is made before your make the selection for
> xenial? If not that is probably the issue, the system checks if any
> selections are made before it adds the default installation ones. This
> is an upgrade artifact of how MAAS handles upgrades from previous
> versions (i know this is a fresh installation).
>
> The issue would be that your automation came in to quickly and made a
> selection, then MAAS checks to set the selection (sees that it already
> has one), then does nothing. As that would be unexpected by the user
> that a selection they didn't want be placed in the system.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1829763
>
> Title:
> bionic boot source selections disappeared, images never synced
>
> Status in MAAS:
> New
>
> Bug description:
> In maas 2.5.3, bionic boot sources are selected by default and usually
> sync on their own. In this deployment, the boot source selections seem
> to have disappeared, or at least, the images never synced.
>
> We added a xenial boot source selection, and waited for images to
> finish syncing to the region, then all rack controllers to report
> boot-images as synced, but bionic was not listed:
>
> 2019-05-19-10:11:18 root DEBUG maas root boot-sources read
> 2019-05-19-10:11:18 root DEBUG maas root boot-source-selections read 1
> 2019-05-19-10:11:19 foundationcloudengine.layers.maaslayer DEBUG
> creating {'release': 'xenial', 'arches': ['amd64']}
> 2019-05-19-10:11:19 root DEBUG maas root boot-source-selections create 1
> os=ubuntu release=xenial subarches=* labels=* arches=amd64
> 2019-05-19-10:11:20 root DEBUG maas root boot-resources import
> 2019-05-19-10:11:20 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:11:51 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:12:22 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:12:52 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:13:23 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:13:54 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:14:24 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:14:55 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:15:26 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:15:57 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:16:27 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:16:58 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:17:29 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:17:59 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:18:30 root DEBUG maas root boot-resources is-importing
> 2019-05-19-10:18:31 root DEBUG maas root ra...

[Blake Rouse (blake-rouse)]

Hmm so it was there, then gone? That is very very strange behavior.

[Konstantinos Kaskavelis (kaskavel)]

Closing this due to inactivity (low number of occurrences, and no hit for more than one year)