Maas reports images synced; but not all images have finished importing

[Alexander Balderson]

Maas was deployed with both Bionic and Focal images set as boot sources. maas then told that focal was a boot source selection, and to import the images. Maas reported that the images were synced but only the bionic image had finished syncing; so when maas was told to use focal as the commissioning series; it reported that focal was not valid/found:

maas root boot-source-selections create 1 os=ubuntu release=focal 'subarches=*' 'labels=*' arches=amd64

maas root boot-resources import

wait for:
maas root boot-resources is-importing

maas root rack-controller list-boot-images mq3fyp (and others)
{'synced'}

[{'images': [{'name': 'ubuntu/bionic', 'architecture': 'amd64', 'subarches': ['ga-18.04', 'ga-18.04-lowlatency', 'generic', 'hwe-18.04', 'hwe-18.04-edge', 'hwe-18.04-lowlatency', 'hwe-18.04-lowlatency-edge']}, {'name': 'bootloader/uefi', 'architecture': 'arm64', 'subarches': ['generic']}, {'name': 'bootloader/uefi', 'architecture': 'amd64', 'subarches': ['generic']}, {'name': 'bootloader/open-firmware', 'architecture': 'ppc64el', 'subarches': ['generic']}, {'name': 'bootloader/pxe', 'architecture': 'i386', 'subarches': ['generic']}], 'connected': True, 'status': 'synced'}, {'images': [{'name': 'ubuntu/bionic', 'architecture': 'amd64', 'subarches': ['ga-18.04', 'ga-18.04-lowlatency', 'generic', 'hwe-18.04', 'hwe-18.04-edge', 'hwe-18.04-lowlatency', 'hwe-18.04-lowlatency-edge']}, {'name': 'bootloader/uefi', 'architecture': 'arm64', 'subarches': ['generic']}, {'name': 'bootloader/uefi', 'architecture': 'amd64', 'subarches': ['generic']}, {'name': 'bootloader/open-firmware', 'architecture': 'ppc64el', 'subarches': ['generic']}, {'name': 'bootloader/pxe', 'architecture': 'i386', 'subarches': ['generic']}], 'connected': True, 'status': 'synced'}, {'images': [{'name': 'ubuntu/bionic', 'architecture': 'amd64', 'subarches': ['ga-18.04', 'ga-18.04-lowlatency', 'generic', 'hwe-18.04', 'hwe-18.04-edge', 'hwe-18.04-lowlatency', 'hwe-18.04-lowlatency-edge']}, {'name': 'bootloader/open-firmware', 'architecture': 'ppc64el', 'subarches': ['generic']}, {'name': 'bootloader/pxe', 'architecture': 'i386', 'subarches': ['generic']}, {'name': 'bootloader/uefi', 'architecture': 'amd64', 'subarches': ['generic']}, {'name': 'bootloader/uefi', 'architecture': 'arm64', 'subarches': ['generic']}], 'connected': True, 'status': 'synced'}]

maas root maas set-config name=commissioning_distro_series value=focal

{"commissioning_distro_series": ["'focal' is not a valid commissioning_distro_series. It should be one of: 'bionic'."]}

the run and logs can be found at:

https://solutions.qa.canonical.com/kubernetes/testRun/0cd0ab0b-e032-4f13-a07e-55be304121d4

[Jason Hobbs (jason-hobbs)]

This is causing sporadic failures during solutions QA runs, marking field-high.

[Adam Collard (adam-collard)]

The linked run used MAAS 2.7.3

[Alberto Donato (ack)]

Has this issue been seen with maas 2.8+?

[Michael Skalka (mskalka)]

@Alberto

We have seen this with a recent 3.0-rc2 release test: https://solutions.qa.canonical.com/testruns/testRun/22db1ece-e2fb-436c-be27-01ef528d0fce
Logs here: https://oil-jenkins.canonical.com/artifacts/22db1ece-e2fb-436c-be27-01ef528d0fce/generated/generated/maas/logs-2021-06-14-23.11.55.tar

[Alberto Donato (ack)]

Looking at the log for this last run, it seems bionic is added as a boot source selection, but then focal is selected as commissioning series:

2021-06-14-23:09:23 root DEBUG [localhost]: maas root boot-sources read
2021-06-14-23:09:27 root DEBUG [localhost]: maas root boot-source-selections read 1
2021-06-14-23:09:32 foundationcloudengine.layers.configuremaas DEBUG creating {'release': 'bionic', 'arches': ['amd64']}
2021-06-14-23:09:32 root DEBUG [localhost]: maas root boot-source-selections create 1 os=ubuntu release=bionic 'subarches=*' 'labels=*' arches=amd64
2021-06-14-23:09:36 root DEBUG [localhost]: maas root boot-resources import
2021-06-14-23:09:41 root DEBUG [localhost]: maas root boot-resources is-importing
2021-06-14-23:09:45 root INFO Boot resources still importing, sleeping 30 seconds.
2021-06-14-23:10:15 root DEBUG [localhost]: maas root boot-resources is-importing
2021-06-14-23:10:19 root INFO Boot resources still importing, sleeping 30 seconds.
2021-06-14-23:10:49 root DEBUG [localhost]: maas root boot-resources is-importing
2021-06-14-23:10:53 root DEBUG [localhost]: maas root rack-controllers read
2021-06-14-23:10:58 root DEBUG [localhost]: maas root rack-controller list-boot-images 4fqh8t
2021-06-14-23:11:02 root DEBUG [localhost]: maas root rack-controller list-boot-images thep3t
2021-06-14-23:11:06 root DEBUG [localhost]: maas root rack-controller list-boot-images bq6747
2021-06-14-23:11:10 root INFO {'synced'}
2021-06-14-23:11:10 root INFO [{'images': [], 'connected': True, 'status': 'synced'}, {'images': [], 'connected': True, 'status': 'synced'}, {'images': [], 'connected': True, 'status': 'synced'}]
2021-06-14-23:11:10 root DEBUG [localhost]: maas root maas set-config name=commissioning_distro_series value=focal
2021-06-14-23:11:14 root ERROR [localhost] Command failed: maas root maas set-config name=commissioning_distro_series value=focal
2021-06-14-23:11:14 root ERROR [localhost] STDOUT follows:
{"commissioning_distro_series": ["'focal' is not a valid commissioning_distro_series. It should be one of: '---'."]}
2021-06-14-23:11:14 root ERROR [localhost] STDERR follows:
b''

Would it be possible to do add a `maas root boot-source-selections create` after the boot-source-selection has been added to confirm what maas is supposed to import?

[Launchpad Janitor (janitor)]

[Expired for MAAS because there has been no activity for 60 days.]