images get stuck syncing

[Natalia Litvinova]

I'm running MAAS 3.2.7-12037-g.c688dd446 with:

maas root boot-resources import

And my images get stuck importing at some random percentage on multiple MAAS redeployments. Moreover, in the gui I can only see one of the resources importing even though I have 3 of them:

$ maas root boot-source-selections read 1
Success.
Machine-readable output follows:
[
    {
        "os": "ubuntu",
        "release": "focal",
        "arches": [
            "amd64"
        ],
        "subarches": [
            "*"
        ],
        "labels": [
            "*"
        ],
        "id": 1,
        "boot_source_id": 1,
        "resource_uri": "/MAAS/api/2.0/boot-sources/1/selections/1/"
    },
    {
        "os": "ubuntu",
        "release": "bionic",
        "arches": [
            "amd64"
        ],
        "subarches": [
            "*"
        ],
        "labels": [
            "*"
        ],
        "id": 2,
        "boot_source_id": 1,
        "resource_uri": "/MAAS/api/2.0/boot-sources/1/selections/2/"
    },
    {
        "os": "ubuntu",
        "release": "jammy",
        "arches": [
            "amd64"
        ],
        "subarches": [
            "*"
        ],
        "labels": [
            "*"
        ],
        "id": 3,
        "boot_source_id": 1,
        "resource_uri": "/MAAS/api/2.0/boot-sources/1/selections/3/"
    }
]

wget on images.maas.io works properly on the machine:

 wget http://images.maas.io/ephemeral-v3/stable/jammy/amd64/20230328/ga-22.04/generic/boot-kernel
--2023-06-06 13:22:31-- http://images.maas.io/ephemeral-v3/stable/jammy/amd64/20230328/ga-22.04/generic/boot-kernel
Resolving images.maas.io (images.maas.io)... 185.125.190.40, 185.125.190.37, 2620:2d:4000:1::17, ...
Connecting to images.maas.io (images.maas.io)|185.125.190.40|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11570216 (11M)
Saving to: ‘boot-kernel’

boot-kernel 100%[==================================>] 11.03M 2.23MB/s in 5.0s

2023-06-06 13:22:36 (2.23 MB/s) - ‘boot-kernel’ saved [11570216/11570216]

[Natalia Litvinova (natalytvinova)]

Screenshot of how it looks in the gui

[Natalia Litvinova (natalytvinova)]

restarting with sudo snap restart maas.supervisor helped to import the focal image which was showing in the gui, but the other 2 images disappeared from the boot-source-selection:

  $ maas root boot-source-selections read 1
Success.
Machine-readable output follows:
[
    {
        "os": "ubuntu",
        "release": "focal",
        "arches": [
            "amd64"
        ],
        "subarches": [
            "*"
        ],
        "labels": [
            "*"
        ],
        "id": 1,
        "boot_source_id": 1,
        "resource_uri": "/MAAS/api/2.0/boot-sources/1/selections/1/"
    }
]

[Alexsander de Souza (alexsander-souza)]

Hi, sorry for the delayed response.

Do you still have this synchronization issue?

From the logs I can see the Rack failing to connect to 10.169.129.50. what is this host?

[Bartosz Woronicz (mastier1)]

10.169.129.50 is VIP , weird

Anyway I encountered this again but tried restarting the whole snap

[Bartosz Woronicz (mastier1)]

I need to remove the selection and add again

Then it was able to download Jammy. There's interesting warning in the logs

2023-06-30T17:23:51+00:00 murus-infra-management-1 maas.bootsources: [info] Updated boot sources cache.
2023-06-30T17:23:51+00:00 murus-infra-management-1 maas.bootresources: [info] Started importing of boot images from 1 source(s).
2023-06-30T17:23:51+00:00 murus-infra-management-1 maas.import-images: [info] Downloading image descriptions from http://images.maas.io/ephemeral-v3/stable/
2023-06-30T17:23:51+00:00 murus-infra-management-1 maas.import-images: [info] Region downloading image descriptions from 'http://images.maas.io/ephemeral-v3/stable/'.
2023-06-30T17:23:51+00:00 murus-infra-management-1 maas.bootresources: [info] Importing images from source: http://images.maas.io/ephemeral-v3/stable/
2023-06-30T17:23:52+00:00 murus-infra-management-1 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:18.04:amd64:ga-18.04 20230324
2023-06-30T17:23:52+00:00 murus-infra-management-1 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:18.04:amd64:ga-18.04-lowlatency 20230324
2023-06-30T17:23:52+00:00 murus-infra-management-1 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:18.04:amd64:hwe-18.04 20230324
2023-06-30T17:23:52+00:00 murus-infra-management-1 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:18.04:amd64:hwe-18.04-edge 20230324
2023-06-30T17:23:52+00:00 murus-infra-management-1 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:18.04:amd64:hwe-18.04-lowlatency 20230324
2023-06-30T17:23:52+00:00 murus-infra-management-1 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:18.04:amd64:hwe-18.04-lowlatency-edge 20230324
2023-06-30T17:23:52+00:00 murus-infra-management-1 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:20.04:amd64:ga-20.04 20230327
2023-06-30T17:23:52+00:00 murus-infra-management-1 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:20.04:amd64:ga-20.04-lowlatency 20230327
2023-06-30T17:23:52+00:00 murus-infra-management-1 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:20.04:amd64:hwe-20.04 20230327
2023-06-30T17:23:52+00:00 murus-infra-management-1 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:20.04:amd64:hwe-20.04-edge 20230327
2023-06-30T17:23:52+00:00 murus-infra-management-1 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:20.04:amd64:hwe-20.04-lowlatency 20230327
2023-06-30T17:23:52+00:00 murus-infra-management-1 maas.bootresources: [warn] Ignoring unsupported filetype(manifest) from com.ubuntu.maas.stable:v3:boot:20.04:amd64:hwe-20.04-lowlatency-edge 20230327
2023-06-30T17:23:53.259959+00:00 murus-infra-management-2 maas.rpc....