Horizon could not identify image id (Recoverable error: volume_image_metadata)

[Hemant Sonawane]

openstack horizon stable/victoria issue

In our case we migrated some volumes from different env to our current one and its strange that when we try to boot the instance from the migrated volume and then try to see the instance details we encountered some issue in horizon stable/victoria

"Error: Failed to get attached volume. Details volume_image_metadata".

I also tried to see in horizon logs but i could only able find this

2021-02-25 12:25:27.511270 Recoverable error: volume_image_metadata

apache logs:

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.233.80.6. Set the 'ServerName' directive globally to suppress this message
[Thu Feb 25 08:40:03.711351 2021] [mpm_event:notice] [pid 1:tid 140542017477568] AH00489: Apache/2.4.29 (Ubuntu) mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
[Thu Feb 25 08:40:03.711503 2021] [core:notice] [pid 1:tid 140542017477568] AH00094: Command line: 'apache2 -D FOREGROUND'

It looks like horizon could not identify the image from migrated volumes.

Then i tried to create volume without image and made it bootable and then launch instance from it and i encountered same issue. which means if image gets deleted or in fact there is no image in volume the Keyerror gets triggered. But in my case the migrated volume does contain image and it instance works completely fine its horizon which could not identify the image id in metadata section.

so please take a look and try to reproduce it on your end and let me know if there is any solution for this bug. Thanks

[Akihiro Motoki (amotoki)]

I succeeded to reproduce this by creating a new volume without being backed by an image, mark it as bootable. (openstack volume create --size 1 vol1, openstack volume set --bootable vol1) and then create a server using the volume.
When I visit the server detail page, I confirmed this bug. I got AttributeError rather than KeyError though.

Perhaps this issue happens when a server is created using a volume which was created before the volume_image_metadata feature was implemented.

[Akihiro Motoki (amotoki)]

https://review.opendev.org/c/openstack/horizon/+/777654 was proposed.

[Akihiro Motoki (amotoki)]

> But in my case the migrated volume does contain
image and it instance works completely fine its horizon which could not
identify the image id in metadata section.

Does your volume have "volume_image_metadata" field?
I would like to check the command result of "openstack volume show <your volume>".

[Hemant Sonawane (hemson95)]

Hello Akihiro, No we do not see "volume_image_metadata" field. Have a look

--------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{'id': 'XXXXX', 'attachment_id': 'XXXXX', 'volume_id': 'XXXXX', 'server_id': 'XXXXX', 'host_name': 'XXXXX', 'device': 'XXX', 'attached_at': 'XXXXX}] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2021-02-24T10:10:26.000000 |
| description | None |
| encrypted | False |
| id | XXXXXX |
| migration_status | None |
| multiattach ...

[Hemant Sonawane (hemson95)]

We also tried to assign fake image_id to the volume but its triggering the same issue.

cinder image-metadata <volume-id> set image_id=dummy-image-id

[Akihiro Motoki (amotoki)]

Hi Hemant,
You seems to try two things (AttributeError and KeyError). I am confused and I cannot move it forward without understanding the sitaution correctly.

The first issue you reported is "Recoverable error: volume_image_metadata". This is apparently not KeyError but AttributeError. If you add LOG.exception to the code, you can confirm AttributeError.

Regading #5, you fake image_id and this leads to a situation that 'volume_image_metadata' is created for the volume. This leads to KeyError unless volume_image_metadata contains BOTH of image_id and image_name. In this case, the error you first mentioned "Recoverable error: volume_image_metadata" is not raised. What are you tring to reproduce?

I am not what you mean by "the same issue". AttributeError you mentioned first in the bug descriptionn, or KeyError you hit when you encounters during tyring to reproduce it. I am totally confused.

[Akihiro Motoki (amotoki)]

In case of KeyError happens, the error message will be Recoverable error: 'image_id'. The detail exception traceback will be:

 Traceback (most recent call last):
   File "/srv/extra/home/user/src/openddev.org/openstack/horizon/openstack_dashboard/dashboards/project/instances/tabs.py", line 49, in get_context_data
     'id': volume.volume_image_metadata['image_id'],
 KeyError: 'image_id'

It looks like you try to reproduce KeyError but the original situation reported is different.

[Hemant Sonawane (hemson95)]

Hello akhiro,

I mean to say the attribute error. But as i applied your fix https://review.opendev.org/c/openstack/horizon/+/777654 it seems to work now. I dont see any attribute error. I was just trying multiple things at the same time to make it work. sorry if you get confused cause of me. But for me your fix working completely fine. Thank you so much for cooperation.

[Akihiro Motoki (amotoki)]

Hermant, no problem. Thanks for testing the patch. Developers soemtimes tend to forget to take into account resources from previous releases. I hit similar cases when I operated OpenStack private clouds.

[Akihiro Motoki (amotoki)]

"Fix Released" means a fix has been merged into the master branch but the fix proposed is still under reviwe.
Resetting the status to "In Progress".

[Hemant Sonawane (hemson95)]

Hi Akihiro Motoki

Sorry my bad this was my first bug report so I hope you understand me. I will keep this in mind.

thank you very much :)

Hemant

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 19.1.0 release.

[Akihiro Motoki (amotoki)]

https://review.opendev.org/c/openstack/horizon/+/777654 has landed.

[[e-mail] (eblock)]

Will this fix be backported to Victoria and/or Ussuri?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/horizon/+/803430

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/horizon/+/803430
Committed: https://opendev.org/openstack/horizon/commit/ad30888c041e2ac1f1bf207165da72ff56d91f5d
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit ad30888c041e2ac1f1bf207165da72ff56d91f5d
Author: Akihiro Motoki <email address hidden>
Date: Fri Feb 26 03:44:15 2021 +0900

    Handle an attached volume without volume_image_metadata

    There is a case where volume_image_metadata attribute does not exist.
    It looks like it happens for example when a volume was created before
    the volume_image_metadata feature was implemented.

    Change-Id: I0b8e6b2e540a1782b9edd9921490a9371d31afc7
    Closes-Bug: #1916891
    (cherry picked from commit b841952906de33ff93c90cc04ea0509afeaab89f)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/horizon/+/814260

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/horizon/+/814260
Committed: https://opendev.org/openstack/horizon/commit/432f77f887ca15ba8a37fa8a45e9987f4766a6b5
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 432f77f887ca15ba8a37fa8a45e9987f4766a6b5
Author: Akihiro Motoki <email address hidden>
Date: Fri Feb 26 03:44:15 2021 +0900

    Handle an attached volume without volume_image_metadata

    There is a case where volume_image_metadata attribute does not exist.
    It looks like it happens for example when a volume was created before
    the volume_image_metadata feature was implemented.

    Change-Id: I0b8e6b2e540a1782b9edd9921490a9371d31afc7
    Closes-Bug: #1916891
    (cherry picked from commit b841952906de33ff93c90cc04ea0509afeaab89f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 18.3.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 18.6.3 release.