Nova resize fails when migration interface is customised

Bug #1956976 reported by Radosław Piliszek
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
Medium
Radosław Piliszek
Victoria
Fix Released
Medium
Radosław Piliszek
Wallaby
Fix Released
Medium
Radosław Piliszek
Xena
Fix Released
Medium
Radosław Piliszek
Yoga
Fix Released
Medium
Radosław Piliszek

Bug Description

nova_ssh listens only on migration_interface_address. By default, migration_interface_address == api_interface_address. When migration_interface_address != api_interface_address, resize operations will fail as they utilise my_ip which is still set to api_interface_address and thus nova_compute tries to connect to nova_ssh at api_interface_address which is not listened on by nova_ssh.

An example error is:

Exception during message handling: nova.exception.ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.
Command: ssh -o BatchMode=yes API_ADDR mkdir -p /var/lib/nova/instances/REDACTED
Exit code: 255
Stdout: ''
Stderr: 'ssh: connect to host API_ADDR port 8022: Connection refused\r\n'
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.7/site-packages/nova/virt/libvirt/driver.py", line 10385, in migrate_disk_and_power_off
    self._remotefs.create_dir(dest, inst_base)
  File "/var/lib/kolla/venv/lib/python3.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 94, in create_dir
    on_completion=on_completion)
  File "/var/lib/kolla/venv/lib/python3.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 185, in create_dir
    on_execute=on_execute, on_completion=on_completion)
  File "/var/lib/kolla/venv/lib/python3.7/site-packages/nova/utils.py", line 105, in ssh_execute
    return processutils.execute(*ssh_cmd, **kwargs)
  File "/var/lib/kolla/venv/lib/python3.7/site-packages/oslo_concurrency/processutils.py", line 441, in execute
    cmd=sanitized_cmd)

Tags: nova
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

The original change, introducing the possibility of this bug, simply switched the addresses: https://review.opendev.org/c/openstack/kolla-ansible/+/569131

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)
Changed in kolla-ansible:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/824036
Committed: https://opendev.org/openstack/kolla-ansible/commit/75b69ea7451ec311e53591b37a2d30eda35048c9
Submitter: "Zuul (22348)"
Branch: master

commit 75b69ea7451ec311e53591b37a2d30eda35048c9
Author: Radosław Piliszek <email address hidden>
Date: Mon Jan 10 17:10:46 2022 +0000

    Make nova_ssh listen on api_interface as well

    This is required as nova_compute tries to reach my_ip of the other
    node when resizing an instance and my_ip is set to
    api_interface_address.

    This potential issue was introduced with [1].

    [1] https://review.opendev.org/c/openstack/kolla-ansible/+/569131

    Closes-Bug: #1956976
    Change-Id: Id57a672c69a2d5aa74e55f252d05bb756bbc945a

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/825737

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/825738

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/825739

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/825739
Committed: https://opendev.org/openstack/kolla-ansible/commit/7593c11539964f255ef91059f47736d68ff55c62
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 7593c11539964f255ef91059f47736d68ff55c62
Author: Radosław Piliszek <email address hidden>
Date: Mon Jan 10 17:10:46 2022 +0000

    Make nova_ssh listen on api_interface as well

    This is required as nova_compute tries to reach my_ip of the other
    node when resizing an instance and my_ip is set to
    api_interface_address.

    This potential issue was introduced with [1].

    [1] https://review.opendev.org/c/openstack/kolla-ansible/+/569131

    Closes-Bug: #1956976
    Change-Id: Id57a672c69a2d5aa74e55f252d05bb756bbc945a
    (cherry picked from commit 75b69ea7451ec311e53591b37a2d30eda35048c9)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/825738
Committed: https://opendev.org/openstack/kolla-ansible/commit/170bca95e7b7e73b1ecf61c4304ebd214a0faa44
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 170bca95e7b7e73b1ecf61c4304ebd214a0faa44
Author: Radosław Piliszek <email address hidden>
Date: Mon Jan 10 17:10:46 2022 +0000

    Make nova_ssh listen on api_interface as well

    This is required as nova_compute tries to reach my_ip of the other
    node when resizing an instance and my_ip is set to
    api_interface_address.

    This potential issue was introduced with [1].

    [1] https://review.opendev.org/c/openstack/kolla-ansible/+/569131

    Closes-Bug: #1956976
    Change-Id: Id57a672c69a2d5aa74e55f252d05bb756bbc945a
    (cherry picked from commit 75b69ea7451ec311e53591b37a2d30eda35048c9)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/825737
Committed: https://opendev.org/openstack/kolla-ansible/commit/0354b39b1737648e42f10d59918c58a35eba62fa
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 0354b39b1737648e42f10d59918c58a35eba62fa
Author: Radosław Piliszek <email address hidden>
Date: Mon Jan 10 17:10:46 2022 +0000

    Make nova_ssh listen on api_interface as well

    This is required as nova_compute tries to reach my_ip of the other
    node when resizing an instance and my_ip is set to
    api_interface_address.

    This potential issue was introduced with [1].

    [1] https://review.opendev.org/c/openstack/kolla-ansible/+/569131

    Closes-Bug: #1956976
    Change-Id: Id57a672c69a2d5aa74e55f252d05bb756bbc945a
    (cherry picked from commit 75b69ea7451ec311e53591b37a2d30eda35048c9)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 14.0.0.0rc1

This issue was fixed in the openstack/kolla-ansible 14.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 11.4.0

This issue was fixed in the openstack/kolla-ansible 11.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 12.4.0

This issue was fixed in the openstack/kolla-ansible 12.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 13.1.0

This issue was fixed in the openstack/kolla-ansible 13.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.