OpenStack Compute (nova)

Instances lose its serial ports during soft-reboot after live-migration

Bug #1614019 reported by Sahid Orentino
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Sahid Orentino
Newton
Fix Released
High
Lee Yarwood

Bug Description

Instances lose its serial ports during soft-reboot if the instances experienced live-migration just before.
Therefore we cannot access to the instance from serial console after soft-reboot.

That is because the method post_live_migration which defines the domain XML to libvirt on the destination host is calling the method get_guest_config instead of just retrieving the domain XML of the migrated and running guest.

Sahid Orentino (sahid-ferdjaoui)
Changed in nova:
assignee: nobody → sahid (sahid-ferdjaoui)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/356335

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/356335
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=dccaf9cb88705b2a635c5b54433c5e6187557849
Submitter: Jenkins
Branch: master

commit dccaf9cb88705b2a635c5b54433c5e6187557849
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Wed Aug 17 05:18:45 2016 -0400

    libvirt: fix serial console not correctly defined after live-migration

    During post live migration the XML defined in libvirt does not contain
    the definition of serial ports. That is because we call the method
    get_guest_config_xml to retrieve the XML definition which is not going
    to return a definition of serial ports because the instance already
    define them. Actually calling this method to retrieve domain XML of an
    instance is not good in every cases because the aim of that method is
    to define a domain XML not to return a domain XML, for that purpose we
    prefer to call XMLDesc(0).

    Also that commit is removing the process to write into a file the
    domain XML which is not used anymore. libvirtd is storing the XML
    definition of domains by itself (see: guest.write_instance_config())

    Closes-Bug: #1614019
    Change-Id: I230031bba3926171a80ae773a98280ac5c61058b

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/386676

Revision history for this message
Paul Carlton (paul-carlton2) wrote :
Download full text (11.4 KiB)

I see issues when doing a nova reboot --hard of instance after live migration

2016-10-28 07:55:21.733 94803 ERROR nova.compute.manager [req-b4bb4ded-c569-4254-a761-af72da2729e1 admin admin] [instance: 199c49e4-1855-470b-b7e3-39985eb26921] Cannot reboot instance: Unsupported VIF type binding_failed convert '_nova_to_osvif_vif_binding_failed'
2016-10-28 07:55:21.738 94803 DEBUG oslo_messaging._drivers.amqpdriver [req-b4bb4ded-c569-4254-a761-af72da2729e1 admin admin] CALL msg_id: 2bcb98ba06d84b8cbc1926658710db3b exchange 'nova' topic 'conductor' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:451
2016-10-28 07:55:21.827 94803 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 2bcb98ba06d84b8cbc1926658710db3b __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:299
2016-10-28 07:55:21.831 94803 DEBUG oslo_messaging._drivers.amqpdriver [req-b4bb4ded-c569-4254-a761-af72da2729e1 admin admin] CALL msg_id: fd03581181e44657b838e1658441e798 exchange 'nova' topic 'conductor' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:451
2016-10-28 07:55:21.854 94803 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: fd03581181e44657b838e1658441e798 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:299
2016-10-28 07:55:21.856 94803 DEBUG oslo_messaging._drivers.amqpdriver [req-b4bb4ded-c569-4254-a761-af72da2729e1 admin admin] CALL msg_id: ba9b00cc1e90430681cbc0539c55a54a exchange 'nova' topic 'conductor' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:451
2016-10-28 07:55:21.875 94803 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: ba9b00cc1e90430681cbc0539c55a54a __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:299
2016-10-28 07:55:21.883 94803 DEBUG oslo_messaging._drivers.amqpdriver [req-b4bb4ded-c569-4254-a761-af72da2729e1 admin admin] CALL msg_id: 1f8e09d179e04159b0d9c6fb78bb51b9 exchange 'nova' topic 'conductor' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:451
2016-10-28 07:55:21.993 94803 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 1f8e09d179e04159b0d9c6fb78bb51b9 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:299
2016-10-28 07:55:22.000 94803 DEBUG oslo_concurrency.lockutils [req-b4bb4ded-c569-4254-a761-af72da2729e1 admin admin] Lock "compute_resources" acquired by "nova.compute.resource_tracker.update_usage" :: waited 0.000s inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:270
2016-10-28 07:55:22.014 94803 DEBUG oslo_messaging._drivers.amqpdriver [req-b4bb4ded-c569-4254-a761-af72da2729e1 admin admin] CALL msg_id: f86891120b13448a9b56a71a8a9d77e7 exchange 'nova' topic 'conductor' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:451
2016-10-28 07:55:22.042 94803 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: f86891120b13448a9b56a71a8a9d77e7 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriv...

Revision history for this message
Paul Carlton (paul-carlton2) wrote :

reverting this change fixes the issue

Revision history for this message
Paul Carlton (paul-carlton2) wrote :

https://review.openstack.org/391418 to revert this change

Revision history for this message
Paul Carlton (paul-carlton2) wrote :

cancel above, my bad it seems to work on a new xenial based devstack env.

Matt Riedemann (mriedem)
tags: added: live-migration
Changed in nova:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/386676
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cbd2b8f0d590aaf2b736bc14a8a1d24d62d7a38f
Submitter: Jenkins
Branch: stable/newton

commit cbd2b8f0d590aaf2b736bc14a8a1d24d62d7a38f
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Wed Aug 17 05:18:45 2016 -0400

    libvirt: fix serial console not correctly defined after live-migration

    During post live migration the XML defined in libvirt does not contain
    the definition of serial ports. That is because we call the method
    get_guest_config_xml to retrieve the XML definition which is not going
    to return a definition of serial ports because the instance already
    define them. Actually calling this method to retrieve domain XML of an
    instance is not good in every cases because the aim of that method is
    to define a domain XML not to return a domain XML, for that purpose we
    prefer to call XMLDesc(0).

    Also that commit is removing the process to write into a file the
    domain XML which is not used anymore. libvirtd is storing the XML
    definition of domains by itself (see: guest.write_instance_config())

    Closes-Bug: #1614019
    Change-Id: I230031bba3926171a80ae773a98280ac5c61058b
    (cherry picked from commit dccaf9cb88705b2a635c5b54433c5e6187557849)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.0.0b1

This issue was fixed in the openstack/nova 15.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.3

This issue was fixed in the openstack/nova 14.0.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.