OpenStack Compute (nova)

  1. Bug #1890501
  2. Activity log

Activity log for bug #1890501

Date Who What changed Old value New value Message
2020-08-05 22:24:22 Lee Yarwood bug added bug
2020-08-05 22:51:39 Jeremy Stanley description Description =========== It appears that this is due to Nova not providing VIR_MIGRATE_PARAM_PERSIST_XML during the migration resulting in the original source domains persistent configuration being used instead: /** * VIR_MIGRATE_PARAM_DEST_XML: * * virDomainMigrate* params field: the new configuration to be used for the * domain on the destination host as VIR_TYPED_PARAM_STRING. The configuration * must include an identical set of virtual devices, to ensure a stable guest * ABI across migration. Only parameters related to host side configuration * can be changed in the XML. Hypervisors which support this field will forbid * migration if the provided XML would cause a change in the guest ABI. This * field cannot be used to rename the domain during migration (use * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the * destination XML must match the original domain name. * * Omitting this parameter keeps the original domain configuration. Using this * field with hypervisors that do not support changing domain configuration * during migration will result in a failure. */ # define VIR_MIGRATE_PARAM_DEST_XML "destination_xml" /** * VIR_MIGRATE_PARAM_PERSIST_XML: * * virDomainMigrate* params field: the new persistent configuration to be used * for the domain on the destination host as VIR_TYPED_PARAM_STRING. * This field cannot be used to rename the domain during migration (use * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the * destination XML must match the original domain name. * * Omitting this parameter keeps the original domain persistent configuration. * Using this field with hypervisors that do not support changing domain * configuration during migration will result in a failure. */ # define VIR_MIGRATE_PARAM_PERSIST_XML "persistent_xml" Steps to reproduce ================== 0) Deploy overcloud with multipath and iscsi/LVM cinder backend. 1) Delete all instances and check no device path remained on both host1 and host2. 2) Boot instances, VM1 on host1 and VM2 on host2. $ cinder create --name cirros1 --volume-type lvm --image cirros 1 $ cinder create --name cirros2 --volume-type lvm --image cirros 1 $ nova boot --block-device-mapping vda=$cirrosvol1 ... --host host1.localdomain testvm1 $ nova boot --block-device-mapping vda=$cirrosvol2 ... --host host2.localdomain testvm2 $ openstack server add floating ip testvm1 xx.xx.xx.xx $ openstack server add floating ip testvm2 yy.yy.yy.yy 3) Soft reboot each instances and check no problem has occured. $ nova reboot testvm1 $ nova reboot testvm2 4) Execute live-migration VM1 to host2, check VMs for the device path setting in each XML. $ nova live-migration testvm1 host2.localdomain 5) Execute soft reboot VM1, check VMs for the device path setting in each XML. $ nova reboot testvm1 6) Login to each VMs and check syslogs. Expected result =============== After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are changed to new value fit to destination host. Actual result ============= After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are the value of source host before migration. Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ Reported downstream against stable/train and libvirt 5.6.0-10. 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? libvirt + KVM 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? LVM/iSCSI with multipath enabled but any host block based storage backend will do. 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A Logs & Configs ============== * Device paths initial state host1 host2 =================================================================================================== VM1 multipath -ll 360014053825c172898b4ba4a5353515c dm-0 --- virsh dumpxml <source dev='/dev/dm-0' index='1'/> --- virsh dumpxml --inactive <source dev='/dev/dm-0'/> --- qemu xml file <source dev='/dev/dm-0'/> --- --------------------------------------------------------------------------------------------------- VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0 virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> qemu xml file --- <source dev='/dev/dm-0'/> * Device paths after VM1 live-migration to host2 host1 host2 =================================================================================================== VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2 virsh dumpxml --- <source dev='/dev/dm-2' index='1'/> virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> <== not dm-2 qemu xml file --- <source dev='/dev/dm-0'/> <== not dm-2 --------------------------------------------------------------------------------------------------- VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0 virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> qemu xml file --- <source dev='/dev/dm-0'/> * Device paths after soft reboot VM1 on host2 host1 host2 =================================================================================================== VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2 virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> <== changed to dm-0 virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> qemu xml file --- <source dev='/dev/dm-0'/> --------------------------------------------------------------------------------------------------- VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0 virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> qemu xml file --- <source dev='/dev/dm-0'/> * VM1 syslog file before live-migration $ cat /var/log/messages ... Jul 28 05:28:38 cirrostestvm1 kern.info kernel: [ 0.780031] usb 1-1: new full-speed USB device number 2 using uhci_hcd Jul 28 05:28:39 cirrostestvm1 kern.info kernel: [ 1.272305] Refined TSC clocksource calibration: 2099.976 MHz. Jul 28 05:28:40 cirrostestvm1 authpriv.info dropbear[260]: Running in background Jul 28 05:28:40 cirrostestvm1 daemon.info init: reloading /etc/inittab Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/ttyS0': '/sbin/getty -L 115200 ttyS0 vt100 ' Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/tty1': '/sbin/getty 115200 tty1' Jul 28 05:28:48 cirrostestvm1 kern.debug kernel: [ 10.992106] eth0: no IPv6 routers present Jul 28 05:29:45 cirrostestvm1 authpriv.info dropbear[301]: Child connection from 172.31.151.1:33648 Jul 28 05:29:48 cirrostestvm1 authpriv.notice dropbear[301]: Password auth succeeded for 'cirros' from 172.31.151.1:33648 $ * VM1 syslog file after soft reboot on host2 hostname command return correct value, but VM1 syslog is recorded by VM2. (in some cases, VM1 and VM2 syslog files are destroyed and cannot be read as text file) $ hostname cirrostestvm1 $ cat /var/log/messages | tail Jul 28 06:03:01 cirrostestvm2 authpriv.info dropbear[325]: Child connection from 172.31.151.1:35894 Jul 28 06:03:05 cirrostestvm2 authpriv.notice dropbear[325]: Password auth succeeded for 'cirros' from 172.31.151.1:35894 Jul 28 06:03:05 cirrostestvm2 authpriv.info dropbear[325]: Exit (cirros): Disconnect received Jul 28 06:03:30 cirrostestvm2 authpriv.info dropbear[328]: Child connection from 172.31.151.1:36352 Jul 28 06:03:34 cirrostestvm2 authpriv.notice dropbear[328]: Password auth succeeded for 'cirros' from 172.31.151.1:36352 Jul 28 06:03:34 cirrostestvm2 authpriv.info dropbear[328]: Exit (cirros): Disconnect received Jul 28 06:03:39 cirrostestvm2 authpriv.info dropbear[331]: Child connection from 172.31.151.1:36484 Jul 28 06:03:41 cirrostestvm2 authpriv.info dropbear[331]: Exit before auth (user 'cirros', 0 fails): Exited normally Jul 28 06:03:45 cirrostestvm2 authpriv.info dropbear[332]: Child connection from 172.31.151.1:36588 Jul 28 06:03:49 cirrostestvm2 authpriv.notice dropbear[332]: Password auth succeeded for 'cirros' from 172.31.151.1:36588 This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments. This embargo shall not extend past $NINETY_DAYS and will be made public by or on that date even if no fix is identified. Description =========== It appears that this is due to Nova not providing VIR_MIGRATE_PARAM_PERSIST_XML during the migration resulting in the original source domains persistent configuration being used instead: /**       * VIR_MIGRATE_PARAM_DEST_XML:       *       * virDomainMigrate* params field: the new configuration to be used for the       * domain on the destination host as VIR_TYPED_PARAM_STRING. The configuration       * must include an identical set of virtual devices, to ensure a stable guest       * ABI across migration. Only parameters related to host side configuration       * can be changed in the XML. Hypervisors which support this field will forbid       * migration if the provided XML would cause a change in the guest ABI. This       * field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain configuration. Using this       * field with hypervisors that do not support changing domain configuration       * during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_DEST_XML "destination_xml"      /**       * VIR_MIGRATE_PARAM_PERSIST_XML:       *       * virDomainMigrate* params field: the new persistent configuration to be used       * for the domain on the destination host as VIR_TYPED_PARAM_STRING.       * This field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain persistent configuration.       * Using this field with hypervisors that do not support changing domain       * configuration during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_PERSIST_XML "persistent_xml" Steps to reproduce ==================    0) Deploy overcloud with multipath and iscsi/LVM cinder backend.    1) Delete all instances and check no device path remained on both host1 and host2.    2) Boot instances, VM1 on host1 and VM2 on host2.       $ cinder create --name cirros1 --volume-type lvm --image cirros 1       $ cinder create --name cirros2 --volume-type lvm --image cirros 1       $ nova boot --block-device-mapping vda=$cirrosvol1 ... --host host1.localdomain testvm1       $ nova boot --block-device-mapping vda=$cirrosvol2 ... --host host2.localdomain testvm2       $ openstack server add floating ip testvm1 xx.xx.xx.xx       $ openstack server add floating ip testvm2 yy.yy.yy.yy    3) Soft reboot each instances and check no problem has occured.       $ nova reboot testvm1       $ nova reboot testvm2    4) Execute live-migration VM1 to host2, check VMs for the device path setting in       each XML.       $ nova live-migration testvm1 host2.localdomain    5) Execute soft reboot VM1, check VMs for the device path setting in each XML.       $ nova reboot testvm1    6) Login to each VMs and check syslogs. Expected result =============== After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are changed to new value fit to destination host. Actual result ============= After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are the value of source host before migration. Environment =========== 1. Exact version of OpenStack you are running. See the following   list for all releases: http://docs.openstack.org/releases/    Reported downstream against stable/train and libvirt 5.6.0-10. 2. Which hypervisor did you use?    (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)    What's the version of that?    libvirt + KVM 2. Which storage type did you use?    (For example: Ceph, LVM, GPFS, ...)    What's the version of that?    LVM/iSCSI with multipath enabled but any host block based storage backend will do. 3. Which networking type did you use?    (For example: nova-network, Neutron with OpenVSwitch, ...)    N/A Logs & Configs ==============    * Device paths initial state                                      host1 host2      ===================================================================================================      VM1 multipath -ll 360014053825c172898b4ba4a5353515c dm-0 ---          virsh dumpxml <source dev='/dev/dm-0' index='1'/> ---          virsh dumpxml --inactive <source dev='/dev/dm-0'/> ---          qemu xml file <source dev='/dev/dm-0'/> ---      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after VM1 live-migration to host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-2' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> <== not dm-2          qemu xml file --- <source dev='/dev/dm-0'/> <== not dm-2      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after soft reboot VM1 on host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> <== changed to dm-0          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * VM1 syslog file before live-migration          $ cat /var/log/messages          ...          Jul 28 05:28:38 cirrostestvm1 kern.info kernel: [ 0.780031] usb 1-1: new full-speed USB device number 2 using uhci_hcd          Jul 28 05:28:39 cirrostestvm1 kern.info kernel: [ 1.272305] Refined TSC clocksource calibration: 2099.976 MHz.          Jul 28 05:28:40 cirrostestvm1 authpriv.info dropbear[260]: Running in background          Jul 28 05:28:40 cirrostestvm1 daemon.info init: reloading /etc/inittab          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/ttyS0': '/sbin/getty -L 115200 ttyS0 vt100 '          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/tty1': '/sbin/getty 115200 tty1'          Jul 28 05:28:48 cirrostestvm1 kern.debug kernel: [ 10.992106] eth0: no IPv6 routers present          Jul 28 05:29:45 cirrostestvm1 authpriv.info dropbear[301]: Child connection from 172.31.151.1:33648          Jul 28 05:29:48 cirrostestvm1 authpriv.notice dropbear[301]: Password auth succeeded for 'cirros' from 172.31.151.1:33648          $    * VM1 syslog file after soft reboot on host2        hostname command return correct value, but VM1 syslog is recorded by VM2.        (in some cases, VM1 and VM2 syslog files are destroyed and cannot be read as text file)          $ hostname          cirrostestvm1          $ cat /var/log/messages | tail          Jul 28 06:03:01 cirrostestvm2 authpriv.info dropbear[325]: Child connection from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.notice dropbear[325]: Password auth succeeded for 'cirros' from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.info dropbear[325]: Exit (cirros): Disconnect received          Jul 28 06:03:30 cirrostestvm2 authpriv.info dropbear[328]: Child connection from 172.31.151.1:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.notice dropbear[328]: Password auth succeeded for 'cirros' from 172.31.151.1:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.info dropbear[328]: Exit (cirros): Disconnect received          Jul 28 06:03:39 cirrostestvm2 authpriv.info dropbear[331]: Child connection from 172.31.151.1:36484          Jul 28 06:03:41 cirrostestvm2 authpriv.info dropbear[331]: Exit before auth (user 'cirros', 0 fails): Exited normally          Jul 28 06:03:45 cirrostestvm2 authpriv.info dropbear[332]: Child connection from 172.31.151.1:36588          Jul 28 06:03:49 cirrostestvm2 authpriv.notice dropbear[332]: Password auth succeeded for 'cirros' from 172.31.151.1:36588
2020-08-05 22:51:52 Jeremy Stanley bug task added ossa
2020-08-05 22:52:28 Jeremy Stanley description This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments. This embargo shall not extend past $NINETY_DAYS and will be made public by or on that date even if no fix is identified. Description =========== It appears that this is due to Nova not providing VIR_MIGRATE_PARAM_PERSIST_XML during the migration resulting in the original source domains persistent configuration being used instead: /**       * VIR_MIGRATE_PARAM_DEST_XML:       *       * virDomainMigrate* params field: the new configuration to be used for the       * domain on the destination host as VIR_TYPED_PARAM_STRING. The configuration       * must include an identical set of virtual devices, to ensure a stable guest       * ABI across migration. Only parameters related to host side configuration       * can be changed in the XML. Hypervisors which support this field will forbid       * migration if the provided XML would cause a change in the guest ABI. This       * field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain configuration. Using this       * field with hypervisors that do not support changing domain configuration       * during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_DEST_XML "destination_xml"      /**       * VIR_MIGRATE_PARAM_PERSIST_XML:       *       * virDomainMigrate* params field: the new persistent configuration to be used       * for the domain on the destination host as VIR_TYPED_PARAM_STRING.       * This field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain persistent configuration.       * Using this field with hypervisors that do not support changing domain       * configuration during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_PERSIST_XML "persistent_xml" Steps to reproduce ==================    0) Deploy overcloud with multipath and iscsi/LVM cinder backend.    1) Delete all instances and check no device path remained on both host1 and host2.    2) Boot instances, VM1 on host1 and VM2 on host2.       $ cinder create --name cirros1 --volume-type lvm --image cirros 1       $ cinder create --name cirros2 --volume-type lvm --image cirros 1       $ nova boot --block-device-mapping vda=$cirrosvol1 ... --host host1.localdomain testvm1       $ nova boot --block-device-mapping vda=$cirrosvol2 ... --host host2.localdomain testvm2       $ openstack server add floating ip testvm1 xx.xx.xx.xx       $ openstack server add floating ip testvm2 yy.yy.yy.yy    3) Soft reboot each instances and check no problem has occured.       $ nova reboot testvm1       $ nova reboot testvm2    4) Execute live-migration VM1 to host2, check VMs for the device path setting in       each XML.       $ nova live-migration testvm1 host2.localdomain    5) Execute soft reboot VM1, check VMs for the device path setting in each XML.       $ nova reboot testvm1    6) Login to each VMs and check syslogs. Expected result =============== After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are changed to new value fit to destination host. Actual result ============= After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are the value of source host before migration. Environment =========== 1. Exact version of OpenStack you are running. See the following   list for all releases: http://docs.openstack.org/releases/    Reported downstream against stable/train and libvirt 5.6.0-10. 2. Which hypervisor did you use?    (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)    What's the version of that?    libvirt + KVM 2. Which storage type did you use?    (For example: Ceph, LVM, GPFS, ...)    What's the version of that?    LVM/iSCSI with multipath enabled but any host block based storage backend will do. 3. Which networking type did you use?    (For example: nova-network, Neutron with OpenVSwitch, ...)    N/A Logs & Configs ==============    * Device paths initial state                                      host1 host2      ===================================================================================================      VM1 multipath -ll 360014053825c172898b4ba4a5353515c dm-0 ---          virsh dumpxml <source dev='/dev/dm-0' index='1'/> ---          virsh dumpxml --inactive <source dev='/dev/dm-0'/> ---          qemu xml file <source dev='/dev/dm-0'/> ---      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after VM1 live-migration to host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-2' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> <== not dm-2          qemu xml file --- <source dev='/dev/dm-0'/> <== not dm-2      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after soft reboot VM1 on host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> <== changed to dm-0          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * VM1 syslog file before live-migration          $ cat /var/log/messages          ...          Jul 28 05:28:38 cirrostestvm1 kern.info kernel: [ 0.780031] usb 1-1: new full-speed USB device number 2 using uhci_hcd          Jul 28 05:28:39 cirrostestvm1 kern.info kernel: [ 1.272305] Refined TSC clocksource calibration: 2099.976 MHz.          Jul 28 05:28:40 cirrostestvm1 authpriv.info dropbear[260]: Running in background          Jul 28 05:28:40 cirrostestvm1 daemon.info init: reloading /etc/inittab          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/ttyS0': '/sbin/getty -L 115200 ttyS0 vt100 '          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/tty1': '/sbin/getty 115200 tty1'          Jul 28 05:28:48 cirrostestvm1 kern.debug kernel: [ 10.992106] eth0: no IPv6 routers present          Jul 28 05:29:45 cirrostestvm1 authpriv.info dropbear[301]: Child connection from 172.31.151.1:33648          Jul 28 05:29:48 cirrostestvm1 authpriv.notice dropbear[301]: Password auth succeeded for 'cirros' from 172.31.151.1:33648          $    * VM1 syslog file after soft reboot on host2        hostname command return correct value, but VM1 syslog is recorded by VM2.        (in some cases, VM1 and VM2 syslog files are destroyed and cannot be read as text file)          $ hostname          cirrostestvm1          $ cat /var/log/messages | tail          Jul 28 06:03:01 cirrostestvm2 authpriv.info dropbear[325]: Child connection from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.notice dropbear[325]: Password auth succeeded for 'cirros' from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.info dropbear[325]: Exit (cirros): Disconnect received          Jul 28 06:03:30 cirrostestvm2 authpriv.info dropbear[328]: Child connection from 172.31.151.1:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.notice dropbear[328]: Password auth succeeded for 'cirros' from 172.31.151.1:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.info dropbear[328]: Exit (cirros): Disconnect received          Jul 28 06:03:39 cirrostestvm2 authpriv.info dropbear[331]: Child connection from 172.31.151.1:36484          Jul 28 06:03:41 cirrostestvm2 authpriv.info dropbear[331]: Exit before auth (user 'cirros', 0 fails): Exited normally          Jul 28 06:03:45 cirrostestvm2 authpriv.info dropbear[332]: Child connection from 172.31.151.1:36588          Jul 28 06:03:49 cirrostestvm2 authpriv.notice dropbear[332]: Password auth succeeded for 'cirros' from 172.31.151.1:36588 This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments. This embargo shall not extend past 2020-11-03 and will be made public by or on that date even if no fix is identified. Description =========== It appears that this is due to Nova not providing VIR_MIGRATE_PARAM_PERSIST_XML during the migration resulting in the original source domains persistent configuration being used instead: /**       * VIR_MIGRATE_PARAM_DEST_XML:       *       * virDomainMigrate* params field: the new configuration to be used for the       * domain on the destination host as VIR_TYPED_PARAM_STRING. The configuration       * must include an identical set of virtual devices, to ensure a stable guest       * ABI across migration. Only parameters related to host side configuration       * can be changed in the XML. Hypervisors which support this field will forbid       * migration if the provided XML would cause a change in the guest ABI. This       * field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain configuration. Using this       * field with hypervisors that do not support changing domain configuration       * during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_DEST_XML "destination_xml"      /**       * VIR_MIGRATE_PARAM_PERSIST_XML:       *       * virDomainMigrate* params field: the new persistent configuration to be used       * for the domain on the destination host as VIR_TYPED_PARAM_STRING.       * This field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain persistent configuration.       * Using this field with hypervisors that do not support changing domain       * configuration during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_PERSIST_XML "persistent_xml" Steps to reproduce ==================    0) Deploy overcloud with multipath and iscsi/LVM cinder backend.    1) Delete all instances and check no device path remained on both host1 and host2.    2) Boot instances, VM1 on host1 and VM2 on host2.       $ cinder create --name cirros1 --volume-type lvm --image cirros 1       $ cinder create --name cirros2 --volume-type lvm --image cirros 1       $ nova boot --block-device-mapping vda=$cirrosvol1 ... --host host1.localdomain testvm1       $ nova boot --block-device-mapping vda=$cirrosvol2 ... --host host2.localdomain testvm2       $ openstack server add floating ip testvm1 xx.xx.xx.xx       $ openstack server add floating ip testvm2 yy.yy.yy.yy    3) Soft reboot each instances and check no problem has occured.       $ nova reboot testvm1       $ nova reboot testvm2    4) Execute live-migration VM1 to host2, check VMs for the device path setting in       each XML.       $ nova live-migration testvm1 host2.localdomain    5) Execute soft reboot VM1, check VMs for the device path setting in each XML.       $ nova reboot testvm1    6) Login to each VMs and check syslogs. Expected result =============== After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are changed to new value fit to destination host. Actual result ============= After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are the value of source host before migration. Environment =========== 1. Exact version of OpenStack you are running. See the following   list for all releases: http://docs.openstack.org/releases/    Reported downstream against stable/train and libvirt 5.6.0-10. 2. Which hypervisor did you use?    (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)    What's the version of that?    libvirt + KVM 2. Which storage type did you use?    (For example: Ceph, LVM, GPFS, ...)    What's the version of that?    LVM/iSCSI with multipath enabled but any host block based storage backend will do. 3. Which networking type did you use?    (For example: nova-network, Neutron with OpenVSwitch, ...)    N/A Logs & Configs ==============    * Device paths initial state                                      host1 host2      ===================================================================================================      VM1 multipath -ll 360014053825c172898b4ba4a5353515c dm-0 ---          virsh dumpxml <source dev='/dev/dm-0' index='1'/> ---          virsh dumpxml --inactive <source dev='/dev/dm-0'/> ---          qemu xml file <source dev='/dev/dm-0'/> ---      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after VM1 live-migration to host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-2' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> <== not dm-2          qemu xml file --- <source dev='/dev/dm-0'/> <== not dm-2      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after soft reboot VM1 on host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> <== changed to dm-0          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * VM1 syslog file before live-migration          $ cat /var/log/messages          ...          Jul 28 05:28:38 cirrostestvm1 kern.info kernel: [ 0.780031] usb 1-1: new full-speed USB device number 2 using uhci_hcd          Jul 28 05:28:39 cirrostestvm1 kern.info kernel: [ 1.272305] Refined TSC clocksource calibration: 2099.976 MHz.          Jul 28 05:28:40 cirrostestvm1 authpriv.info dropbear[260]: Running in background          Jul 28 05:28:40 cirrostestvm1 daemon.info init: reloading /etc/inittab          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/ttyS0': '/sbin/getty -L 115200 ttyS0 vt100 '          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/tty1': '/sbin/getty 115200 tty1'          Jul 28 05:28:48 cirrostestvm1 kern.debug kernel: [ 10.992106] eth0: no IPv6 routers present          Jul 28 05:29:45 cirrostestvm1 authpriv.info dropbear[301]: Child connection from 172.31.151.1:33648          Jul 28 05:29:48 cirrostestvm1 authpriv.notice dropbear[301]: Password auth succeeded for 'cirros' from 172.31.151.1:33648          $    * VM1 syslog file after soft reboot on host2        hostname command return correct value, but VM1 syslog is recorded by VM2.        (in some cases, VM1 and VM2 syslog files are destroyed and cannot be read as text file)          $ hostname          cirrostestvm1          $ cat /var/log/messages | tail          Jul 28 06:03:01 cirrostestvm2 authpriv.info dropbear[325]: Child connection from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.notice dropbear[325]: Password auth succeeded for 'cirros' from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.info dropbear[325]: Exit (cirros): Disconnect received          Jul 28 06:03:30 cirrostestvm2 authpriv.info dropbear[328]: Child connection from 172.31.151.1:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.notice dropbear[328]: Password auth succeeded for 'cirros' from 172.31.151.1:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.info dropbear[328]: Exit (cirros): Disconnect received          Jul 28 06:03:39 cirrostestvm2 authpriv.info dropbear[331]: Child connection from 172.31.151.1:36484          Jul 28 06:03:41 cirrostestvm2 authpriv.info dropbear[331]: Exit before auth (user 'cirros', 0 fails): Exited normally          Jul 28 06:03:45 cirrostestvm2 authpriv.info dropbear[332]: Child connection from 172.31.151.1:36588          Jul 28 06:03:49 cirrostestvm2 authpriv.notice dropbear[332]: Password auth succeeded for 'cirros' from 172.31.151.1:36588
2020-08-05 22:52:32 Jeremy Stanley ossa: status New Incomplete
2020-08-05 22:52:55 Jeremy Stanley bug added subscriber Nova Core security contacts
2020-08-05 23:15:37 Lee Yarwood description This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments. This embargo shall not extend past 2020-11-03 and will be made public by or on that date even if no fix is identified. Description =========== It appears that this is due to Nova not providing VIR_MIGRATE_PARAM_PERSIST_XML during the migration resulting in the original source domains persistent configuration being used instead: /**       * VIR_MIGRATE_PARAM_DEST_XML:       *       * virDomainMigrate* params field: the new configuration to be used for the       * domain on the destination host as VIR_TYPED_PARAM_STRING. The configuration       * must include an identical set of virtual devices, to ensure a stable guest       * ABI across migration. Only parameters related to host side configuration       * can be changed in the XML. Hypervisors which support this field will forbid       * migration if the provided XML would cause a change in the guest ABI. This       * field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain configuration. Using this       * field with hypervisors that do not support changing domain configuration       * during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_DEST_XML "destination_xml"      /**       * VIR_MIGRATE_PARAM_PERSIST_XML:       *       * virDomainMigrate* params field: the new persistent configuration to be used       * for the domain on the destination host as VIR_TYPED_PARAM_STRING.       * This field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain persistent configuration.       * Using this field with hypervisors that do not support changing domain       * configuration during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_PERSIST_XML "persistent_xml" Steps to reproduce ==================    0) Deploy overcloud with multipath and iscsi/LVM cinder backend.    1) Delete all instances and check no device path remained on both host1 and host2.    2) Boot instances, VM1 on host1 and VM2 on host2.       $ cinder create --name cirros1 --volume-type lvm --image cirros 1       $ cinder create --name cirros2 --volume-type lvm --image cirros 1       $ nova boot --block-device-mapping vda=$cirrosvol1 ... --host host1.localdomain testvm1       $ nova boot --block-device-mapping vda=$cirrosvol2 ... --host host2.localdomain testvm2       $ openstack server add floating ip testvm1 xx.xx.xx.xx       $ openstack server add floating ip testvm2 yy.yy.yy.yy    3) Soft reboot each instances and check no problem has occured.       $ nova reboot testvm1       $ nova reboot testvm2    4) Execute live-migration VM1 to host2, check VMs for the device path setting in       each XML.       $ nova live-migration testvm1 host2.localdomain    5) Execute soft reboot VM1, check VMs for the device path setting in each XML.       $ nova reboot testvm1    6) Login to each VMs and check syslogs. Expected result =============== After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are changed to new value fit to destination host. Actual result ============= After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are the value of source host before migration. Environment =========== 1. Exact version of OpenStack you are running. See the following   list for all releases: http://docs.openstack.org/releases/    Reported downstream against stable/train and libvirt 5.6.0-10. 2. Which hypervisor did you use?    (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)    What's the version of that?    libvirt + KVM 2. Which storage type did you use?    (For example: Ceph, LVM, GPFS, ...)    What's the version of that?    LVM/iSCSI with multipath enabled but any host block based storage backend will do. 3. Which networking type did you use?    (For example: nova-network, Neutron with OpenVSwitch, ...)    N/A Logs & Configs ==============    * Device paths initial state                                      host1 host2      ===================================================================================================      VM1 multipath -ll 360014053825c172898b4ba4a5353515c dm-0 ---          virsh dumpxml <source dev='/dev/dm-0' index='1'/> ---          virsh dumpxml --inactive <source dev='/dev/dm-0'/> ---          qemu xml file <source dev='/dev/dm-0'/> ---      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after VM1 live-migration to host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-2' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> <== not dm-2          qemu xml file --- <source dev='/dev/dm-0'/> <== not dm-2      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after soft reboot VM1 on host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> <== changed to dm-0          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * VM1 syslog file before live-migration          $ cat /var/log/messages          ...          Jul 28 05:28:38 cirrostestvm1 kern.info kernel: [ 0.780031] usb 1-1: new full-speed USB device number 2 using uhci_hcd          Jul 28 05:28:39 cirrostestvm1 kern.info kernel: [ 1.272305] Refined TSC clocksource calibration: 2099.976 MHz.          Jul 28 05:28:40 cirrostestvm1 authpriv.info dropbear[260]: Running in background          Jul 28 05:28:40 cirrostestvm1 daemon.info init: reloading /etc/inittab          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/ttyS0': '/sbin/getty -L 115200 ttyS0 vt100 '          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/tty1': '/sbin/getty 115200 tty1'          Jul 28 05:28:48 cirrostestvm1 kern.debug kernel: [ 10.992106] eth0: no IPv6 routers present          Jul 28 05:29:45 cirrostestvm1 authpriv.info dropbear[301]: Child connection from 172.31.151.1:33648          Jul 28 05:29:48 cirrostestvm1 authpriv.notice dropbear[301]: Password auth succeeded for 'cirros' from 172.31.151.1:33648          $    * VM1 syslog file after soft reboot on host2        hostname command return correct value, but VM1 syslog is recorded by VM2.        (in some cases, VM1 and VM2 syslog files are destroyed and cannot be read as text file)          $ hostname          cirrostestvm1          $ cat /var/log/messages | tail          Jul 28 06:03:01 cirrostestvm2 authpriv.info dropbear[325]: Child connection from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.notice dropbear[325]: Password auth succeeded for 'cirros' from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.info dropbear[325]: Exit (cirros): Disconnect received          Jul 28 06:03:30 cirrostestvm2 authpriv.info dropbear[328]: Child connection from 172.31.151.1:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.notice dropbear[328]: Password auth succeeded for 'cirros' from 172.31.151.1:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.info dropbear[328]: Exit (cirros): Disconnect received          Jul 28 06:03:39 cirrostestvm2 authpriv.info dropbear[331]: Child connection from 172.31.151.1:36484          Jul 28 06:03:41 cirrostestvm2 authpriv.info dropbear[331]: Exit before auth (user 'cirros', 0 fails): Exited normally          Jul 28 06:03:45 cirrostestvm2 authpriv.info dropbear[332]: Child connection from 172.31.151.1:36588          Jul 28 06:03:49 cirrostestvm2 authpriv.notice dropbear[332]: Password auth succeeded for 'cirros' from 172.31.151.1:36588 This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments. This embargo shall not extend past 2020-11-03 and will be made public by or on that date even if no fix is identified. Description =========== When live migrating instances with attached volumes Nova will first ensure that the volumes are connected on the destination before updating the underlying domain XML to be used on the destination to correctly map to these volumes. At present in the case of volumes connected over iSCSI or FC this ensures that the instance points to the correct host block devices as these may differ from the source. However if a user requests a soft reboot of an instance after a successful live migration the underlying libvirt domain will rollback to the XML definition used on the source. In the case of volumes provided over iSCSI or FC etc this can potentially lead to the wrong volume being attached to the instance on the destination leading to possible data exfiltration or corruption. It appears that this is due to Nova not providing VIR_MIGRATE_PARAM_PERSIST_XML during the migration resulting in the original source domains persistent configuration being used instead: /**       * VIR_MIGRATE_PARAM_DEST_XML:       *       * virDomainMigrate* params field: the new configuration to be used for the       * domain on the destination host as VIR_TYPED_PARAM_STRING. The configuration       * must include an identical set of virtual devices, to ensure a stable guest       * ABI across migration. Only parameters related to host side configuration       * can be changed in the XML. Hypervisors which support this field will forbid       * migration if the provided XML would cause a change in the guest ABI. This       * field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain configuration. Using this       * field with hypervisors that do not support changing domain configuration       * during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_DEST_XML "destination_xml"      /**       * VIR_MIGRATE_PARAM_PERSIST_XML:       *       * virDomainMigrate* params field: the new persistent configuration to be used       * for the domain on the destination host as VIR_TYPED_PARAM_STRING.       * This field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain persistent configuration.       * Using this field with hypervisors that do not support changing domain       * configuration during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_PERSIST_XML "persistent_xml" Steps to reproduce ==================    0) Deploy overcloud with multipath and iscsi/LVM cinder backend.    1) Delete all instances and check no device path remained on both host1 and host2.    2) Boot instances, VM1 on host1 and VM2 on host2.       $ cinder create --name cirros1 --volume-type lvm --image cirros 1       $ cinder create --name cirros2 --volume-type lvm --image cirros 1       $ nova boot --block-device-mapping vda=$cirrosvol1 ... --host host1.localdomain testvm1       $ nova boot --block-device-mapping vda=$cirrosvol2 ... --host host2.localdomain testvm2       $ openstack server add floating ip testvm1 xx.xx.xx.xx       $ openstack server add floating ip testvm2 yy.yy.yy.yy    3) Soft reboot each instances and check no problem has occured.       $ nova reboot testvm1       $ nova reboot testvm2    4) Execute live-migration VM1 to host2, check VMs for the device path setting in       each XML.       $ nova live-migration testvm1 host2.localdomain    5) Execute soft reboot VM1, check VMs for the device path setting in each XML.       $ nova reboot testvm1    6) Login to each VMs and check syslogs. Expected result =============== After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are changed to new value fit to destination host. Actual result ============= After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are the value of source host before migration. Environment =========== 1. Exact version of OpenStack you are running. See the following   list for all releases: http://docs.openstack.org/releases/    Reported downstream against stable/train and libvirt 5.6.0-10. 2. Which hypervisor did you use?    (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)    What's the version of that?    libvirt + KVM 2. Which storage type did you use?    (For example: Ceph, LVM, GPFS, ...)    What's the version of that?    LVM/iSCSI with multipath enabled but any host block based storage backend will do. 3. Which networking type did you use?    (For example: nova-network, Neutron with OpenVSwitch, ...)    N/A Logs & Configs ==============    * Device paths initial state                                      host1 host2      ===================================================================================================      VM1 multipath -ll 360014053825c172898b4ba4a5353515c dm-0 ---          virsh dumpxml <source dev='/dev/dm-0' index='1'/> ---          virsh dumpxml --inactive <source dev='/dev/dm-0'/> ---          qemu xml file <source dev='/dev/dm-0'/> ---      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after VM1 live-migration to host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-2' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> <== not dm-2          qemu xml file --- <source dev='/dev/dm-0'/> <== not dm-2      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after soft reboot VM1 on host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> <== changed to dm-0          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * VM1 syslog file before live-migration          $ cat /var/log/messages          ...          Jul 28 05:28:38 cirrostestvm1 kern.info kernel: [ 0.780031] usb 1-1: new full-speed USB device number 2 using uhci_hcd          Jul 28 05:28:39 cirrostestvm1 kern.info kernel: [ 1.272305] Refined TSC clocksource calibration: 2099.976 MHz.          Jul 28 05:28:40 cirrostestvm1 authpriv.info dropbear[260]: Running in background          Jul 28 05:28:40 cirrostestvm1 daemon.info init: reloading /etc/inittab          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/ttyS0': '/sbin/getty -L 115200 ttyS0 vt100 '          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/tty1': '/sbin/getty 115200 tty1'          Jul 28 05:28:48 cirrostestvm1 kern.debug kernel: [ 10.992106] eth0: no IPv6 routers present          Jul 28 05:29:45 cirrostestvm1 authpriv.info dropbear[301]: Child connection from 172.31.151.1:33648          Jul 28 05:29:48 cirrostestvm1 authpriv.notice dropbear[301]: Password auth succeeded for 'cirros' from 172.31.151.1:33648          $    * VM1 syslog file after soft reboot on host2        hostname command return correct value, but VM1 syslog is recorded by VM2.        (in some cases, VM1 and VM2 syslog files are destroyed and cannot be read as text file)          $ hostname          cirrostestvm1          $ cat /var/log/messages | tail          Jul 28 06:03:01 cirrostestvm2 authpriv.info dropbear[325]: Child connection from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.notice dropbear[325]: Password auth succeeded for 'cirros' from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.info dropbear[325]: Exit (cirros): Disconnect received          Jul 28 06:03:30 cirrostestvm2 authpriv.info dropbear[328]: Child connection from 172.31.151.1:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.notice dropbear[328]: Password auth succeeded for 'cirros' from 172.31.151.1:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.info dropbear[328]: Exit (cirros): Disconnect received          Jul 28 06:03:39 cirrostestvm2 authpriv.info dropbear[331]: Child connection from 172.31.151.1:36484          Jul 28 06:03:41 cirrostestvm2 authpriv.info dropbear[331]: Exit before auth (user 'cirros', 0 fails): Exited normally          Jul 28 06:03:45 cirrostestvm2 authpriv.info dropbear[332]: Child connection from 172.31.151.1:36588          Jul 28 06:03:49 cirrostestvm2 authpriv.notice dropbear[332]: Password auth succeeded for 'cirros' from 172.31.151.1:36588
2020-08-05 23:19:59 Lee Yarwood description This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments. This embargo shall not extend past 2020-11-03 and will be made public by or on that date even if no fix is identified. Description =========== When live migrating instances with attached volumes Nova will first ensure that the volumes are connected on the destination before updating the underlying domain XML to be used on the destination to correctly map to these volumes. At present in the case of volumes connected over iSCSI or FC this ensures that the instance points to the correct host block devices as these may differ from the source. However if a user requests a soft reboot of an instance after a successful live migration the underlying libvirt domain will rollback to the XML definition used on the source. In the case of volumes provided over iSCSI or FC etc this can potentially lead to the wrong volume being attached to the instance on the destination leading to possible data exfiltration or corruption. It appears that this is due to Nova not providing VIR_MIGRATE_PARAM_PERSIST_XML during the migration resulting in the original source domains persistent configuration being used instead: /**       * VIR_MIGRATE_PARAM_DEST_XML:       *       * virDomainMigrate* params field: the new configuration to be used for the       * domain on the destination host as VIR_TYPED_PARAM_STRING. The configuration       * must include an identical set of virtual devices, to ensure a stable guest       * ABI across migration. Only parameters related to host side configuration       * can be changed in the XML. Hypervisors which support this field will forbid       * migration if the provided XML would cause a change in the guest ABI. This       * field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain configuration. Using this       * field with hypervisors that do not support changing domain configuration       * during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_DEST_XML "destination_xml"      /**       * VIR_MIGRATE_PARAM_PERSIST_XML:       *       * virDomainMigrate* params field: the new persistent configuration to be used       * for the domain on the destination host as VIR_TYPED_PARAM_STRING.       * This field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain persistent configuration.       * Using this field with hypervisors that do not support changing domain       * configuration during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_PERSIST_XML "persistent_xml" Steps to reproduce ==================    0) Deploy overcloud with multipath and iscsi/LVM cinder backend.    1) Delete all instances and check no device path remained on both host1 and host2.    2) Boot instances, VM1 on host1 and VM2 on host2.       $ cinder create --name cirros1 --volume-type lvm --image cirros 1       $ cinder create --name cirros2 --volume-type lvm --image cirros 1       $ nova boot --block-device-mapping vda=$cirrosvol1 ... --host host1.localdomain testvm1       $ nova boot --block-device-mapping vda=$cirrosvol2 ... --host host2.localdomain testvm2       $ openstack server add floating ip testvm1 xx.xx.xx.xx       $ openstack server add floating ip testvm2 yy.yy.yy.yy    3) Soft reboot each instances and check no problem has occured.       $ nova reboot testvm1       $ nova reboot testvm2    4) Execute live-migration VM1 to host2, check VMs for the device path setting in       each XML.       $ nova live-migration testvm1 host2.localdomain    5) Execute soft reboot VM1, check VMs for the device path setting in each XML.       $ nova reboot testvm1    6) Login to each VMs and check syslogs. Expected result =============== After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are changed to new value fit to destination host. Actual result ============= After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are the value of source host before migration. Environment =========== 1. Exact version of OpenStack you are running. See the following   list for all releases: http://docs.openstack.org/releases/    Reported downstream against stable/train and libvirt 5.6.0-10. 2. Which hypervisor did you use?    (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)    What's the version of that?    libvirt + KVM 2. Which storage type did you use?    (For example: Ceph, LVM, GPFS, ...)    What's the version of that?    LVM/iSCSI with multipath enabled but any host block based storage backend will do. 3. Which networking type did you use?    (For example: nova-network, Neutron with OpenVSwitch, ...)    N/A Logs & Configs ==============    * Device paths initial state                                      host1 host2      ===================================================================================================      VM1 multipath -ll 360014053825c172898b4ba4a5353515c dm-0 ---          virsh dumpxml <source dev='/dev/dm-0' index='1'/> ---          virsh dumpxml --inactive <source dev='/dev/dm-0'/> ---          qemu xml file <source dev='/dev/dm-0'/> ---      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after VM1 live-migration to host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-2' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> <== not dm-2          qemu xml file --- <source dev='/dev/dm-0'/> <== not dm-2      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after soft reboot VM1 on host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> <== changed to dm-0          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * VM1 syslog file before live-migration          $ cat /var/log/messages          ...          Jul 28 05:28:38 cirrostestvm1 kern.info kernel: [ 0.780031] usb 1-1: new full-speed USB device number 2 using uhci_hcd          Jul 28 05:28:39 cirrostestvm1 kern.info kernel: [ 1.272305] Refined TSC clocksource calibration: 2099.976 MHz.          Jul 28 05:28:40 cirrostestvm1 authpriv.info dropbear[260]: Running in background          Jul 28 05:28:40 cirrostestvm1 daemon.info init: reloading /etc/inittab          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/ttyS0': '/sbin/getty -L 115200 ttyS0 vt100 '          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/tty1': '/sbin/getty 115200 tty1'          Jul 28 05:28:48 cirrostestvm1 kern.debug kernel: [ 10.992106] eth0: no IPv6 routers present          Jul 28 05:29:45 cirrostestvm1 authpriv.info dropbear[301]: Child connection from 172.31.151.1:33648          Jul 28 05:29:48 cirrostestvm1 authpriv.notice dropbear[301]: Password auth succeeded for 'cirros' from 172.31.151.1:33648          $    * VM1 syslog file after soft reboot on host2        hostname command return correct value, but VM1 syslog is recorded by VM2.        (in some cases, VM1 and VM2 syslog files are destroyed and cannot be read as text file)          $ hostname          cirrostestvm1          $ cat /var/log/messages | tail          Jul 28 06:03:01 cirrostestvm2 authpriv.info dropbear[325]: Child connection from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.notice dropbear[325]: Password auth succeeded for 'cirros' from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.info dropbear[325]: Exit (cirros): Disconnect received          Jul 28 06:03:30 cirrostestvm2 authpriv.info dropbear[328]: Child connection from 172.31.151.1:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.notice dropbear[328]: Password auth succeeded for 'cirros' from 172.31.151.1:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.info dropbear[328]: Exit (cirros): Disconnect received          Jul 28 06:03:39 cirrostestvm2 authpriv.info dropbear[331]: Child connection from 172.31.151.1:36484          Jul 28 06:03:41 cirrostestvm2 authpriv.info dropbear[331]: Exit before auth (user 'cirros', 0 fails): Exited normally          Jul 28 06:03:45 cirrostestvm2 authpriv.info dropbear[332]: Child connection from 172.31.151.1:36588          Jul 28 06:03:49 cirrostestvm2 authpriv.notice dropbear[332]: Password auth succeeded for 'cirros' from 172.31.151.1:36588 This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments. This embargo shall not extend past 2020-11-03 and will be made public by or on that date even if no fix is identified. Description =========== When live migrating instances with attached volumes Nova will first ensure that the volumes are connected on the destination before updating the underlying domain XML to be used on the destination to correctly map to these volumes. At present in the case of volumes connected over iSCSI or FC this ensures that the instance points to the correct host block devices as these may differ from the source. However if a user requests a soft reboot of an instance after a successful live migration the underlying libvirt domain will rollback to the XML definition used on the source. In the case of volumes provided over iSCSI or FC etc this can potentially lead to the wrong  volume being attached to the instance on the destination leading to possible data exfiltration or corruption. It appears that this is due to Nova not providing VIR_MIGRATE_PARAM_PERSIST_XML during the migration resulting in the original source domains persistent configuration being used instead: /**       * VIR_MIGRATE_PARAM_DEST_XML:       *       * virDomainMigrate* params field: the new configuration to be used for the       * domain on the destination host as VIR_TYPED_PARAM_STRING. The configuration       * must include an identical set of virtual devices, to ensure a stable guest       * ABI across migration. Only parameters related to host side configuration       * can be changed in the XML. Hypervisors which support this field will forbid       * migration if the provided XML would cause a change in the guest ABI. This       * field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain configuration. Using this       * field with hypervisors that do not support changing domain configuration       * during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_DEST_XML "destination_xml"      /**       * VIR_MIGRATE_PARAM_PERSIST_XML:       *       * virDomainMigrate* params field: the new persistent configuration to be used       * for the domain on the destination host as VIR_TYPED_PARAM_STRING.       * This field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain persistent configuration.       * Using this field with hypervisors that do not support changing domain       * configuration during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_PERSIST_XML "persistent_xml" Steps to reproduce ==================    0) Deploy overcloud with multipath and iscsi/LVM cinder backend.    1) Delete all instances and check no device path remained on both host1 and host2.    2) Boot instances, VM1 on host1 and VM2 on host2.       $ cinder create --name cirros1 --volume-type lvm --image cirros 1       $ cinder create --name cirros2 --volume-type lvm --image cirros 1       $ nova boot --block-device-mapping vda=$cirrosvol1 ... --host host1.localdomain testvm1       $ nova boot --block-device-mapping vda=$cirrosvol2 ... --host host2.localdomain testvm2       $ openstack server add floating ip testvm1 xx.xx.xx.xx       $ openstack server add floating ip testvm2 yy.yy.yy.yy    3) Soft reboot each instances and check no problem has occured.       $ nova reboot testvm1       $ nova reboot testvm2    4) Execute live-migration VM1 to host2, check VMs for the device path setting in       each XML.       $ nova live-migration testvm1 host2.localdomain    5) Execute soft reboot VM1, check VMs for the device path setting in each XML.       $ nova reboot testvm1    6) Login to each VMs and check syslogs. Expected result =============== After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are changed to new value fit to destination host. Actual result ============= After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are the value of source host before migration. Environment =========== 1. Exact version of OpenStack you are running. See the following   list for all releases: http://docs.openstack.org/releases/    Reported downstream against stable/train and libvirt 5.6.0-10. 2. Which hypervisor did you use?    (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)    What's the version of that?    libvirt + KVM 2. Which storage type did you use?    (For example: Ceph, LVM, GPFS, ...)    What's the version of that?    LVM/iSCSI with multipath enabled but any host block based storage backend will do. 3. Which networking type did you use?    (For example: nova-network, Neutron with OpenVSwitch, ...)    N/A Logs & Configs ============== The following test env logs are copied verbatim from a private downstream security bug: https://bugzilla.redhat.com/show_bug.cgi?id=1862353    * Device paths initial state                                      host1 host2      ===================================================================================================      VM1 multipath -ll 360014053825c172898b4ba4a5353515c dm-0 ---          virsh dumpxml <source dev='/dev/dm-0' index='1'/> ---          virsh dumpxml --inactive <source dev='/dev/dm-0'/> ---          qemu xml file <source dev='/dev/dm-0'/> ---      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after VM1 live-migration to host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-2' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> <== not dm-2          qemu xml file --- <source dev='/dev/dm-0'/> <== not dm-2      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after soft reboot VM1 on host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> <== changed to dm-0          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * VM1 syslog file before live-migration          $ cat /var/log/messages          ...          Jul 28 05:28:38 cirrostestvm1 kern.info kernel: [ 0.780031] usb 1-1: new full-speed USB device number 2 using uhci_hcd          Jul 28 05:28:39 cirrostestvm1 kern.info kernel: [ 1.272305] Refined TSC clocksource calibration: 2099.976 MHz.          Jul 28 05:28:40 cirrostestvm1 authpriv.info dropbear[260]: Running in background          Jul 28 05:28:40 cirrostestvm1 daemon.info init: reloading /etc/inittab          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/ttyS0': '/sbin/getty -L 115200 ttyS0 vt100 '          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/tty1': '/sbin/getty 115200 tty1'          Jul 28 05:28:48 cirrostestvm1 kern.debug kernel: [ 10.992106] eth0: no IPv6 routers present          Jul 28 05:29:45 cirrostestvm1 authpriv.info dropbear[301]: Child connection from **.**.**.**:33648          Jul 28 05:29:48 cirrostestvm1 authpriv.notice dropbear[301]: Password auth succeeded for 'cirros' from **.**.**.**:33648          $    * VM1 syslog file after soft reboot on host2        hostname command return correct value, but VM1 syslog is recorded by VM2.        (in some cases, VM1 and VM2 syslog files are destroyed and cannot be read as text file)          $ hostname          cirrostestvm1          $ cat /var/log/messages | tail          Jul 28 06:03:01 cirrostestvm2 authpriv.info dropbear[325]: Child connection from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.notice dropbear[325]: Password auth succeeded for 'cirros' from **.**.**.**:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.info dropbear[325]: Exit (cirros): Disconnect received          Jul 28 06:03:30 cirrostestvm2 authpriv.info dropbear[328]: Child connection from **.**.**.**:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.notice dropbear[328]: Password auth succeeded for 'cirros' from **.**.**.**:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.info dropbear[328]: Exit (cirros): Disconnect received          Jul 28 06:03:39 cirrostestvm2 authpriv.info dropbear[331]: Child connection from **.**.**.**:36484          Jul 28 06:03:41 cirrostestvm2 authpriv.info dropbear[331]: Exit before auth (user 'cirros', 0 fails): Exited normally          Jul 28 06:03:45 cirrostestvm2 authpriv.info dropbear[332]: Child connection from **.**.**.**:36588          Jul 28 06:03:49 cirrostestvm2 authpriv.notice dropbear[332]: Password auth succeeded for 'cirros' from **.**.**.**:36588
2020-08-06 16:31:09 Lee Yarwood attachment added 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-during.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5399347/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-during.patch
2020-08-06 17:50:52 Jeremy Stanley ossa: status Incomplete Confirmed
2020-08-06 21:13:49 Jeremy Stanley ossa: status Confirmed Triaged
2020-08-07 14:53:42 Jeremy Stanley cve linked 2020-17376
2020-08-07 14:54:09 Jeremy Stanley summary Soft reboot after live-migration reverts instance to original source domain XML Soft reboot after live-migration reverts instance to original source domain XML (CVE-2020-17376)
2020-08-07 21:03:47 Lee Yarwood attachment removed 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-during.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5399347/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-during.patch
2020-08-07 21:38:16 Lee Yarwood attachment added 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-master.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5399636/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-master.patch
2020-08-07 21:38:31 Lee Yarwood attachment added 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-ussuri.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5399637/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-ussuri.patch
2020-08-07 21:38:47 Lee Yarwood attachment added 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-ussuri.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5399638/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-ussuri.patch
2020-08-07 21:38:57 Lee Yarwood attachment removed 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-ussuri.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5399638/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-ussuri.patch
2020-08-07 21:39:16 Lee Yarwood attachment added 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-train.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5399639/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-train.patch
2020-08-07 21:39:32 Lee Yarwood attachment added 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-stein.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5399640/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-stein.patch
2020-08-07 22:10:40 Lee Yarwood attachment added 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-rocky.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5399649/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-rocky.patch
2020-08-07 22:31:46 Lee Yarwood attachment added 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-queens.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5399650/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-queens.patch
2020-08-10 18:52:51 Lee Yarwood attachment removed 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-rocky.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5399649/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-rocky.patch
2020-08-10 18:52:58 Lee Yarwood attachment removed 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-queens.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5399650/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-queens.patch
2020-08-10 19:18:57 Lee Yarwood attachment added 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-rocky.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5400503/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-rocky.patch
2020-08-10 19:19:14 Lee Yarwood attachment added 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-queens.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5400504/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-queens.patch
2020-08-10 19:19:49 Lee Yarwood bug added subscriber melanie witt
2020-08-10 19:20:06 Lee Yarwood bug added subscriber Stephen Finucane
2020-08-10 19:28:55 Lee Yarwood bug added subscriber Nick Tait
2020-08-19 13:24:13 Jeremy Stanley bug added subscriber Mohammed Naser
2020-08-19 14:32:23 Lee Yarwood attachment added 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-pike.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5402795/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-pike.patch
2020-08-19 14:59:10 Jeremy Stanley bug added subscriber Robert Frohl
2020-08-20 15:31:07 Jeremy Stanley bug added subscriber Mark DeVerter
2020-08-25 14:45:50 Jeremy Stanley description This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments. This embargo shall not extend past 2020-11-03 and will be made public by or on that date even if no fix is identified. Description =========== When live migrating instances with attached volumes Nova will first ensure that the volumes are connected on the destination before updating the underlying domain XML to be used on the destination to correctly map to these volumes. At present in the case of volumes connected over iSCSI or FC this ensures that the instance points to the correct host block devices as these may differ from the source. However if a user requests a soft reboot of an instance after a successful live migration the underlying libvirt domain will rollback to the XML definition used on the source. In the case of volumes provided over iSCSI or FC etc this can potentially lead to the wrong  volume being attached to the instance on the destination leading to possible data exfiltration or corruption. It appears that this is due to Nova not providing VIR_MIGRATE_PARAM_PERSIST_XML during the migration resulting in the original source domains persistent configuration being used instead: /**       * VIR_MIGRATE_PARAM_DEST_XML:       *       * virDomainMigrate* params field: the new configuration to be used for the       * domain on the destination host as VIR_TYPED_PARAM_STRING. The configuration       * must include an identical set of virtual devices, to ensure a stable guest       * ABI across migration. Only parameters related to host side configuration       * can be changed in the XML. Hypervisors which support this field will forbid       * migration if the provided XML would cause a change in the guest ABI. This       * field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain configuration. Using this       * field with hypervisors that do not support changing domain configuration       * during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_DEST_XML "destination_xml"      /**       * VIR_MIGRATE_PARAM_PERSIST_XML:       *       * virDomainMigrate* params field: the new persistent configuration to be used       * for the domain on the destination host as VIR_TYPED_PARAM_STRING.       * This field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain persistent configuration.       * Using this field with hypervisors that do not support changing domain       * configuration during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_PERSIST_XML "persistent_xml" Steps to reproduce ==================    0) Deploy overcloud with multipath and iscsi/LVM cinder backend.    1) Delete all instances and check no device path remained on both host1 and host2.    2) Boot instances, VM1 on host1 and VM2 on host2.       $ cinder create --name cirros1 --volume-type lvm --image cirros 1       $ cinder create --name cirros2 --volume-type lvm --image cirros 1       $ nova boot --block-device-mapping vda=$cirrosvol1 ... --host host1.localdomain testvm1       $ nova boot --block-device-mapping vda=$cirrosvol2 ... --host host2.localdomain testvm2       $ openstack server add floating ip testvm1 xx.xx.xx.xx       $ openstack server add floating ip testvm2 yy.yy.yy.yy    3) Soft reboot each instances and check no problem has occured.       $ nova reboot testvm1       $ nova reboot testvm2    4) Execute live-migration VM1 to host2, check VMs for the device path setting in       each XML.       $ nova live-migration testvm1 host2.localdomain    5) Execute soft reboot VM1, check VMs for the device path setting in each XML.       $ nova reboot testvm1    6) Login to each VMs and check syslogs. Expected result =============== After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are changed to new value fit to destination host. Actual result ============= After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are the value of source host before migration. Environment =========== 1. Exact version of OpenStack you are running. See the following   list for all releases: http://docs.openstack.org/releases/    Reported downstream against stable/train and libvirt 5.6.0-10. 2. Which hypervisor did you use?    (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)    What's the version of that?    libvirt + KVM 2. Which storage type did you use?    (For example: Ceph, LVM, GPFS, ...)    What's the version of that?    LVM/iSCSI with multipath enabled but any host block based storage backend will do. 3. Which networking type did you use?    (For example: nova-network, Neutron with OpenVSwitch, ...)    N/A Logs & Configs ============== The following test env logs are copied verbatim from a private downstream security bug: https://bugzilla.redhat.com/show_bug.cgi?id=1862353    * Device paths initial state                                      host1 host2      ===================================================================================================      VM1 multipath -ll 360014053825c172898b4ba4a5353515c dm-0 ---          virsh dumpxml <source dev='/dev/dm-0' index='1'/> ---          virsh dumpxml --inactive <source dev='/dev/dm-0'/> ---          qemu xml file <source dev='/dev/dm-0'/> ---      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after VM1 live-migration to host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-2' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> <== not dm-2          qemu xml file --- <source dev='/dev/dm-0'/> <== not dm-2      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after soft reboot VM1 on host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> <== changed to dm-0          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * VM1 syslog file before live-migration          $ cat /var/log/messages          ...          Jul 28 05:28:38 cirrostestvm1 kern.info kernel: [ 0.780031] usb 1-1: new full-speed USB device number 2 using uhci_hcd          Jul 28 05:28:39 cirrostestvm1 kern.info kernel: [ 1.272305] Refined TSC clocksource calibration: 2099.976 MHz.          Jul 28 05:28:40 cirrostestvm1 authpriv.info dropbear[260]: Running in background          Jul 28 05:28:40 cirrostestvm1 daemon.info init: reloading /etc/inittab          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/ttyS0': '/sbin/getty -L 115200 ttyS0 vt100 '          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/tty1': '/sbin/getty 115200 tty1'          Jul 28 05:28:48 cirrostestvm1 kern.debug kernel: [ 10.992106] eth0: no IPv6 routers present          Jul 28 05:29:45 cirrostestvm1 authpriv.info dropbear[301]: Child connection from **.**.**.**:33648          Jul 28 05:29:48 cirrostestvm1 authpriv.notice dropbear[301]: Password auth succeeded for 'cirros' from **.**.**.**:33648          $    * VM1 syslog file after soft reboot on host2        hostname command return correct value, but VM1 syslog is recorded by VM2.        (in some cases, VM1 and VM2 syslog files are destroyed and cannot be read as text file)          $ hostname          cirrostestvm1          $ cat /var/log/messages | tail          Jul 28 06:03:01 cirrostestvm2 authpriv.info dropbear[325]: Child connection from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.notice dropbear[325]: Password auth succeeded for 'cirros' from **.**.**.**:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.info dropbear[325]: Exit (cirros): Disconnect received          Jul 28 06:03:30 cirrostestvm2 authpriv.info dropbear[328]: Child connection from **.**.**.**:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.notice dropbear[328]: Password auth succeeded for 'cirros' from **.**.**.**:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.info dropbear[328]: Exit (cirros): Disconnect received          Jul 28 06:03:39 cirrostestvm2 authpriv.info dropbear[331]: Child connection from **.**.**.**:36484          Jul 28 06:03:41 cirrostestvm2 authpriv.info dropbear[331]: Exit before auth (user 'cirros', 0 fails): Exited normally          Jul 28 06:03:45 cirrostestvm2 authpriv.info dropbear[332]: Child connection from **.**.**.**:36588          Jul 28 06:03:49 cirrostestvm2 authpriv.notice dropbear[332]: Password auth succeeded for 'cirros' from **.**.**.**:36588 Description =========== When live migrating instances with attached volumes Nova will first ensure that the volumes are connected on the destination before updating the underlying domain XML to be used on the destination to correctly map to these volumes. At present in the case of volumes connected over iSCSI or FC this ensures that the instance points to the correct host block devices as these may differ from the source. However if a user requests a soft reboot of an instance after a successful live migration the underlying libvirt domain will rollback to the XML definition used on the source. In the case of volumes provided over iSCSI or FC etc this can potentially lead to the wrong  volume being attached to the instance on the destination leading to possible data exfiltration or corruption. It appears that this is due to Nova not providing VIR_MIGRATE_PARAM_PERSIST_XML during the migration resulting in the original source domains persistent configuration being used instead: /**       * VIR_MIGRATE_PARAM_DEST_XML:       *       * virDomainMigrate* params field: the new configuration to be used for the       * domain on the destination host as VIR_TYPED_PARAM_STRING. The configuration       * must include an identical set of virtual devices, to ensure a stable guest       * ABI across migration. Only parameters related to host side configuration       * can be changed in the XML. Hypervisors which support this field will forbid       * migration if the provided XML would cause a change in the guest ABI. This       * field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain configuration. Using this       * field with hypervisors that do not support changing domain configuration       * during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_DEST_XML "destination_xml"      /**       * VIR_MIGRATE_PARAM_PERSIST_XML:       *       * virDomainMigrate* params field: the new persistent configuration to be used       * for the domain on the destination host as VIR_TYPED_PARAM_STRING.       * This field cannot be used to rename the domain during migration (use       * VIR_MIGRATE_PARAM_DEST_NAME field for that purpose). Domain name in the       * destination XML must match the original domain name.       *       * Omitting this parameter keeps the original domain persistent configuration.       * Using this field with hypervisors that do not support changing domain       * configuration during migration will result in a failure.       */      # define VIR_MIGRATE_PARAM_PERSIST_XML "persistent_xml" Steps to reproduce ==================    0) Deploy overcloud with multipath and iscsi/LVM cinder backend.    1) Delete all instances and check no device path remained on both host1 and host2.    2) Boot instances, VM1 on host1 and VM2 on host2.       $ cinder create --name cirros1 --volume-type lvm --image cirros 1       $ cinder create --name cirros2 --volume-type lvm --image cirros 1       $ nova boot --block-device-mapping vda=$cirrosvol1 ... --host host1.localdomain testvm1       $ nova boot --block-device-mapping vda=$cirrosvol2 ... --host host2.localdomain testvm2       $ openstack server add floating ip testvm1 xx.xx.xx.xx       $ openstack server add floating ip testvm2 yy.yy.yy.yy    3) Soft reboot each instances and check no problem has occured.       $ nova reboot testvm1       $ nova reboot testvm2    4) Execute live-migration VM1 to host2, check VMs for the device path setting in       each XML.       $ nova live-migration testvm1 host2.localdomain    5) Execute soft reboot VM1, check VMs for the device path setting in each XML.       $ nova reboot testvm1    6) Login to each VMs and check syslogs. Expected result =============== After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are changed to new value fit to destination host. Actual result ============= After live-migration and soft reboot instance, device paths indicated by virsh dumpxml --inactive and qemu XML file are the value of source host before migration. Environment =========== 1. Exact version of OpenStack you are running. See the following   list for all releases: http://docs.openstack.org/releases/    Reported downstream against stable/train and libvirt 5.6.0-10. 2. Which hypervisor did you use?    (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)    What's the version of that?    libvirt + KVM 2. Which storage type did you use?    (For example: Ceph, LVM, GPFS, ...)    What's the version of that?    LVM/iSCSI with multipath enabled but any host block based storage backend will do. 3. Which networking type did you use?    (For example: nova-network, Neutron with OpenVSwitch, ...)    N/A Logs & Configs ============== The following test env logs are copied verbatim from a private downstream security bug: https://bugzilla.redhat.com/show_bug.cgi?id=1862353    * Device paths initial state                                      host1 host2      ===================================================================================================      VM1 multipath -ll 360014053825c172898b4ba4a5353515c dm-0 ---          virsh dumpxml <source dev='/dev/dm-0' index='1'/> ---          virsh dumpxml --inactive <source dev='/dev/dm-0'/> ---          qemu xml file <source dev='/dev/dm-0'/> ---      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after VM1 live-migration to host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-2' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/> <== not dm-2          qemu xml file --- <source dev='/dev/dm-0'/> <== not dm-2      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * Device paths after soft reboot VM1 on host2                                      host1 host2      ===================================================================================================      VM1 multipath -ll --- 360014053825c172898b4ba4a5353515c dm-2          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/> <== changed to dm-0          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>      ---------------------------------------------------------------------------------------------------      VM2 multipath -ll --- 36001405fc681536d0124af2a9fd99c10 dm-0          virsh dumpxml --- <source dev='/dev/dm-0' index='1'/>          virsh dumpxml --inactive --- <source dev='/dev/dm-0'/>          qemu xml file --- <source dev='/dev/dm-0'/>    * VM1 syslog file before live-migration          $ cat /var/log/messages          ...          Jul 28 05:28:38 cirrostestvm1 kern.info kernel: [ 0.780031] usb 1-1: new full-speed USB device number 2 using uhci_hcd          Jul 28 05:28:39 cirrostestvm1 kern.info kernel: [ 1.272305] Refined TSC clocksource calibration: 2099.976 MHz.          Jul 28 05:28:40 cirrostestvm1 authpriv.info dropbear[260]: Running in background          Jul 28 05:28:40 cirrostestvm1 daemon.info init: reloading /etc/inittab          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/ttyS0': '/sbin/getty -L 115200 ttyS0 vt100 '          Jul 28 05:28:40 cirrostestvm1 daemon.info init: starting pid 1, tty '/dev/tty1': '/sbin/getty 115200 tty1'          Jul 28 05:28:48 cirrostestvm1 kern.debug kernel: [ 10.992106] eth0: no IPv6 routers present          Jul 28 05:29:45 cirrostestvm1 authpriv.info dropbear[301]: Child connection from **.**.**.**:33648          Jul 28 05:29:48 cirrostestvm1 authpriv.notice dropbear[301]: Password auth succeeded for 'cirros' from **.**.**.**:33648          $    * VM1 syslog file after soft reboot on host2        hostname command return correct value, but VM1 syslog is recorded by VM2.        (in some cases, VM1 and VM2 syslog files are destroyed and cannot be read as text file)          $ hostname          cirrostestvm1          $ cat /var/log/messages | tail          Jul 28 06:03:01 cirrostestvm2 authpriv.info dropbear[325]: Child connection from 172.31.151.1:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.notice dropbear[325]: Password auth succeeded for 'cirros' from **.**.**.**:35894          Jul 28 06:03:05 cirrostestvm2 authpriv.info dropbear[325]: Exit (cirros): Disconnect received          Jul 28 06:03:30 cirrostestvm2 authpriv.info dropbear[328]: Child connection from **.**.**.**:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.notice dropbear[328]: Password auth succeeded for 'cirros' from **.**.**.**:36352          Jul 28 06:03:34 cirrostestvm2 authpriv.info dropbear[328]: Exit (cirros): Disconnect received          Jul 28 06:03:39 cirrostestvm2 authpriv.info dropbear[331]: Child connection from **.**.**.**:36484          Jul 28 06:03:41 cirrostestvm2 authpriv.info dropbear[331]: Exit before auth (user 'cirros', 0 fails): Exited normally          Jul 28 06:03:45 cirrostestvm2 authpriv.info dropbear[332]: Child connection from **.**.**.**:36588          Jul 28 06:03:49 cirrostestvm2 authpriv.notice dropbear[332]: Password auth succeeded for 'cirros' from **.**.**.**:36588
2020-08-25 14:45:57 Jeremy Stanley information type Private Security Public Security
2020-08-25 14:46:37 Jeremy Stanley ossa: status Triaged In Progress
2020-08-25 15:00:37 OpenStack Infra nova: status New In Progress
2020-08-25 15:00:37 OpenStack Infra nova: assignee Lee Yarwood (lyarwood)
2020-08-25 20:18:05 OpenStack Infra nova: status In Progress Fix Released
2020-08-26 09:45:56 OpenStack Infra tags libvirt live-migration security in-stable-ussuri libvirt live-migration security
2020-08-26 10:22:46 Lee Yarwood attachment added 0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-newton.patch https://bugs.launchpad.net/nova/+bug/1890501/+attachment/5404693/+files/0001-libvirt-Provide-VIR_MIGRATE_PARAM_PERSIST_XML-newton.patch
2020-08-26 12:13:47 Lee Yarwood nominated for series nova/queens
2020-08-26 12:13:47 Lee Yarwood bug task added nova/queens
2020-08-26 12:13:47 Lee Yarwood nominated for series nova/ussuri
2020-08-26 12:13:47 Lee Yarwood bug task added nova/ussuri
2020-08-26 12:13:47 Lee Yarwood nominated for series nova/train
2020-08-26 12:13:47 Lee Yarwood bug task added nova/train
2020-08-26 12:13:47 Lee Yarwood nominated for series nova/stein
2020-08-26 12:13:47 Lee Yarwood bug task added nova/stein
2020-08-26 12:13:47 Lee Yarwood nominated for series nova/victoria
2020-08-26 12:13:47 Lee Yarwood bug task added nova/victoria
2020-08-26 12:13:47 Lee Yarwood nominated for series nova/rocky
2020-08-26 12:13:47 Lee Yarwood bug task added nova/rocky
2020-08-26 12:13:59 Lee Yarwood nova/ussuri: assignee Lee Yarwood (lyarwood)
2020-08-26 12:14:02 Lee Yarwood nova/train: assignee Lee Yarwood (lyarwood)
2020-08-26 12:14:06 Lee Yarwood nova/stein: assignee Lee Yarwood (lyarwood)
2020-08-26 12:14:10 Lee Yarwood nova/queens: assignee Lee Yarwood (lyarwood)
2020-08-26 12:14:13 Lee Yarwood nova/rocky: assignee Lee Yarwood (lyarwood)
2020-08-26 12:14:18 Lee Yarwood nova/ussuri: importance Undecided Critical
2020-08-26 12:14:21 Lee Yarwood nova/stein: importance Undecided Critical
2020-08-26 12:14:28 Lee Yarwood nova/victoria: importance Undecided Critical
2020-08-26 12:14:32 Lee Yarwood nova/queens: importance Undecided Critical
2020-08-26 12:14:37 Lee Yarwood nova/rocky: importance Undecided Critical
2020-08-26 12:14:41 Lee Yarwood nova/train: importance Undecided Critical
2020-08-26 12:14:57 Lee Yarwood nova/ussuri: status New Fix Released
2020-08-26 12:15:01 Lee Yarwood nova/train: status New In Progress
2020-08-26 12:15:06 Lee Yarwood nova/queens: status New In Progress
2020-08-26 12:15:09 Lee Yarwood nova/rocky: status New In Progress
2020-08-26 12:15:13 Lee Yarwood nova/stein: status New In Progress
2020-08-26 17:03:03 OpenStack Infra nova/train: status In Progress Fix Committed
2020-08-28 09:40:34 OpenStack Infra nova/stein: status In Progress Fix Committed
2020-08-30 15:17:59 Jeremy Stanley ossa: assignee Jeremy Stanley (fungi)
2020-08-30 15:18:09 Jeremy Stanley ossa: importance Undecided High
2020-08-30 15:18:14 Jeremy Stanley ossa: status In Progress Fix Released
2020-08-30 15:18:42 Jeremy Stanley summary Soft reboot after live-migration reverts instance to original source domain XML (CVE-2020-17376) [OSSA-2020-006] Soft reboot after live-migration reverts instance to original source domain XML (CVE-2020-17376)
2020-09-02 01:54:45 melanie witt nominated for series nova/pike
2020-09-02 01:54:45 melanie witt bug task added nova/pike
2020-09-02 01:54:55 melanie witt nova/pike: importance Undecided Critical
2020-09-03 15:15:52 OpenStack Infra nova/rocky: status In Progress Fix Committed
2020-09-18 15:04:08 OpenStack Infra nova/queens: assignee Lee Yarwood (lyarwood) Elod Illes (elod-illes)
2020-09-19 01:42:31 OpenStack Infra nova/queens: status In Progress Fix Committed
2020-09-21 08:23:21 OpenStack Infra nova/pike: status New In Progress
2020-09-21 08:23:21 OpenStack Infra nova/pike: assignee Elod Illes (elod-illes)
2020-09-23 22:56:41 Lee Yarwood nova/train: status Fix Committed Fix Released
2020-09-23 22:56:44 Lee Yarwood nova/stein: status Fix Committed Fix Released
2020-09-23 22:56:49 Lee Yarwood nova/queens: status Fix Committed Fix Released
2020-09-23 22:56:55 Lee Yarwood nova/rocky: status Fix Committed Fix Released
2020-09-23 22:57:04 Lee Yarwood nova/queens: assignee Elod Illes (elod-illes) Lee Yarwood (lyarwood)
2020-09-23 22:57:46 Lee Yarwood nova/pike: assignee Elod Illes (elod-illes) Lee Yarwood (lyarwood)
2020-09-25 15:26:37 OpenStack Infra nova/pike: assignee Lee Yarwood (lyarwood) melanie witt (melwitt)
2020-09-25 15:37:51 melanie witt nova/pike: assignee melanie witt (melwitt) Lee Yarwood (lyarwood)
2020-09-25 23:35:47 OpenStack Infra nova/pike: status In Progress Fix Committed
2020-11-06 11:58:22 Lee Yarwood nova/pike: status Fix Committed Fix Released
2022-08-02 15:19:07 Nick Tait removed subscriber Nick Tait