[SRU] migration was active, but no RAM info was set

[Brett Milford]

[Impact]

 * While live-migrating many instances concurrently, libvirt sometimes return `internal error: migration was active, but no RAM info was set:`

 * Effects of this bug are mostly observed in large scale clusters with a lot of live migration activity.

 * Has second order effects for consumers of migration monitor such as libvirt and openstack.

[Test Case]

Synthetic reproducer with GDB in comment #21.

Steps to Reproduce:
1. live evacuate a compute
2. live migration of one or more instances fails with the above error

N.B Due to the nature of this bug it is difficult consistently reproduce.
In an environment where it has been observed it is estimated to occur approximately 1/1000 migrations.

[Where problems could occur]
 * In the event of a regression the migration monitor may report an inconsistent state.

[Original Bug Description]

While live-migrating many instances concurrently, libvirt sometimes return internal error: migration was active, but no RAM info was set:
~~~
2022-03-30 06:08:37.197 7 WARNING nova.virt.libvirt.driver [req-5c3296cf-88ee-4af6-ae6a-ddba99935e23 - - - - -] [instance: af339c99-1182-4489-b15c-21e52f50f724] Error monitoring migration: internal error: migration was active, but no RAM info was set: libvirt.libvirtError: internal error: migration was active, but no RAM info was set
~~~

From upstream bug: https://bugzilla.redhat.com/show_bug.cgi?id=2074205

[Other Information]
Related bug: https://bugs.launchpad.net/nova/+bug/1982284

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1994002-qemu-ussuri.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Thomas Huth (th-huth)]

If you need something from upstream QEMU, please use the new bug tracker here: https://gitlab.com/qemu-project/qemu/-/issues

[Mauricio Faria de Oliveira (mfo)]

Hi Brett,

Thanks for the debdiffs!

I just reviewed them, and there are changes that should be made.

I could do those myself, but that wouldn't be an opportunity to
learn/practice some details for SRUs for you, so I'll add notes.

*However*, if you're too busy and can't do that, do let me know.

cheers,
Mauricio

...

qemu.git

$ git describe --contains 552de79bfdd5e9e53847eb3c6d6e4cd898a4370e
v7.1.0-rc0~136^2

ubuntu archive:

$ rmadison -a source qemu
...
 qemu | 1:2.11+dfsg-1ubuntu7 | bionic | source
 qemu | 1:2.11+dfsg-1ubuntu7.40 | bionic-security | source
 qemu | 1:2.11+dfsg-1ubuntu7.40 | bionic-updates | source
 qemu | 1:4.2-3ubuntu6 | focal | source
 qemu | 1:4.2-3ubuntu6.23 | focal-security | source
 qemu | 1:4.2-3ubuntu6.23 | focal-updates | source
 qemu | 1:6.2+dfsg-2ubuntu6 | jammy | source
 qemu | 1:6.2+dfsg-2ubuntu6.2 | jammy-security | source
 qemu | 1:6.2+dfsg-2ubuntu6.5 | jammy-updates | source
 qemu | 1:7.0+dfsg-7ubuntu2 | kinetic | source
 qemu | 1:7.0+dfsg-7ubuntu2 | lunar | source

0) Development release

The development release (lunar) still doesn't have the patch.
That is required for SRU / stable releases.

We'll need a debdiff for lunar, slightly different than kinetic
(release name and greater version string for the upgrade path).

I just checked w/ Christian and we shouldn't wait on qemu 7.1
merge from Debian (sid), which would include the patch, since
the merge from Debian should happen in January to get qemu 7.2.

1) Oldest LTS in standard support

Would Bionic benefit from this fix on the long run as well,
just before it goes into expanded/out of standard- support?

Apparently, some deployments/clouds still use Bionic on kvm
compute nodes.

If so, the backport targets qmp_query_migrate()/same file,
per commit 65ace0604551 ("migration: add postcopy total blocktime into query-migrate").

2) Debdiffs:

- version strings: the 'lp*' version suffix is fine for
test builds, but for official packages usually (see [1]):
just increment '.1' on stable releases, and '1' on dev.

example:
kinetic (sru): 1:7.0+dfsg-7ubuntu2 -> ubuntu2.1
luanr (devel): 1:7.0+dfsg-7ubuntu2 -> ubuntu3

- changelog: mostly good! (d/p/file.patch; LP: #number?; releases).

The LP bug number 1982284 refers to another/openstack bug,
but the Ubuntu SRUs are coming through this bug, apparently.

Since this is the bug where Ubuntu Archive/Cloud Archive
have packages/series on, to be closed when SRUs land in
-proposed and -updates (and UCA), we should change:
1) the LP bug number in the changelog
2) and patch file names
3) also, it's a good idea to link to other LP bug
in the SRU template '[Other Info]' section.

(you could also just move the SRU template/packages/
series/tracks to the other LP bug, I guess. Up to you.)

- quilt patch: add DEP3 headers [2] (Origin:/Bug-Ubuntu:)

- quilt series: missing 'ubuntu/' dir on k/j (not on f)

- duplications: jammy has duplicated messages, and focal
has that plus duplicated changelog entries? -- for HA? x)

[1] https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging
[2] https://dep-team.pages.debi...

[Mauricio Faria de Oliveira (mfo)]

Brett, per our email conversation, please ignore this:
> - quilt series: missing 'ubuntu/' dir on k/j (not on f)

I missed that focal uses `d/p/ubuntu/` too (it just wasn't present in `d/p/series` context lines in the debdiff, for CVEs).
Sorry for the confusion, and thanks for checking!

[Brett Milford (brettmilford)]

Hi Mauricio,

Thanks for your review.
I've made the changes you've requested.
Looking forward to your feedback.

[Mauricio Faria de Oliveira (mfo)]

Thanks, Brett!

Very minor nitpicks left (changelog entry/release for lunar,
and URL for Origin:), I can handle those.

For Lunar/devel release, I'll send a MR for Christian to
review/upload (my upload rights are for stable releases).

This includes a fix to FTBFS per a package change in the
last 24 hours :) happy to catch it now!

It's currently (re)build-testing on all supported archs.
If all goes well now, I'll send the MR for Lunar, and
once it lands, we'll proceed w/ SRUs.

...

I also played with GDB for a synthetic reproducer.
It seems to be possible, but needs a little more study
on the monitor path. We can sync on that later!

[Mauricio Faria de Oliveira (mfo)]

All archs finished building successfully on ppa:mfo/lp1994002v2.

Just sent the MR for Lunar. If/once it lands, I can do the SRUs.

https://code.launchpad.net/~mfo/ubuntu/+source/qemu/+git/qemu/+merge/434118

[Mauricio Faria de Oliveira (mfo)]

For documentation purposes,

The qemu package in lunar-proposed has its migration blocked to lunar(-release)
because of autopkgtests failures (sbuild), which have been analyzed/understood.

We're waiting on the autopkgtests queue to run sbuild w/ triggers on qemu _and_
sbuild from lunar-proposed, which should address the error w/ sbuild/unshare
(lack of adduser command in the sbuild chroot, as apt no longer deps on that).

Once that runs, we'll check if any other errors happen, and address those.

cheers,
Mauricio

[Mauricio Faria de Oliveira (mfo)]

The sbuild autopkgtest failure on the 'unshare' test
is indeed fixed w/ sbuild in lunar-proposed; however,
now the test 'unshare-qemuwrapper' timed out.

autopkgtest [23:36:43]: @@@@@@@@@@@@@@@@@@@@ summary
build-procenv PASS
unshare-qemuwrapper FAIL timed out
unshare PASS

It timed out on the 'guestfish' command, so I enabled
`export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1` there,
and run autopkgtests against its build in a PPA [1].

Then it finished successfully w/out timing out! x)

autopkgtest [16:17:39]: @@@@@@@@@@@@@@@@@@@@ summary
build-procenv PASS
unshare-qemuwrapper PASS
unshare PASS

Not a very useful result, but it did show that an
step in guestfish took ~25 minutes; 30 mins total:

autopkgtest [15:22:52]: test unshare-qemuwrapper: [-----------------------
...
+ export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
+ guestfish <...>
...
libguestfs: trace: tar_in "/tmp/.../ubuntu-lunar-host.tar" "/"
...
tar -C /sysroot/ -xf - 2> /tmp/tarSfYHJX
...
guestfsd: => tar_in (0x45) took 1489.08 secs
...
autopkgtest [15:52:27]: test unshare-qemuwrapper: -----------------------]
unshare-qemuwrapper PASS

So, well, it might have been due to load in the
autopkgtest infrastructure at the time tests ran,
so just triggered retries on sbuild and sbuild+qemu.

Hopefully they will pass and unblock proposed migration
for both sbuild & qemu.

[1] https://autopkgtest.ubuntu.com/results/autopkgtest-lunar-mfo-build/lunar/amd64/s/sbuild/20221215_161801_a2772@/log.gz

[Mauricio Faria de Oliveira (mfo)]

The sbuild autopkgtests need a fix for lunar-proposed;
reported bug 2000015 w/ analysis and debdiff attached.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:7.0+dfsg-7ubuntu3

---------------
qemu (1:7.0+dfsg-7ubuntu3) lunar; urgency=medium

  [ Brett Milford ]
  * d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
    error 'migration was active, but no RAM info was set' (LP: #1994002)

  [ Mauricio Faria de Oliveira ]
  * d/p/u/ebpf-replace-deprecated-bpf_program__set_socket_filt.patch:
    Fix FTBFS with libbpf 1.0.1-2.

 -- Mauricio Faria de Oliveira <email address hidden> Wed, 30 Nov 2022 12:17:51 -0300

[Mauricio Faria de Oliveira (mfo)]

These are the steps for a synthetic reproducer with GDB and QEMU.

It's sufficient to validate the change on QEMU and move forward with SRUs.
(We can add libvirt for reaching the error message too, but it's a plus.)

I'll check/coordinate on the QEMU uploads next week.

cheers,
Mauricio

...

Original code.

The race condition is, one thread can read 's->state' as SETUP (line 1078/1083/1086),
and another thread changes it to ACTIVE _before_ this thread assigns (the different)
's->state' to 'info->status' (line 1123), which will NOT have RAM info expected for
the ACTIVE status (line 1087/1096).

 1056 static void fill_source_migration_info(MigrationInfo *info)
 1057 {
 1058 MigrationState *s = migrate_get_current();
 ...
 1078 switch (s->state) {
 ...
 1083 case MIGRATION_STATUS_SETUP:
 ...
 1086 break;
 1087 case MIGRATION_STATUS_ACTIVE:
 ...
 1096 populate_ram_info(info, s);
 ...
 1123 info->status = s->state;
 1124 }

We'll break on this function that changes migration states.

 void migrate_set_state(int *state, int old_state, int new_state)

Terminal 1)

QEMU process for incoming migration (ie, destination)

qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

Terminal 2)

QEMU process for outgoing migration (ie, source)
with QEMU monitor on port 3333,
with GDB in non-stop threads mode
(and debuginfod for debug symbols),

gdb \
  -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' \
  -iex 'set debuginfod enabled on' -iex 'set debuginfod urls "https://debuginfod.ubuntu.com"' \
  qemu-system-x86_64

Set breakpoints on the functions above
(and line number of the 'break' statement under 'case MIGRATION_STATUS_SETUP'):

b migrate_set_state
b migration/migration.c:1086

run -nodefaults -nographic -S -monitor tcp:0:3333,server,wait=off
...

Terminal 3)

Connect to QEMU monitor with netcat,
and start the the migration:

nc 127.0.0.1 3333
...
(qemu) migrate -d tcp:127.0.0.1:4444

Terminal 2)

GDB breaks on change from migration status NONE to SETUP.

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566f8628, old_state=0, new_state=1) at ../../migration/migration.c:1746

(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE

(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP

Just continue.

(gdb) c

GDB breaks on change from migration status SETUP to ACTIVE.

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566f8628, old_state=1, new_state=4) at ../../migration/migration.c:1746

(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP

(gdb) p (MigrationStatus) 4
$3 = MIGRATION_STATUS_ACTIVE

Let's explore the race condition.

Terminal 2)

Check the migration information:

(qemu) info migrate

Terminal 3)

GDB breaks on the migration information function.

The status is now observed as SETUP (not yet ACTIVE),
and is not yet propagated to the migration information
to be returned to the monitor.

Thread 1 "qemu-system-x86" hit Breakpoint 2, fill_source_migration_info (info=0x555556d65c70) at ../../migration/migration.c:1086

(gdb) p (MigrationStatus) s.state
$4 = MIGRATION_STATUS_SETUP

(gdb) p info.status
$5 = MIGRA...

[Mauricio Faria de Oliveira (mfo)]

Uploaded to k/j/f/b.

[Steve Langasek (vorlon)]

Hello Brett, or anyone else affected,

Accepted qemu into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:7.0+dfsg-7ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:7.0+dfsg-7ubuntu2.2) for kinetic have finished running.
The following regressions have been reported in tests triggered by the package:

sbuild/0.83.1ubuntu1 (amd64)
ubuntu-image/2.2+22.04ubuntu3 (ppc64el)
dropbear/2022.82-4 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/kinetic/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Mauricio Faria de Oliveira (mfo)]

Autopkgtests regressions cleared for kinetic.

[Mauricio Faria de Oliveira (mfo)]

Verification done on kinetic-proposed.

The migration status during the race condition is
not 'active' (which is expected to have RAM info, but didn't)
but 'setup' (which is not, thus no issues).

(qemu) info migrate

-updates:
...
Migration status: active
total time: 0 ms

-proposed:
...
Migration status: setup
total time: 0 ms

Detailed steps:
==============

$ lxc launch ubuntu:kinetic qemu-k
$ lxc exec qemu-k -- su - ubuntu

Packages from -updates: FAIL
----------------------

$ sudo apt install --yes --no-install-recommends qemu-system-x86 gdb dpkg-dev

$ dpkg -s qemu-system-x86 | grep Version:
Version: 1:7.0+dfsg-7ubuntu2.1

...

Source: get line number for breakpoint.

$ sudo add-apt-repository -ys
$ apt source qemu

$ head -n1 qemu-*/debian/changelog
qemu (1:7.0+dfsg-7ubuntu2.1) kinetic-security; urgency=medium

$ vim qemu-*/migration/migration.c

1073 static void fill_source_migration_info(MigrationInfo *info)
1074 {
...
1100 case MIGRATION_STATUS_SETUP:
...
1103 break;
...

...

Terminal 1)

$ qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

Terminal 2)

gdb \
  -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' \
  -iex 'set debuginfod enabled on' -iex 'set debuginfod urls https://debuginfod.ubuntu.com' \
  qemu-system-x86_64

(gdb) b migrate_set_state
...
Breakpoint 1 at 0x47ed10: migrate_set_state. (2 locations)
(gdb) b migration/migration.c:1103
...
Breakpoint 2 at 0x47dba0: file ../../migration/migration.c, line 1103.

(gdb) run -nodefaults -nographic -S -monitor tcp:0:3333,server,wait=off

Terminal 3)

nc 127.0.0.1 3333

(qemu) migrate -d tcp:127.0.0.1:4444

Terminal 2)

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x555556779618, old_state=0, new_state=1) at ../../migration/migration.c:1763

(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE
(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP
(gdb) c

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x555556779618, old_state=1, new_state=4) at ../../migration/migration.c:1763

(gdb) p (MigrationStatus) 1
$3 = MIGRATION_STATUS_SETUP
(gdb) p (MigrationStatus) 4
$4 = MIGRATION_STATUS_ACTIVE

Terminal 3)

(qemu) info migrate

Terminal 2)

Thread 1 "qemu-system-x86" hit Breakpoint 2, fill_source_migration_info (info=0x555556dc6c60) at ../../migration/migration.c:1103

(gdb) p (MigrationStatus) s.state
$6 = MIGRATION_STATUS_SETUP
(gdb) p info.status
$7 = MIGRATION_STATUS_NONE

(gdb) info threads
  Id Target Id Frame
* 1 Thread 0x7ffff6c32340 (LWP 2368) "qemu-system-x86" fill_source_migration_info (info=0x555556dc6c60) at ../../migration/migration.c:1103
  2 Thread 0x7ffff65ff6c0 (LWP 2369) "qemu-system-x86" (running)
  3 Thread 0x7ffff5d7c6c0 (LWP 2370) "qemu-system-x86" (running)
  5 Thread 0x7ffff49ff6c0 (LWP 2373) "qemu-system-x86" migrate_set_state (state=0x555556779618, old_state=1, new_state=4) at ../../migration/migration.c:1763

(gdb) thread 5
(gdb) continue &

(gdb) info threads
  Id Target Id Frame
  1 Thread 0x7ffff6c32340 (LWP 2368) "qemu-system-x86" fill_source_migr...

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:7.0+dfsg-7ubuntu2.2

---------------
qemu (1:7.0+dfsg-7ubuntu2.2) kinetic; urgency=medium

  [ Brett Milford ]
  * d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
    error 'migration was active, but no RAM info was set' (LP: #1994002)

  [ Mauricio Faria de Oliveira ]
  * d/p/u/lp2009048-vfio_map_dma_einval_amd_iommu_1tb.patch: Add hint
    to VFIO_MAP_DMA error on AMD IOMMU for VMs with ~1TB+ RAM (LP: #2009048)

 -- Mauricio Faria de Oliveira <email address hidden> Thu, 02 Mar 2023 17:29:05 -0300

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Andreas Hasenack (ahasenack)]

Hello Brett, or anyone else affected,

Accepted qemu into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:6.2+dfsg-2ubuntu6.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Brett, or anyone else affected,

Accepted qemu into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:4.2-3ubuntu6.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Brett, or anyone else affected,

Accepted qemu into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.11+dfsg-1ubuntu7.42 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:6.2+dfsg-2ubuntu6.7) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.140ubuntu13.1 (amd64)
ubuntu-image/2.2+22.04ubuntu3 (amd64, arm64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:4.2-3ubuntu6.25) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

livecd-rootfs/2.664.46 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:2.11+dfsg-1ubuntu7.42) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/237-3ubuntu10.57 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Mauricio Faria de Oliveira (mfo)]

autopkgtests regressions cleared; unrelated to these changes.

[Brett Milford (brettmilford)]

Verification done on jammy-proposed.

Followed the instructions as per: https://bugs.launchpad.net/cloud-archive/+bug/1994002/comments/26

With the exception that I had to install the debug symbols package as per https://wiki.ubuntu.com/Debug%20Symbol%20Packages for -proposed.

jammy-updates -- Fail:
ubuntu@qemu-j:~$ nc 127.0.0.1 3333
QEMU 6.2.0 monitor - type 'help' for more information
(qemu) migrate -d tcp:127.0.0.1:4444
migrate -d tcp:127.0.0.1:4444

(qemu)
(qemu) info migrate
info migrate

globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: active
total time: 0 ms
(qemu)
(qemu) quit
quit

jammy-proposed - Pass:
ubuntu@qemu-j2:~$ nc 127.0.0.1 3333
QEMU 6.2.0 monitor - type 'help' for more information
(qemu) migrate -d tcp:127.0.0.1:4444
migrate -d tcp:127.0.0.1:4444
(qemu) info migrate
info migrate
globals:
store-global-state: on
only-migratable: off
send-configuration: on
send-section-footer: on
decompress-error-check: on
clear-bitmap-shift: 18
Migration status: setup
total time: 0 ms
(qemu)

Full gdb session output: https://pastebin.ubuntu.com/p/mkhQzCXKdk/

[Mauricio Faria de Oliveira (mfo)]

Verification done for focal-proposed.

focal-updates: FAIL (status: active)

 (qemu) info migrate
 ...
 Migration status: active
 total time: 0 milliseconds

focal-proposed: PASS (status: setup)

 (qemu) info migrate
 ...
 Migration status: setup
 total time: 0 milliseconds

Details:
=======

$ lsb_release -cs
focal

focal-updates: FAIL
-------------

$ curl http://ddebs.ubuntu.com/dbgsym-release-key.asc | sudo apt-key add -
$ sudo add-apt-repository -y 'deb http://ddebs.ubuntu.com/ubuntu focal-updates main'

$ sudo apt install --yes qemu-system-x86 qemu-system-x86-dbgsym

$ dpkg -s qemu-system-x86 | grep Version:
Version: 1:4.2-3ubuntu6.24

$ dpkg -s qemu-system-x86-dbgsym | grep Version:
Version: 1:4.2-3ubuntu6.24

...

$ sudo add-apt-repository -ys 'deb http://archive.ubuntu.com/ubuntu focal-updates main'

$ apt source qemu
$ head -n1 qemu-*/debian/changelog
qemu (1:4.2-3ubuntu6.24) focal-security; urgency=medium

$ vim qemu-*/migration/migration.c

 915 static void fill_source_migration_info(MigrationInfo *info)
...
 925 case MIGRATION_STATUS_SETUP:
 926 info->has_status = true;
 927 info->has_total_time = false;
 928 break;

...

T1)

$ qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

T2)

gdb \
  -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' \
  qemu-system-x86_64

(gdb) b migrate_set_state
Breakpoint 1 at 0x6d3aa0: migrate_set_state. (2 locations)

(gdb) b migration/migration.c:928
Breakpoint 2 at 0x6d317b: file ./migration/migration.c, line 928.

(gdb) run -nodefaults -nographic -S -monitor tcp:0:3333,server,wait=off

T3)

nc 127.0.0.1 3333

(qemu) migrate -d tcp:127.0.0.1:4444

T2)

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566949d8, old_state=0, new_state=1) at ./migration/migration.c:1463
1463 ./migration/migration.c: No such file or directory.
(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE
(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP
(gdb) c

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566949d8, old_state=1, new_state=4) at ./migration/migration.c:1463
1463 in ./migration/migration.c
(gdb) p (MigrationStatus) 1
$3 = MIGRATION_STATUS_SETUP
(gdb) p (MigrationStatus) 4
$4 = MIGRATION_STATUS_ACTIVE
(gdb)

T3)

(qemu) info migrate

T2)

Thread 1 "qemu-system-x86" hit Breakpoint 2, fill_source_migration_info (info=0x555556850590) at ./migration/migration.c:928
928 in ./migration/migration.c

(gdb) p (MigrationStatus) s.state
$6 = MIGRATION_STATUS_SETUP
(gdb) p info.status
$7 = MIGRATION_STATUS_NONE

(gdb) info threads
  Id Target Id Frame
* 1 Thread 0x7ffff5ee55c0 (LWP 5066) "qemu-system-x86" fill_source_migration_info (info=0x555556850590) at ./migration/migration.c:928
  2 Thread 0x7ffff5ee1700 (LWP 5070) "qemu-system-x86" (running)
  3 Thread 0x7ffff565f700 (LWP 5071) "qemu-system-x86" (running)
  5 Thread 0x7fffedfff700 (LWP 5075) "qemu-system-x86" migrate_set_state (state=0x5555566949d8, old_state=1, new_state=4) at ./migration/migration.c:1463
(gdb) thread 5
[Switching to thread 5 (Thread 0x7fffedfff700 (LWP 5075)...

[Mauricio Faria de Oliveira (mfo)]

Verification done for bionic-proposed.

bionic-updates: FAIL (status: active)

 (qemu) info migrate
 info migrate
 globals: store-global-state=1, only_migratable=0, send-configuration=1, send-section-footer=1
 capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off block: off return-path: off pause-before-switchover: off x-multifd: off
 Migration status: active
 total time: 0 milliseconds

bionic-proposed: PASS (status: setup)

 (qemu) info migrate
 info migrate
 globals: store-global-state=1, only_migratable=0, send-configuration=1, send-section-footer=1
 capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: off x-colo: off release-ram: off block: off return-path: off pause-before-switchover: off x-multifd: off
 Migration status: setup
 total time: 0 milliseconds

Details:
=======

$ lsb_release -cs
bionic

bionic-updates: FAIL
--------------

$ curl http://ddebs.ubuntu.com/dbgsym-release-key.asc | sudo apt-key add -
OK
$ sudo add-apt-repository -y 'deb http://ddebs.ubuntu.com/ubuntu bionic-updates main'

$ sudo apt install --yes qemu-system-x86 qemu-system-x86-dbgsym

$ dpkg -s qemu-system-x86 | grep Version:
Version: 1:2.11+dfsg-1ubuntu7.41

$ dpkg -s qemu-system-x86-dbgsym | grep Version:
Version: 1:2.11+dfsg-1ubuntu7.41

...

$ sudo add-apt-repository -ys 'deb http://archive.ubuntu.com/ubuntu bionic-updates main'
$ apt source qemu
$ head -n1 qemu-*/debian/changelog
qemu (1:2.11+dfsg-1ubuntu7.41) bionic-security; urgency=medium

$ vim qemu-*/migration/migration.c
 594 MigrationInfo *qmp_query_migrate(Error **errp)
 ...
 603 case MIGRATION_STATUS_SETUP:
 604 info->has_status = true;
 605 info->has_total_time = false;
 606 break;
...

T1)

$ qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

T2)

(gdb) b migrate_set_state
Breakpoint 1 at 0x58dd60: file ./migration/migration.c, line 1025.

(gdb) b migration/migration.c:603
Breakpoint 2 at 0x58d670: file ./migration/migration.c, line 603.

# Using 603 vs 606 as the code didn't break on 606 for some reason.

(gdb) run -nodefaults -nographic -S -monitor tcp:0:3333,server,wait=off

T3)

$ nc 127.0.0.1 3333

(qemu) migrate -d tcp:127.0.0.1:4444

T2)

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566ac6f0, old_state=0, new_state=1) at ./migration/migration.c:1025
1025 ./migration/migration.c: No such file or directory.
(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE
(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP
(gdb) c
Continuing.
[New Thread 0x7fffcf9ff700 (LWP 4909)]
[New Thread 0x7fffcf1fe700 (LWP 4910)]
[Thread 0x7fffcf9ff700 (LWP 4909) exited]

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566ac6f0, old_state=1, new_state=4) at ./migration/migration.c:1025
1025 in ./migration/migration.c
(gdb) p (MigrationStatus) 1
$3 = MIGRATION_STATUS_SETUP
(gdb) p (MigrationStatus) 4
$4 = MIGRATION_STATUS_ACTIVE

T3)

(qemu) info migrate

T2)

Thread 1 "qemu-system-x86" hit Breakpoint 2, qmp_query_migrate (errp=errp@entry...

[Andreas Hasenack (ahasenack)]

I verified the test results and am satisfied that they show the executed planned test case, and that the results are correct.

The package built correctly in all architectures and Ubuntu releases it was meant for.

There are no DEP8 regressions at the moment.

There is no SRU freeze ongoing at the moment.

There is no halted phasing on the previous update.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:6.2+dfsg-2ubuntu6.7

---------------
qemu (1:6.2+dfsg-2ubuntu6.7) jammy; urgency=medium

  [ Brett Milford ]
  * d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
    error 'migration was active, but no RAM info was set' (LP: #1994002)

  [ Mauricio Faria de Oliveira ]
  * d/p/u/lp2009048-vfio_map_dma_einval_amd_iommu_1tb.patch: Add hint
    to VFIO_MAP_DMA error on AMD IOMMU for VMs with ~1TB+ RAM (LP: #2009048)
  * d/rules: move "Disable LTO on non-amd64" before buildflags.mk on Jammy.

  [ Michal Maloszewski ]
  * d/rules: Disable LTO on non-amd 64 architectures to prevent QEMU
    coroutines from failing (LP: #1921664)

 -- Mauricio Faria de Oliveira <email address hidden> Mon, 06 Mar 2023 17:00:46 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:4.2-3ubuntu6.25

---------------
qemu (1:4.2-3ubuntu6.25) focal; urgency=medium

  [ Brett Milford ]
  * d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
    error 'migration was active, but no RAM info was set' (LP: #1994002)

  [ Mauricio Faria de Oliveira ]
  * d/p/u/lp2009048-vfio_map_dma_einval_amd_iommu_1tb.patch: Add hint
    to VFIO_MAP_DMA error on AMD IOMMU for VMs with ~1TB+ RAM (LP: #2009048)

 -- Mauricio Faria de Oliveira <email address hidden> Thu, 02 Mar 2023 18:07:21 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu7.42

---------------
qemu (1:2.11+dfsg-1ubuntu7.42) bionic; urgency=medium

  [ Brett Milford ]
  * d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
    error 'migration was active, but no RAM info was set' (LP: #1994002)

  [ Mauricio Faria de Oliveira ]
  * d/p/u/lp2009048-vfio_map_dma_einval_amd_iommu_1tb.patch: Add hint
    to VFIO_MAP_DMA error on AMD IOMMU for VMs with ~1TB+ RAM (LP: #2009048)

 -- Mauricio Faria de Oliveira <email address hidden> Thu, 02 Mar 2023 18:26:12 -0300

[Mauricio Faria de Oliveira (mfo)]

Marking the UCA tasks as Invalid, as apparently not needed; please revert otherwise.

[Corey Bryant (corey.bryant)]

Hello Brett, or anyone else affected,

Accepted qemu into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Mauricio Faria de Oliveira (mfo)]

Verification done on ussuri-proposed.
Steps explained in previous comments.

The migration status with the synthetic reproducer in GDB
is now still 'SETUP' (which is not expected to have RAM statistics),
instead of 'ACTIVE' (which is, and caused the issue):

(qemu) info migrate
...
Migration status: setup

...

$ lsb_release -cs
bionic

$ apt policy qemu-system-x86
qemu-system-x86:
  Installed: 1:4.2-3ubuntu6.27~cloud0
  Candidate: 1:4.2-3ubuntu6.27~cloud0
  Version table:
 *** 1:4.2-3ubuntu6.27~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages
        100 /var/lib/dpkg/status
...

$ file $(which qemu-system-x86_64) | grep -o 'BuildID.*,'
BuildID[sha1]=82a4159294ae653e770be24bbcfbb35703e60005,

(Corey provided the .ddeb packages which is not yet exposed/published in PPAs/archive.)

$ dpkg-deb -c qemu-system-x86-dbgsym_4.2-3ubuntu6.27~cloud0_amd64.ddeb | fgrep .debug
-rw-r--r-- root/root 21271712 2023-10-26 14:08 ./usr/lib/debug/.build-id/48/bd78ceee4a669d37efd9ac8d851947205de4f7.debug
-rw-r--r-- root/root 21321832 2023-10-26 14:08 ./usr/lib/debug/.build-id/82/a4159294ae653e770be24bbcfbb35703e60005.debug

$ sudo apt install ./qemu-system-x86-dbgsym_4.2-3ubuntu6.27~cloud0_amd64.ddeb

$ apt source qemu

$ head -n1 qemu-4.2/debian/changelog
qemu (1:4.2-3ubuntu6.27~cloud0) bionic-ussuri; urgency=medium

 915 static void fill_source_migration_info(MigrationInfo *info)
...
 926 case MIGRATION_STATUS_SETUP:
 927 info->has_status = true;
 928 info->has_total_time = false;
 929 break;

Terminal 1)

$ qemu-system-x86_64 -nodefaults -nographic -S -incoming tcp:0:4444

Terminal 2)

$ gdb \
  -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' \
  qemu-system-x86_64
...
Reading symbols from qemu-system-x86_64...Reading symbols from /usr/lib/debug/.build-id/82/a4159294ae653e770be24bbcfbb35703e60005.debug...done.
done.

(gdb) b migrate_set_state
Breakpoint 1 at 0x6ba8c0: file ./migration/migration.c, line 1464.

(gdb) b migration/migration.c:928
Breakpoint 2 at 0x6b9fb3: file ./migration/migration.c, line 928.

(gdb) run -nodefaults -nographic -S -monitor tcp:0:3333,server,wait=off
...

Terminal 3)

$ nc 127.0.0.1 3333
QEMU 4.2.1 monitor - type 'help' for more information
(qemu) migrate -d tcp:127.0.0.1:4444

Terminal 2)

Thread 1 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566a11d8, old_state=0, new_state=1) at ./migration/migration.c:1464
1464 ./migration/migration.c: No such file or directory.

(gdb) p (MigrationStatus) 0
$1 = MIGRATION_STATUS_NONE

(gdb) p (MigrationStatus) 1
$2 = MIGRATION_STATUS_SETUP

(gdb) c
Continuing.
...

Thread 5 "qemu-system-x86" hit Breakpoint 1, migrate_set_state (state=0x5555566a11d8, old_state=1, new_state=4) at ./migration/migration.c:1464
1464 in ./migration/migration.c

(gdb) p (MigrationStatus) 1
$3 = MIGRATION_STATUS_SETUP

(gdb) p (MigrationStatus) 4
$4 = MIGRATION_STATUS_ACTIVE

(gdb)

Terminal 3)

(qemu) info migrate

Terminal 2)

Thread 1 "qemu-system-x86" hit Breakpoint 2, fill_source_migration_info (info=0x5555572d29b0) at ./migration/migration.c:928
928 in ./migration/mi...

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package qemu - 1:4.2-3ubuntu6.27~cloud0
---------------

 qemu (1:4.2-3ubuntu6.27~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 qemu (1:4.2-3ubuntu6.27) focal-security; urgency=medium
 .
   * SECURITY UPDATE: user-after-free issue
     - debian/patches/CVE-2022-1050.patch: Protect against buggy or
       malicious guest driver
     - CVE-2022-1050
   * SECURITY UPDATE: Out-of-bounds read
     - debian/patches/CVE-2022-4144-*.patch: Have qxl_log_command Return
       early if no log_cmd handler; Document qxl_phys2virt(); Pass requested
       buffer size to qxl_phys2virt(); Avoid buffer overrun in qxl_phys2virt;
       Assert memory slot fits in preallocated MemoryRegion
     - CVE-2022-4144
   * SECURITY UPDATE: reentrancy problem
     - debian/patches/CVE-2023-0330.patch: Fix reentrancy issues in the LSI
       controller
     - CVE-2023-0330
 .
 qemu (1:4.2-3ubuntu6.26) focal; urgency=medium
 .
   * d/p/u/lp-1999885-s390x-tod-kvm-don-t-save-restore-the-TOD-in-PV-guest.patch:
     avoid timer issues in s390x secure execution guests (LP: #1999885)
 .
 qemu (1:4.2-3ubuntu6.25) focal; urgency=medium
 .
   [ Brett Milford ]
   * d/p/u/lp1994002-migration-Read-state-once.patch: Fix for libvirt
     error 'migration was active, but no RAM info was set' (LP: #1994002)
 .
   [ Mauricio Faria de Oliveira ]
   * d/p/u/lp2009048-vfio_map_dma_einval_amd_iommu_1tb.patch: Add hint
     to VFIO_MAP_DMA error on AMD IOMMU for VMs with ~1TB+ RAM (LP: #2009048)