clean focal install, crash report qemu-guest-agent

[Harry Coin]

[Impact]

 * On shutdown the qemu-guest-agent crashes, that will
   - trigger the crash reports
   - throw two errors/warnings into the logs
   - can cause hang (not seen in all cases)

 * Since generally the shutdown seems to still work it is "only" medium,
   maybe high in prio. But really should be resolved to avoid that we find
   even more severe consequences (like bad guest state tracking into hard
   kills, I can imagine these)

 * After outlining the issue above the fix was cerated and is rather small
   and easy. It applies to E-G as-is.

[Test Case]

* Create a guest VM with the target release you want to test.
  Ensure you have the channel for gueqt-agent set up and making sure it is
  installed, up and running:

● qemu-guest-agent.service - QEMU Guest Agent
     Loaded: loaded (/lib/systemd/system/qemu-guest-agent.service; static; vendor preset: enabled)
     Active: active (running) since Tue 2020-06-02 05:25:04 UTC; 3min 8s ago
   Main PID: 624 (qemu-ga)
      Tasks: 1 (limit: 533)
     Memory: 900.0K
     CGroup: /system.slice/qemu-guest-agent.service
             └─624 /usr/sbin/qemu-ga

* Pass a shutdown command through virsh:
    $ virsh shutdown focal --mode agent
  An alternative way to trigger the same is:
    $ virsh qemu-agent-command focal '{"execute": "guest-shutdown"}'

The guest almost immediately went down usually, but on reboot you can check the journal of the service and will find the crashes:
  $ journalctl -u qemu-guest-agent
  Jun 02 05:29:54 focal qemu-ga[624]: ERROR:/build/qemu-74sXTC/qemu-4.2/qga/main.c:532:send_response: assertion failed: (rsp && s->channel)
  Jun 02 05:29:54 focal qemu-ga[624]: Bail out! ERROR:/build/qemu-74sXTC/qemu-4.2/qga/main.c:532:send_response: assertion failed: (rsp && s->channel)

[Regression Potential]

 * Formerly qemu was asserting on a bad pointer and exiting non gracefully.
   That is replaced by a clean handling of the case. The different return
   path will slightly change behavior - to the better. I can't see an issue
   that would likely happen - but never the less if we look for regressions
   they will be on the guest shutdown path.

[Other Info]

 * Hangs got reported by affected users, but yet unable to be reproduced on
   other systems.

---

Completely new/fresh focal / mate install. Crash during initial apt install of many packages after first log-in, including qemu-guest-agent. No other important load other than apt running. OS running in a vm as a guest (obviously...) No idea what caused it. Including crash report.
 VM was 2 penryn class cpus with 3G ram.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: qemu-guest-agent 1:4.2-3ubuntu6
ProcVersionSignature: Ubuntu 5.4.0-29.33-generic 5.4.30
Uname: Linux 5.4.0-29-generic x86_64
ApportVersion: 2.20.11-0ubuntu27
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: MATE
Date: Fri May 15 13:20:02 2020
InstallationDate: Installed on 2020-05-13 (1 days ago)
InstallationMedia: Ubuntu-MATE 20.04 LTS "Focal Fossa" - Release amd64 (20200423)
SourcePackage: qemu
UpgradeStatus: No upgrade log present (probably fresh install)

[Harry Coin (hcoin)]

crash report.
I suppose there ought to be another crash report/bug report about how this crash report didn't get auto-uploaded upon detection as well.

[Dan Streetman (ddstreet)]

@hcoin are segfaults attracted to you somehow? ;-) at least this doesn't *seem* to be related to bug 1873607.

[Harry Coin (hcoin)]

Segfaults attracted to me?    That's possible.  Or, I'm the only ubuntu
user that actually reports bugs.   It's lonely out here.

More seriously, I think there's a problem with the apport system more
generally.    What's supposed to happen after a crash is a window pops
up upon log-in, offering to send the report.  That happens.  I click
'yes'.  Then after some grinding and a couple extra files created in
/var/crash ... nothing.   What's supposed to happen is firefox pops up
and offers the usual process with all the attachments.  What actually
happens is nothing.    I go run ubuntu-bug manually, file the stuff,
attach the crash report manually.

The only clue I have to offer is:   At install time, I do NOT install
the restricted third party options, and I go for the 'minimal
install'.   As opposed to all the desktop utility stuff.   I'm guessing
there's something taken out of the minimal install that apport needs to
launch the browser.

But it's sort of confidence sapping, to see accumulated items in
/var/crash, to be asked upon each reboot to file a report, then agree,
then have it... simply just nothing.

On 5/15/20 2:31 PM, Dan Streetman wrote:
> @hcoin are segfaults attracted to you somehow? ;-) at least this
> doesn't *seem* to be related to bug 1873607.
>

[Christian Ehrhardt  (paelzer)]

Hmm, I spawned a clean focal as well, installed and started qmeu-guest-agent but it works as expected:

ubuntu@focal:~$ systemctl status qemu-guest-agent
● qemu-guest-agent.service - QEMU Guest Agent
     Loaded: loaded (/lib/systemd/system/qemu-guest-agent.service; static; vendor preset: enabled)
     Active: active (running) since Mon 2020-05-18 09:15:05 UTC; 2s ago
   Main PID: 36945 (qemu-ga)
      Tasks: 1 (limit: 533)
     Memory: 824.0K
     CGroup: /system.slice/qemu-guest-agent.service
             └─36945 /usr/sbin/qemu-ga
May 18 09:15:05 focal systemd[1]: Started QEMU Guest Agent.

The crash has not used any special arguments, from your dump:
  ProcCmdline: /usr/sbin/qemu-ga

This hits an assertion in the code og the guest agent, from the crash when thrown into gdb:

(gdb) frame 4
#4 0x0000556edcfb3451 in send_response (s=0x556ede07b940, s=0x556ede07b940, rsp=0x0) at ./qga/main.c:532
532 g_assert(rsp && s->channel);
(gdb) p rsp
$1 = (const QDict *) 0x0
(gdb) p s->channel
$2 = (GAChannel *) 0x556ede07be90

This is rsp which is 0x0 is created at process_event:
    rsp = qmp_dispatch(&ga_commands, obj, false);

It is unchecked and passes it to send_response which fails on the assert.

On dispatch it runs:
    ret = do_qmp_dispatch(cmds, request, allow_oob, &err);
    if (err) {
        rsp = qmp_error_response(err);
    } else if (ret) {
        rsp = qdict_new();
        qdict_put_obj(rsp, "return", ret);
    } else {
        /* Can only happen for commands with QCO_NO_SUCCESS_RESP */
        rsp = NULL;
    }

Since we get a NULL back from here we likely have hit the third case and the code forgot to check for it.

The debug info at this stage isn't enough to see why it failed.
(gdb) p *obj
$6 = {base = {type = QTYPE_QDICT, refcnt = 1}}
gdb) p ga_commands
$4 = {tqh_first = 0x556ede077d30, tqh_circ = {tql_next = 0x556ede077d30, tql_prev = 0x556ede0785c8}}

So we have two issues here:
1. when processing a command fails qemu-ga gets into a hard crash.
   Instead if rsp == NULL the function process_event should go into its error path I'd think.
2. processing of the command failed.
   To go further with your case as well as a patch for #1 we'd need to understand
   (to reproduce and submit a patch) what exactly triggers this.

You said focal/mate on install. Is this reproducible? If so might this align to one of the jobs qemu-ga has like resolution changes or any of it? Can you later start and run qemu-qa without issues? If so can you identify which then would trigger a new crash?

[Christian Ehrhardt  (paelzer)]

The list of commands it uses seems fine to me:

(gdb) p *((struct QmpCommand *) 0x556ede077d30)
$50 = {name = 0x556edcfe460c "guest-sync-delimited", fn = 0x556edcfbe2b0 <qmp_marshal_guest_sync_delimited>, options = QCO_NO_OPTIONS, node = {tqe_next = 0x556ede077d70, tqe_circ = {
      tql_next = 0x556ede077d70, tql_prev = 0x556edd002e00 <ga_commands>}}, enabled = true}
(gdb) p *((struct QmpCommand *) 0x556ede077d70)
$51 = {name = 0x556edcfe4601 "guest-sync", fn = 0x556edcfbe400 <qmp_marshal_guest_sync>, options = QCO_NO_OPTIONS, node = {tqe_next = 0x556ede077db0, tqe_circ = {tql_next = 0x556ede077db0,
      tql_prev = 0x556ede077d48}}, enabled = true}
(gdb) p *((struct QmpCommand *) 0x556ede077db0)
$52 = {name = 0x556edcfe4331 "guest-ping", fn = 0x556edcfc1850 <qmp_marshal_guest_ping>, options = QCO_NO_OPTIONS, node = {tqe_next = 0x556ede077df0, tqe_circ = {tql_next = 0x556ede077df0,
      tql_prev = 0x556ede077d88}}, enabled = true}
(gdb) p *((struct QmpCommand *) 0x556ede077df0)
$53 = {name = 0x556edcfe7122 "guest-get-time", fn = 0x556edcfbe550 <qmp_marshal_guest_get_time>, options = QCO_NO_OPTIONS, node = {tqe_next = 0x556ede077e30, tqe_circ = {
      tql_next = 0x556ede077e30, tql_prev = 0x556ede077dc8}}, enabled = true}
(gdb) p *((struct QmpCommand *) 0x556ede077e30)
$54 = {name = 0x556edcfe7131 "guest-set-time", fn = 0x556edcfbe6b0 <qmp_marshal_guest_set_time>, options = QCO_NO_OPTIONS, node = {tqe_next = 0x556ede077e70, tqe_circ = {
      tql_next = 0x556ede077e70, tql_prev = 0x556ede077e08}}, enabled = true}
(gdb) p *((struct QmpCommand *) 0x556ede077e70)
$55 = {name = 0x556edcfe4326 "guest-info", fn = 0x556edcfbe7e0 <qmp_marshal_guest_info>, options = QCO_NO_OPTIONS, node = {tqe_next = 0x556ede077eb0, tqe_circ = {tql_next = 0x556ede077eb0,
      tql_prev = 0x556ede077e48}}, enabled = true}
(gdb) p *((struct QmpCommand *) 0x556ede077eb0)
$56 = {name = 0x556edcfe7140 "guest-shutdown", fn = 0x556edcfbe9f0 <qmp_marshal_guest_shutdown>, options = QCO_NO_SUCCESS_RESP, node = {tqe_next = 0x556ede077ef0, tqe_circ = {
      tql_next = 0x556ede077ef0, tql_prev = 0x556ede077e88}}, enabled = true}
(gdb) p *((struct QmpCommand *) 0x556ede077ef0)
$57 = {name = 0x556edcfe714f "guest-file-open", fn = 0x556edcfbeb20 <qmp_marshal_guest_file_open>, options = QCO_NO_OPTIONS, node = {tqe_next = 0x556ede077f30, tqe_circ = {
      tql_next = 0x556ede077f30, tql_prev = 0x556ede077ec8}}, enabled = true}
(gdb) p *((struct QmpCommand *) 0x556ede077f30)
$58 = {name = 0x556edcfe715f "guest-file-close", fn = 0x556edcfbec90 <qmp_marshal_guest_file_close>, options = QCO_NO_OPTIONS, node = {tqe_next = 0x556ede077f70, tqe_circ = {
      tql_next = 0x556ede077f70, tql_prev = 0x556ede077f08}}, enabled = true}
(gdb) p *((struct QmpCommand *) 0x556ede077f70)
$59 = {name = 0x556edcfe7170 "guest-file-read", fn = 0x556edcfbedc0 <qmp_marshal_guest_file_read>, options = QCO_NO_OPTIONS, node = {tqe_next = 0x556ede077fb0, tqe_circ = {
      tql_next = 0x556ede077fb0, tql_prev = 0x556ede077f48}}, enabled = true}
(gdb) p *((struct QmpCommand *) 0x556ede077fb0)
$60 = {name = 0x556edcfe7180 "guest-file-write", fn = 0x556edcfbefc0 <qmp_marshal_guest_file_write>, options = QCO_NO_...

[Christian Ehrhardt  (paelzer)]

I got this reproduced when retrying again for another case-
Actually not too hart to hit :-/

Taking a Focal system with the version of qemu-guest-agent you reported and then making sure it is installed, up and running:

● qemu-guest-agent.service - QEMU Guest Agent
     Loaded: loaded (/lib/systemd/system/qemu-guest-agent.service; static; vendor preset: enabled)
     Active: active (running) since Tue 2020-06-02 05:25:04 UTC; 3min 8s ago
   Main PID: 624 (qemu-ga)
      Tasks: 1 (limit: 533)
     Memory: 900.0K
     CGroup: /system.slice/qemu-guest-agent.service
             └─624 /usr/sbin/qemu-ga

Pass a shutdown command through virsh:
  $ virsh shutdown focal --mode agent

The guest almost immediately went down, without "functional" issues.
But when checking the logs later I found:
Jun 02 05:29:54 focal qemu-ga[624]: ERROR:/build/qemu-74sXTC/qemu-4.2/qga/main.c:532:send_response: assertion failed: (rsp && s->channel)
Jun 02 05:29:54 focal qemu-ga[624]: Bail out! ERROR:/build/qemu-74sXTC/qemu-4.2/qga/main.c:532:send_response: assertion failed: (rsp && s->channel)

[Christian Ehrhardt  (paelzer)]

Note the other - unlikely but still potentially - related bug is bug 1881668

[Christian Ehrhardt  (paelzer)]

I was trying other commands through qemu-guest-agent and they worked fine.

$ virsh guestinfo focal
user.count : 1
user.0.name : ubuntu
user.0.login-time : 1591075860941
...

$ virsh guestvcpus focal
vcpus : 0
online : 0
offlinable :

Also free form commands were fine and here we can also see what is enabled.

$ virsh qemu-agent-command focal '{"execute": "guest-info"}' | jq .
{
  "return": {
    "version": "4.2.0",
    "supported_commands": [
      {
        "enabled": true,
        "name": "guest-get-osinfo",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-get-timezone",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-get-users",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-get-host-name",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-exec",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-exec-status",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-get-memory-block-info",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-set-memory-blocks",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-get-memory-blocks",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-set-user-password",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-get-fsinfo",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-set-vcpus",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-get-vcpus",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-network-get-interfaces",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-suspend-hybrid",
        "success-response": false
      },
      {
        "enabled": true,
        "name": "guest-suspend-ram",
        "success-response": false
      },
      {
        "enabled": true,
        "name": "guest-suspend-disk",
        "success-response": false
      },
      {
        "enabled": true,
        "name": "guest-fstrim",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-fsfreeze-thaw",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-fsfreeze-freeze-list",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-fsfreeze-freeze",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-fsfreeze-status",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-file-flush",
        "success-response": true
      },
      {
        "enabled": true,
        "name": "guest-file-seek",
        "success-res...

[Christian Ehrhardt  (paelzer)]

If I run shutdown via that there is more evidence hat something is wrong, it seems like the other side of the qemu-guest-agent assert&exit.

Host:
$ virsh qemu-agent-command focal '{"execute": "guest-shutdown"}'

Then I see in the guest:
Jun 02 06:22:17 focal qemu-ga[623]: ERROR:/build/qemu-74sXTC/qemu-4.2/qga/main.c:532:send_response: assertion failed: (rsp && s->channel)
Jun 02 06:22:17 focal qemu-ga[623]: Bail out! ERROR:/build/qemu-74sXTC/qemu-4.2/qga/main.c:532:send_response: assertion failed: (rsp && s->channel)

And then in the host the "consequence":
error: Guest agent is not responding: Guest agent disappeared while executing command

[Christian Ehrhardt  (paelzer)]

The actual assert is from "forever" [3] (v0.15) which is the initial addition of qemu guest agent in 2011. That was later restructured in [1] (v1.1) and [2] (v4.0).

[1]: https://git.qemu.org/?p=qemu.git;a=commit;h=125b310e1d62
[2]: https://git.qemu.org/?p=qemu.git;a=commit;h=781f2b3d1e5e
[3]: https://git.qemu.org/?p=qemu.git;a=commit;h=48ff7a625b36

So assuming the restructuring didn't functionally change it that wasn't changed at all in years.

And even if we assume the restructuring has broken it then >=Eoan (4.0) would be affected as well.

Chances are that the "other side" of this communication (in libvirt [likely] or host qemu [unlikely to be related]) has changed, so I'll need to try this with a few different combinations and that might lead us to the actual root cause.

[Christian Ehrhardt  (paelzer)]

Q = Host qemu
L = Host libvirt
A = Guest qemu-agent

An initial Matrix might look like this:
Host: Q 2.11 L 4.0 (Bionic) - G 2.11 (Bionic)
Host: Q 4.0 L 5.4 (Eoan) - G 2.11 (Bionic)
Host: Q 4.2 L 6.0 (Focal) - G 2.11 (Bionic)
Host: Q 2.11 L 4.0 (Bionic) - G 4.0 (Eoan)
Host: Q 4.0 L 5.4 (Eoan) - G 4.0 (Eoan)
Host: Q 4.2 L 6.0 (Focal) - G 4.0 (Eoan)
Host: Q 2.11 L 4.0 (Bionic) - G 4.2 (Focal)
Host: Q 4.0 L 5.4 (Eoan) - G 4.2 (Focal)
Host: Q 4.2 L 6.0 (Focal) - G 4.2 (Focal)

That shouldn't be too hard with some container magic ...
From there we can then alternate libvirt/qemu independently to narrow it down to just one component.

[Christian Ehrhardt  (paelzer)]

Test results:
1. ensured we had the full matrix in place
2. ensured guest-agent was running fine in guest
3. issued virsh qemu-agent-command $rel '{"execute": "guest-shutdown"}'

1) Host: Q 2.11 L 4.0 (Bionic) - G 2.11 (Bionic)
2) Host: Q 4.0 L 5.4 (Eoan) - G 2.11 (Bionic)
3) Host: Q 4.2 L 6.0 (Focal) - G 2.11 (Bionic)
4) Host: Q 2.11 L 4.0 (Bionic) - G 4.0 (Eoan)
5) Host: Q 4.0 L 5.4 (Eoan) - G 4.0 (Eoan)
6) Host: Q 4.2 L 6.0 (Focal) - G 4.0 (Eoan)
7) Host: Q 2.11 L 4.0 (Bionic) - G 4.2 (Focal)
8) Host: Q 4.0 L 5.4 (Eoan) - G 4.2 (Focal)
9) Host: Q 4.2 L 6.0 (Focal) - G 4.2 (Focal)

Test a - shutdown functionality:
- 1-9: shutting down correctly
- 1-3,7-9: showed "error: Guest agent is not responding: Guest agent disappeared while executing command"
- 4-6: showed "error: Guest agent is not responding: Guest agent not available for now"

So the missing response warning doesn't seem to be new, and OTOH shutdown works fine on all of them.

In regard to the assert that we were looking for:
Test b - the assert
1,4,7: no assert
2-3,5-6,8-9: the assert triggers in the logs

It seems to NOT depend on the host version of qemu/libvirt.
It affects Eoan as well, and given the changes we see it seems [2] was the change that introduced this assert to be hit.

[2]: https://git.qemu.org/?p=qemu.git;a=commit;h=781f2b3d1e5e

[Christian Ehrhardt  (paelzer)]

This also was the time of adding a proper systemd service, so the old
  /usr/sbin/qemu-ga --daemonize -m virtio-serial -p /dev/virtio-ports/org.qemu.guest_agent.0
became just
  /usr/sbin/qemu-ga

But this is no issue:
-m virtio-serial -> is the default anyway
-p /dev/virtio-ports/org.qemu.guest_agent.0 -> is the default anyway
--daemonize - well the service keeps it under control

[Christian Ehrhardt  (paelzer)]

I have thrown a build for Focal with [2] reverted into a PPA [1].
I'm not sure it is a fix, but a good next step for testing.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4081
[2]: https://git.qemu.org/?p=qemu.git;a=commit;h=781f2b3d1e5e

[Christian Ehrhardt  (paelzer)]

Ok, the test build with 781f2b3d1e5e reverted fix the error messages and the crashes.
I'll report this upstream to the Author to consider a follow on fix we then can backport to E&F

[Christian Ehrhardt  (paelzer)]

Reported upstream at https://lists.nongnu.org/archive/html/qemu-devel/2020-06/msg00917.html

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Ehrhardt  (paelzer)]

Marked the other bug as dup since it was confirmed that the PPA helped to resolve the issue (hang on shutdown).

[Christian Ehrhardt  (paelzer)]

Patch at https://lists.nongnu.org/archive/html/qemu-devel/2020-06/msg00962.html

This will soon be accepted and I'll then build fixes for Eoan/Focal/Groovy.

I'll prepare an SRU template already ...

[Christian Ehrhardt  (paelzer)]

FYI: Uploaded to Groovy

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:4.2-3ubuntu9

---------------
qemu (1:4.2-3ubuntu9) groovy; urgency=medium

  * debian/patches/ubuntu/lp-1878973-*: fix assert in qemu-guest-agent that
    crashes it on shutdown (LP: #1878973)
  * d/p/ubuntu/lp-1882774-*: fix issues with VMX subfeatures on systems not
    supporting to set them (LP: #1882774)

 -- Christian Ehrhardt <email address hidden> Tue, 02 Jun 2020 10:42:49 +0200

[Christian Ehrhardt  (paelzer)]

Complete in groovy, preparing for SRUs - SRU-Template ready, MPs are up.

@SRU Team - already ahead as hint please release qemu 1:4.2-3ubuntu6.2/1:4.0+dfsg-0ubuntu9.7 of bug 1805256 which is ready, before accepting this one.

[Christian Ehrhardt  (paelzer)]

Ok, the other SRU is released and the MP reviewed.
Uploaded this one to E/F -unapproved

[Łukasz Zemczak (sil2100)]

Hello Harry, or anyone else affected,

Accepted qemu into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:4.2-3ubuntu6.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Harry, or anyone else affected,

Accepted qemu into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:4.0+dfsg-0ubuntu9.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:4.0+dfsg-0ubuntu9.8) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

ganeti/2.16.0-5ubuntu1 (amd64, ppc64el, arm64)
edk2/0~20190606.20d2e5a1-2ubuntu1.1 (armhf)
systemd/242-7ubuntu3.9 (amd64)
ubuntu-image/1.9+19.10ubuntu1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:4.2-3ubuntu6.3) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

livecd-rootfs/2.664.2 (amd64)
systemd/245.4-4ubuntu3.1 (ppc64el)
edk2/0~20191122.bd85bf54-2ubuntu3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Christian Ehrhardt  (paelzer)]

I've started to look into the autpkgtest fails, so far all look like unreliable tests not triggered by the upload. I have re-triggered accordingly and will investigate in case one stays unresolved.

[Christian Ehrhardt  (paelzer)]

Prior to the fix following the tests steps:

Focal
Jun 23 09:10:03 focal systemd[1]: Started QEMU Guest Agent.
Jun 23 09:10:12 focal qemu-ga[2212]: info: guest-shutdown called, mode: (null)
Jun 23 09:10:12 focal qemu-ga[2212]: **
Jun 23 09:10:12 focal qemu-ga[2212]: ERROR:/build/qemu-FC5BvZ/qemu-4.2/qga/main.c:532:send_response: assertion failed: (rsp && s->channel)
Jun 23 09:10:12 focal qemu-ga[2212]: Bail out! ERROR:/build/qemu-FC5BvZ/qemu-4.2/qga/main.c:532:send_response: assertion failed: (rsp && s->channel)
Jun 23 09:10:12 focal systemd[1]: Stopping QEMU Guest Agent...

Eoan
Jun 23 09:09:56 eoan systemd[1]: Started QEMU Guest Agent.
Jun 23 09:10:17 eoan qemu-ga[2177]: info: guest-shutdown called, mode: (null)
Jun 23 09:10:17 eoan qemu-ga[2177]: **
Jun 23 09:10:17 eoan qemu-ga[2177]: ERROR:/build/qemu-alOsBA/qemu-4.0+dfsg/qga/main.c:532:send_response: assertion failed: (rsp && s->channel)
Jun 23 09:10:17 eoan qemu-ga[2177]: Bail out! ERROR:/build/qemu-alOsBA/qemu-4.0+dfsg/qga/main.c:532:send_response: assertion failed: (rsp && s->channel)
Jun 23 09:10:17 eoan systemd[1]: Stopping QEMU Guest Agent...
Jun 23 09:10:17 eoan systemd[1]: qemu-guest-agent.service: Main process exited, code=dumped, status=6/ABRT

Upgrading to -proposed

Focal
ubuntu@focal:~$ sudo apt install qemu-guest-agent
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  qemu-guest-agent
1 upgraded, 0 newly installed, 0 to remove and 22 not upgraded.
Need to get 194 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/universe amd64 qemu-guest-agent amd64 1:4.2-3ubuntu6.3 [194 kB]
Fetched 194 kB in 0s (1185 kB/s)
(Reading database ... 63108 files and directories currently installed.)
Preparing to unpack .../qemu-guest-agent_1%3a4.2-3ubuntu6.3_amd64.deb ...
Unpacking qemu-guest-agent (1:4.2-3ubuntu6.3) over (1:4.2-3ubuntu6.2) ...
Setting up qemu-guest-agent (1:4.2-3ubuntu6.3) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for systemd (245.4-4ubuntu3.1) ...

Eoan
ubuntu@eoan:~$ sudo apt install qemu-guest-agent
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  qemu-guest-agent
1 upgraded, 0 newly installed, 0 to remove and 25 not upgraded.
Need to get 200 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu eoan-proposed/universe amd64 qemu-guest-agent amd64 1:4.0+dfsg-0ubuntu9.8 [200 kB]
Fetched 200 kB in 0s (1284 kB/s)
(Reading database ... 61964 files and directories currently installed.)
Preparing to unpack .../qemu-guest-agent_1%3a4.0+dfsg-0ubuntu9.8_amd64.deb ...
Unpacking qemu-guest-agent (1:4.0+dfsg-0ubuntu9.8) over (1:4.0+dfsg-0ubuntu9.7) ...
Setting up qemu-guest-agent (1:4.0+dfsg-0ubuntu9.8) ...
Processing triggers for man-db (2.8.7-3) ...
Processing triggers for systemd (242-7ubuntu3.9) ...

Newly testing the shutdown feature:

Focal
-- Reboot --
Jun 23 09:10:42 focal systemd[1]: Started QEMU Guest Agent.
Jun 23 09...

[Christian Ehrhardt  (paelzer)]

Prior to the fix starting a host-model guest on GCE:

ubuntu@nested-vm:~$ virsh start focal
error: Failed to start domain focal
error: internal error: process exited while connecting to monitor: 2020-06-23T09:54:47.062832Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-06-23T09:54:47.062847Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-06-23T09:54:47.062862Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-ept-execonly [bit 0]
2020-06-23T09:54:47.062875Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48CH).vmx-eptad [bit 21]
2020-06-23T09:54:47.062891Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(491H).vmx-eptp-switching [bit 0]
2020-06-23T09:54:47.063762Z qemu-system-x86_64: error: failed to set MSR 0x48b to 0x159ff00000000
qemu-system-x86_64: /build/qemu-FC5BvZ/qemu-4.2/target/i386/kvm.c:2680: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

Updating to focal-proposed worked nicely, retesting the case ...

ubuntu@nested-vm:~$ virsh start focal
Domain focal started

The guest is up and running

ubuntu@nested-vm:~$ virsh list
 Id Name State
-----------------------
 3 focal running

The guest log is now without errors:
2020-06-23 09:57:29.845+0000: starting up libvirt version: 6.0.0, package: 0ubuntu8.1 (Christian Ehrhardt <email address hidden> Wed, 20 May 2020 06:59:57 +0200), qemu version: 4.2.0Debian 1:4.2-3ubuntu6.3, kernel: 5.4.0-1015-gcp, hostname: nested-vm.c.prjparide.internal
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-3-focal \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-3-focal/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-3-focal/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-3-focal/.config \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name guest=focal,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3-focal/master-key.aes \
-machine pc-q35-focal,accel=kvm,usb=off,dump-guest-core=off \
-cpu Broadwell-IBRS,vme=on,ss=on,vmx=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,abm=on,ibpb=on,amd-ssbd=on,rsba=on,skip-l1dfl-vmentry=on \
-m 512 \
-overcommit mem-lock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 2e2a0717-613b-4f9c-91eb-59f93f5ecddb \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=30,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2....

[Christian Ehrhardt  (paelzer)]

This update was for the other bug in this SRU, but this one is good as well - so no status change needed

[Christian Ehrhardt  (paelzer)]

All autppkgtest errors resolved and verification done - ready once it is long enough in proposed without an issue found.

[Łukasz Zemczak (sil2100)]

Ok, I think it's as good as it gets, waiting any longer won't make much difference. Thank you for the verification - releasing!

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:4.0+dfsg-0ubuntu9.8

---------------
qemu (1:4.0+dfsg-0ubuntu9.8) eoan; urgency=medium

  * debian/patches/ubuntu/lp-1878973-*: fix assert in qemu-guest-agent that
    crashes it on shutdown (LP: #1878973)

 -- Christian Ehrhardt <email address hidden> Tue, 02 Jun 2020 10:42:49 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:4.2-3ubuntu6.3

---------------
qemu (1:4.2-3ubuntu6.3) focal; urgency=medium

  * debian/patches/ubuntu/lp-1878973-*: fix assert in qemu-guest-agent that
    crashes it on shutdown (LP: #1878973)
  * d/p/ubuntu/lp-1882774-*: fix issues with VMX subfeatures on systems not
    supporting to set them (LP: #1882774)

 -- Christian Ehrhardt <email address hidden> Tue, 02 Jun 2020 10:42:49 +0200

[Harry Coin (hcoin)]

I agree.  No issues here these many days.

On 7/6/20 9:42 AM, Łukasz Zemczak wrote:
> Ok, I think it's as good as it gets, waiting any longer won't make much
> difference. Thank you for the verification - releasing!
>