implement a machine type based control of host-phys-bits

Bug #1776189 reported by  Christian Ehrhardt  on 2018-06-11
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
qemu (Ubuntu)
Undecided
Unassigned
Bionic
Undecided
Unassigned
seabios (Ubuntu)
Undecided
Unassigned

Bug Description

[Impact]

 * Qemu supports running guests >1TB but currently the virtualization
   stack above has no good way to control that (e.g. libvirt/openstack).

 * Long term we'd want to see bug 1769053 (this is where all started)
   implemented in libvirt and exploited by at least the major upper
   stacks. But until then we want/need to make a way available that lets
   users run huge guests.

 * Following the example of CentOS (but not switching the default for
   better compatibility) we provide a machine type which has the host-
   phys-bit setting of qemu default-on. Since upper layers of the virt
   stack already can control machine types that allows most use cases to
   work until the long term solution can take over down the road.

[Test Case]

 * You can do this on swap, but it will be so slow that you will suffer.
   So better get a machine with a lot of memory - if possible >1TB

 * Spawn a guest with uvtool and ensure that you are running fine

 * 'virsh edit' the guest and increase the memory consumption to >1TB

 * When this guest is started it works at first but later on
   initialization from the guest will touch unreachable adressing bits and
   crash - this will appear as hard freeze to the guest and as KVM
   emulation failure in the host log of guest execution.

 * With the fix you can modify the default type (which should be pc-
   i440fx-bionic) to use the -hpb suffix so it would be:
      pc-i440fx-bionic-hpb
   Use virsh edit to do so.

 * Start the guest again, now the guest will work (be aware that
   initializing so much memory can take some time)

 * This can be tested from PPA build at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3292

[Regression Potential]

 * We were tempted to change the default on the normal type which would
   have had the easiest "ease of use" but then huge regression potential.
   We decided against it so we have a new type that people need to select,
   this means two things:
   1. no change => no regression for users of existing types
   2. users using the new type might find new issues we have never seen
      (tests are good thou) but that is not a regression, but a
      theoretical set of issues with the new function.

[Other Info]

 * The SRUability in my opinion falls under BOTH paragraphs from the SRU
   page

 * For Long Term Support releases we regularly want to enable new
   hardware. Such changes are appropriate provided that we can ensure not
   to affect upgrades on existing hardware. For example, modaliases of
   newly introduced drivers must not overlap with previously shipped
   drivers. This also includes updating hardware description data such as
   udev's keymaps, media-player-info, mobile broadband vendors, or PCI
   vendor/product list updates.
   => This is ensuring that - as machines grow - Bionuc users
      can put all their ram to good use in KVM guests.

 * For Long Term Support releases we sometimes want to introduce new
   features. They must not change the behaviour on existing installations
   (e. g. entirely new packages are usually fine). If existing software
   needs to be modified to make use of the new feature, it must be
   demonstrated that these changes are unintrusive, have a minimal
   regression potential, and have been tested properly. To avoid
   regressions on upgrade, any such feature must then also be added to any
   newer supported Ubuntu release. Once a new feature/package has been
   introduced, subsequent changes to it are subject to the usual
   requirements of SRUs to avoid regressions.
   => This is exactly what we achieved by adding new types instead of
      modifying an existing ones.

---

Hi,
on the discussions and tests around bug 1769053 it was identified to long term it would be nice to expose host-pyhs-bits and phys-bits through libvirt.

That is what bug 1769053 and the related upstream BZ will continue to track.

But there are two shortcomings on this for now:
1. it takes way too long to get that done for people wanting to use this right now
2. even if it would be implemented in libvirt, the stacks above do not yet exploit the theoretical pyhs-bits related attributes.

To get Ubuntu Users with huge systems a rather quick access to this feature including usability in higher parts of the virtualization stack we want to somewhat follow what RedHat/CentOs do at [1]

There they just switch the default to "on" in the default machine type.
This has some potential drawbacks if you want to migrate across different generations of machines, therefore we don't want it to be the default in Ubuntu.

Instead we would prefer to add a type with a -large suffix could carry that.
Note: Bikeshedding on the name is appreciated until the change is done, then it is as it is.

I have tested code that does so and it makes maintaining Ubuntu machine types not much harder while at the same time providing our users mentioned benefits for huge guests.

Eventually this will lower the pain for those running this huge guests, and by that increase the time until someone does the exposure through libvirt and all the upper stack exploitation. But that is acceptable as price to get this fast and more easy at least to Cosmic and Bionic releases without breaking too much and the mentioned instant exploitation by components higher in the virt stack.

[1]: http://mirrors.ibiblio.org/ovirt/pub/ovirt-4.0/src/qemu-kvm-ev/kvm-target-i386-Enable-host-phys-bits-on-RHEL.patch

Related branches

I already did some tests with these changes from ppa [1].
This also includes changes to seabios which seem safe for users with less physical bits and required for those huge guests to work properly.

[1]: https://launchpad.net/~canonical-server/+archive/ubuntu/large-virt

Changed in qemu (Ubuntu):
status: New → Triaged
Changed in seabios (Ubuntu):
status: New → Triaged

I'll run less case-bound tests in the form of a full pre-upload regression test on these changes.
Then if safe IMHO can start pushing this to Cosmic and from there consider Bionic SRU.

Note: due to the fact that Cosmic has no "own" qemu version yet the change there will also just modify/extend the Bionic types. Only later on a qemu merge for Cosmic will add cosmic named types.

So far all tests work, but they are still ongoing.

While that is testing I was submitting a request for discussion upstream why the change is not integrated there.
See: https://mail.coreboot.org/pipermail/seabios/2018-June/012310.html

I hope that the feedback will help to evaluate if we want to carry this Delta or if there are any reasons against it.

FYI on the bug as I mentione dthis being controllable in higher stacks.
Libvirt is via type="" attribute.

Openstack can control this globally or per image
 - global: https://docs.openstack.org/mitaka/config-reference/compute/config-options.html
 - per image: https://docs.openstack.org/image-guide/image-metadata.html
 - implementation: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L4125

As assumed the seabios change really IS implemented differently.
That means I have the todo to restest without those.

RE: bikeshedding on the name; I don't think -large or -big maps well
the the feature, which is more like -cpu host; ie, give the host cpu
attributes to the guest vcpu.
In this case, we're asking to have host physical addressing bits
applied to the guest vcpus.

So maybe -hpb for HostPhysicalBits?

On Mon, Jun 11, 2018 at 8:57 AM,  Christian Ehrhardt 
<email address hidden> wrote:
> As assumed the seabios change really IS implemented differently.
> That means I have the todo to restest without those.
>
> --
> You received this bug notification because you are subscribed to qemu in
> Ubuntu.
> https://bugs.launchpad.net/bugs/1776189
>
> Title:
> implement a machine type based control of host-phys-bits
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1776189/+subscriptions

Yep, now all things fit together.
- My old (swap based) tests that worked without a change
- working on real >1TB machines without a change.

That said the change to seabios is really not needed.
Setting that task to invalid.

Changed in seabios (Ubuntu):
status: Triaged → Invalid

We did a short brainstorm/bikeshed sesssion on the name and decided to:
- use "hpb" for host-phys-bits abbreviation as the former -large could be a lot of other things
- acked on not start type proliferation in general (only in very special cases)
- if possible use a "+" instead of "-" in the type
- Add desc in text line for the type
- add a doc or news entry along the change

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu11

---------------
qemu (1:2.11+dfsg-1ubuntu11) cosmic; urgency=medium

  * d/p/ubuntu/machine-type-hpb.patch: add -hpb machine type
    for host-phys-bits=true (LP: #1776189)
    - add an info about this change in debian/qemu-system-x86.NEWS

 -- Christian Ehrhardt <email address hidden> Tue, 12 Jun 2018 09:01:00 +0200

Changed in qemu (Ubuntu):
status: Triaged → Fix Released
no longer affects: seabios (Ubuntu Bionic)
description: updated
Changed in qemu (Ubuntu Bionic):
status: New → Triaged

- Final test build and test from PPA for target release
- Added SRU Template
- Uploaded to review/acceptance by the SRU Team
- Pushed to package repositories as we consider it proposed

Waiting on SRU Team now.

Robie Basak (racb) wrote :

SRU review for Bionic.

This looks reasonable to me. I'm not so sure about the hardware enablement exception, but as a new feature it seems like something we'd want for an LTS anyway, and complies with all the SRU requirements under that exception. It's an entirely new machine type which isn't expected to impact anything existing.

I'm a little concerned that the feature hasn't had much time to age in Cosmic, so I suggest a longer aging period in bionic-proposed than normal for quality reasons, since normally when we backport a feature it is already well established in the development release or upstream.

Changed in qemu (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic

Hello , or anyone else affected,

Accepted qemu into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.11+dfsg-1ubuntu7.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Upgraded:
qemu-block-extra/bionic-proposed 1:2.11+dfsg-1ubuntu7.4 amd64 [upgradable from: 1:2.11+dfsg-1ubuntu7.3]
qemu-kvm/bionic-proposed 1:2.11+dfsg-1ubuntu7.4 amd64 [upgradable from: 1:2.11+dfsg-1ubuntu7.3]
qemu-system-common/bionic-proposed 1:2.11+dfsg-1ubuntu7.4 amd64 [upgradable from: 1:2.11+dfsg-1ubuntu7.3]
qemu-system-x86/bionic-proposed 1:2.11+dfsg-1ubuntu7.4 amd64 [upgradable from: 1:2.11+dfsg-1ubuntu7.3]
qemu-user-static/bionic-proposed 1:2.11+dfsg-1ubuntu7.4 amd64 [upgradable from: 1:2.11+dfsg-1ubuntu7.3]
qemu-utils/bionic-proposed 1:2.11+dfsg-1ubuntu7.4 amd64 [upgradable from: 1:2.11+dfsg-1ubuntu7.3]

Upgrade worked fine as expected.

New types look good
$ kvm -M ? | grep hpb
pc-i440fx-bionic-hpb Ubuntu 18.04 PC (i440FX + PIIX, +host-phys-bits=true, 1996)
pc-q35-bionic-hpb Ubuntu 18.04 PC (Q35 + ICH9, +host-phys-bits=true, 2009)

Starting the guest with that type and 1.4 TB works (also workes with smaller mem - tested 256G).

In addition we had the same pushed through regression tests (on ppa with same content).

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers