UbuntuKVM: migration using NFS mount fails #190

[bugproxy]

[Impact]

 * Users performing live migration of guests with image files
   shared over NFS between the source and target host systems
   may experience I/O errors in the guests if user id of the
   libvirt-qemu user differs between the host systems, due to
   permission errors when accessing the image files.

 * The 16.04 LTS is an important stream for KVM (at least on
   Power), and guest live migration over NFS is an important
   feature on it.

 * The proposed fix (a minimal backport from what is applied
   on Zesty/Debian, so to be very conservative for the LTS)
   simply tries to use the reserved uid for the libvirt-qemu
   user on new installations (when the user is created) if
   the reserved uid is not taken by another user (no failures
   occur if the libvirt-qemu user already exists or the uid
   is taken.)

[Test Case]

 * Setup 2x systems with Ubuntu 16.04 LTS as KVM hosts (e.g., micro and tiny)
   (check whether the libvirt-qemu uid differs between them;
    e.g. # id libvirt-qemu )

 * Create a guest in the source KVM host system (e.g, microg5)

 * Live migrate the guest to the destination KVM host system (e.g., tiny)

   root@micro:~# virsh migrate --live --domain microg5 qemu+ssh://tiny/system
                 --verbose --undefinesource --persistent --timeout 60
   Migration: [100 %]

 * Check whether the guest is now listed in the destination KVM host system:

    root@tiny:~# virsh list --all
    Id Name State

    12 microg5 running

 * Check whether I/O errors are seen in that guest:

    root@microg5:~# dmesg
    ...
    [ 60.818955] blk_update_request: I/O error, dev vdc, sector 96749232
    [ 60.819113] Aborting journal on device vdc2-8.
    [ 60.820121] blk_update_request: I/O error, dev vdc, sector 9084320
    [ 60.820643] EXT4-fs warning (device vdc2): ext4_end_bio:329: I/O error
                 -5 writing to inode 393279 (offset 0 size 0 ...

* Simplified test of the wanted effect:
  - install libvirt on a system that didn't have it before and check the
    id of libvirt-qemu
      $ id libvirt-qemu

[Regression Potential]

 * On installations in which the libvirt-qemu user does not exist
   (e.g., first time installation of libvirt-bin) its assigned uid
   might be different from what the user previously expected, since
   now it's assigned a number from the reserved range.

 * Overall, the fix is very conservative (the uid assignment only
   happens in case: 1) the libvirt-qemu user is being created, and
   2) if the desired uid is not taken by another user, e.g. LDAP).

[Other Info]

 * None at this time.

<...>

Please see comment #8 for the problem description, and summary of originally bridged comments in the description in later comments.

Sorry about the inconvenience.

<...>

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[Mauricio Faria de Oliveira (mfo)]

Sigh.
The bug bridge ate the comment delimiters.

I'll paste the comments in order here in LP, and chop a bit.

Hopefully that will allow humans to understand this.

[Mauricio Faria de Oliveira (mfo)]

Problem:

We setup 2 Ubuntu KVM host with the same mount point and try to migration the guest between 2 HOST. The migration is success, the guest appear on other Host after the migration but it shows some I/O error on the guest.
<...>

On the first host, run this

root@micro:~# virsh migrate --live --domain microg5 qemu+ssh://10.33.10.115/system --verbose --undefinesource --persistent --timeout 60
Migration: [100 %]

The guest appear on other HOST:

root@tiny:~# virsh list --all
Id Name State

2 tinyg1 running
3 tinyg2 running
5 tinyg4 running
6 tinyg5 running
7 tinyg6 running
9 tinyg3 running
12 microg5 running <<< this guest is from HOST "Micro"

Checking status of the guest, I can see this error....

root@microg5:~# dmesg |tail -20
[ 60.818955] blk_update_request: I/O error, dev vdc, sector 96749232
[ 60.819113] Aborting journal on device vdc2-8.
[ 60.820121] blk_update_request: I/O error, dev vdc, sector 9084320
[ 60.820643] EXT4-fs warning (device vdc2): ext4_end_bio:329: I/O error -5 writing to inode 393279 (offset 0 size 0 starting block 1135541)
<...>

[Mauricio Faria de Oliveira (mfo)]

Comment:

The problem is due to a configuration problem in the NFSv4 server and/or clients.
Probably related to the NFSv4 ID <-> Name Mapping (idmap).

Thus, this is not an I/O-related problem.

The solution/requirement is to make sure that the libvirt-qemu user has the same UID/GID on all systems that it might be migrated too.

[Mauricio Faria de Oliveira (mfo)]

Comment:

The reservation of an UID/GID in Debian/Ubuntu follows an allocation process governed by Debian.
I have submitted an allocation request, and will prepare the patches for libvirt-qemu in Debian and Ubuntu.

[Mauricio Faria de Oliveira (mfo)]

Comment:

Hi Chanh,

Can you please test the libvirt-bin & libvirt0 packages?
http://ausgsa.ibm.com/~mauricfo/public/bugs/bz145069/v2/

Please confirm if they resolve the issue.

Thanks!

Details
---

The test packages assume that the UID & GID 64055 will be allocated by Debian, and user this value for libvirt-qemu user/group.

[Mauricio Faria de Oliveira (mfo)]

Comment:

Hi Mauricio,

Yes, after applied this patch, the migration is working fine without any IO error.

I am able to migrate between 2 Host (Tiny & Micro) using the NFS mount method without any IO issue.

root@micro:~# id libvirt-qemu
uid=64055(libvirt-qemu) gid=117(kvm) groups=117(kvm),64055(libvirt-qemu)

root@tiny:~# id libvirt-qemu
uid=64055(libvirt-qemu) gid=116(kvm) groups=116(kvm),64055(libvirt-qemu)

[Mauricio Faria de Oliveira (mfo)]

Comment:

Hi Canonical,
@taco-screen-team

The attached patches are for Zesty, Xenial, and Debian sid (which I plan to submit if the UID/GID allocation request is granted).

afaik @cjwatson is the maintainer of base-passwd on Debian, and could review/grant/deny the allocation request.
Per Ubuntu Policy, we'd need it ack'ed on Debian first.

Then the libvirt patches.. all packages were verified.
For Z, X, and sid, the test-case follows this pattern, and has correct behavior/results.

Thanks!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-10-31 19:06 EDT-------
*** Bug 147941 has been marked as a duplicate of this bug. ***

[Mauricio Faria de Oliveira (mfo)]

The previous comment mentions a test-case and pattern, which were missing.
Here they are:

Test-case
===

Package is not installed -- no libvirt-qemu user/group:
---

# getent passwd libvirt-qemu
#

# getent group libvirt-qemu
#

Package is installed -- libvirt-qemu user/group is created w/ allocated uid/gid:
---

# dpkg -i libvirt{-bin,0}_1.3.1-1ubuntu10.5uidgid1_*.deb

# getent passwd libvirt-qemu
libvirt-qemu:x:64055:117:Libvirt Qemu,,,:/var/lib/libvirt:/bin/false

# getent group libvirt-qemu
libvirt-qemu:x:64055:libvirt-qemu

Package is uninstalled -- libvirt-qemu user/group is removed:
---

# apt-get --yes purge libvirt-bin

# getent passwd libvirt-qemu
#

# getent group libvirt-qemu
#

Package is installed with uid/gid taken -- libvirt-qemu user/group is created with other uid/gid:
---

# useradd --uid 64055 testuser
# groupadd --gid 64055 testgroup

# dpkg -i libvirt{-bin,0}_1.3.1-1ubuntu10.5uidgid1_*.deb
# echo $?
0

# getent passwd libvirt-qemu
libvirt-qemu:x:113:117:Libvirt Qemu,,,:/var/lib/libvirt:/bin/false

# getent group libvirt-qemu
libvirt-qemu:x:118:libvirt-qemu

[Christian Ehrhardt  (paelzer)]

Just skimming over the issue and the proposed solution I'm ok considering for libvirt provided that the change is ok and gone into base-passwd. Adding that as a task to make that clear.

[Christian Ehrhardt  (paelzer)]

@Mauricio - thanks for your great pre-work on this!
You mentioned you push to Debian - do you also have a debian bug already that we should/could link?

[Mauricio Faria de Oliveira (mfo)]

Hi @paelzer,

Thanks!

I had in mind to submit the patch for Debian libvirt if/once the allocation request is granted on Debian. Of course, I can submit in advance and mention it's awaiting for base-passwd.

Could you help w/ the base-passwd process? I believe it's @cjwatson who handles this on Debian and Ubuntu, right?

For reference, this is the e-mail sent to Debian base-passwd:

-------- Forwarded Message --------
Subject: request uid and gid allocation for libvirt-qemu
Date: Tue, 25 Oct 2016 09:27:00 -0200
From: Mauricio Faria de Oliveira <email address hidden>
To: <email address hidden>
CC: <email address hidden>

Hi,

I'd like to request an uid and gid for the libvirt-qemu user/group.

This is in support of guest migration with disk images over NFS. That
requires the same uid/gid on source and target host systems, otherwise
IO requests for disk images (owned by libvirt-qemu) fail with -EACCES,
which causes IO errors in a guest undergoing migration.

The patches for libvirt will be submitted if the allocation is granted.

According to 'base-passwd/README':
> You *may not* use any uids or gids in the 60000-64999 range without
> *first* requesting an allocation from <email address hidden>
> and waiting for confirmation that the allocation has been granted.

Thanks,

--
Mauricio Faria de Oliveira
IBM Linux Technology Center

[Christian Ehrhardt  (paelzer)]

On Wed, Nov 9, 2016 at 6:48 PM, Mauricio Faria de Oliveira <
<email address hidden>> wrote:

> I had in mind to submit the patch for Debian libvirt if/once the
> allocation request is granted on Debian. Of course, I can submit in
> advance and mention it's awaiting for base-passwd.
>

I meant a Debian bug against base-passwd to begin with not a libvirt bug.
I think the Mail you have written is a good start, yet IMHO a related bug
can't hurt t track and make them more aware.

> Could you help w/ the base-passwd process? I believe it's @cjwatson who
> handles this on Debian and Ubuntu, right?
>

Yes it is primarily @cjwatson in both cases, but he is great and
usually pretty responsive - so no "help" needed on that.
If you happen to think this takes too long I'd recommend catching him under
the same IRC nick e.g. on #ubuntu-devel and just ask.

[Mauricio Faria de Oliveira (mfo)]

Hi @paelzer,

> I meant a Debian bug against base-passwd to begin with not a libvirt bug.
> I think the Mail you have written is a good start, yet IMHO a related bug
> can't hurt t track and make them more aware.

Ah, ok. Yes, you're right. That is BTS #843881 [1].

> Yes it is primarily @cjwatson in both cases, but he is great and
> usually pretty responsive - so no "help" needed on that.

Absolutely. I know Colin's work and interacted w/ him a few times; that "help" is not what I meant. :)

That was just in case you were also / could be involved in some part of the process before libvirt; I guess I should have asked 'Can you' instead of 'Could you'; but either way, I understand that's still vague.

So, alright; i'll keep watching the Debian bug, and ping him if that takes a bit longer.

Thank you.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843881

[Colin Watson (cjwatson)]

I've made this allocation; see the Debian bug for details. Please note that it is *not* necessary to take any special steps to upload base-passwd to Ubuntu first, and you should definitely not waste time SRUing base-passwd to update the registry; all that is necessary is that the registry held in git be up-to-date.

[Launchpad Janitor (janitor)]

This bug was fixed in the package base-passwd - 3.5.41

---------------
base-passwd (3.5.41) unstable; urgency=medium

  * Allocate uid/gid 64055 for libvirt guest migration support, by request
    of Mauricio Faria de Oliveira (closes: #843881, LP: #1637601).

 -- Colin Watson <email address hidden> Fri, 11 Nov 2016 21:45:48 +0000

[Christian Ehrhardt  (paelzer)]

Thanks Colin for making the allocation and outlining the details of base-passswd handling.

[Christian Ehrhardt  (paelzer)]

@Mauricio Faria de Oliveira - with the passwd change in I looked deeper into the proposed libvirt debdiff - Sorry to realize that only now, but please help me to clarify/avoid these issues?

While overall I think it is a great fix to define a known uid/gid I'm not so happy with the fact that it changes the group used for libvirt/qemu on build. I think it needs some more fine tuning to be more usable and less interuptive. Therefore I wanted to discuss the following issues:

#1 On a given Xenial install as of today the files are owned by libvirt-qemu:kvm (e.g. pools and images it creates). Those files will still be owned by group "kvm", but after the upgrade libvirt/qemu will fail to access properly right?

#2 I see on the xenial diff you remove the groups in postrm. That will leave the files unowned, just with ther numeric gid. Is there any reason to do this delgroup/deluser? And if so why not in Zesty?

#3 despite the other changes you left the --ingroup kvm - I think it is right to leave it, but if you go the route of changing it to libvirt-qemu this would need to change as well.

Once sorted out Colin might reassign (or a new one :-/) the gid to "kvm" and you could implement it as libvirt-qemu:kvm staying compatible with all setups out there already.

My suggestion would be:
- stick to libvirt-qemu:kvm
- do not drop the user/group on postrm
Do you think that would work for you then?

[Mauricio Faria de Oliveira (mfo)]

Hi Christian Ehrhardt,

> Sorry to realize that only now, but please help me to clarify/avoid these issues?

no problem; sure.

> #1 On a given Xenial install as of today the files are owned by libvirt-qemu:kvm (e.g. pools and images it creates).
> Those files will still be owned by group "kvm", but after the upgrade libvirt/qemu will fail to access properly right?

You're correct. The user permissions can still allow it to work, but it's not really correct.

For Xenial, I'll upload a patch v2 without the group change. That's more conservative, and correct.
For Zesty and later, I believe it's better to keep the change (aligns w/ Debian).

> #2 I see on the xenial diff you remove the groups in postrm. That will leave the files unowned, just with ther numeric gid.
> Is there any reason to do this delgroup/deluser? And if so why not in Zesty?

Yes, but that's only in the 'purge' case, not in 'remove'.
I think that removing the user/group in 'purge' would be correct, but I admit not to know the right rule for this case.
Per your question, and not to change a established behavior in the LTS, it seems better not to make this change. (removed in v2)

The reason to do it is to align with Debian, but is not strictly required.

In Zesty, that is already done, thus no change required.

> #3 despite the other changes you left the --ingroup kvm - I think it is right to leave it,
> but if you go the route of changing it to libvirt-qemu this would need to change as well.

I don't know if the kvm group is important for something else in the virtualization stack, but that is still aligned w/ Debian, so I opted to keep it.
Keeping it in v2 per Debian, you think it's right, and not doing the libvirt-qemu group change.

> Once sorted out Colin might reassign (or a new one :-/) the gid to "kvm" and you could
> implement it as libvirt-qemu:kvm staying compatible with all setups out there already.

Humbly, I believe I have to defer this discussion to the maintainers in Debian/Ubuntu, people who are more experienced in libvirt/qemu, as the changes (libvirt-qemu user/group and kvm group) were introduced in Debian at some point, for some reason, and those might play a role in other components of the libvirt/qemu/kvm stack. Sorry about the uncertainty, but I'm not really a virtualization guy nowadays. :)

> My suggestion would be:
> - stick to libvirt-qemu:kvm
> - do not drop the user/group on postrm
> Do you think that would work for you then?

Agreed. I'll submit those on patch v2.

Thanks for the careful review and suggestions!

[Mauricio Faria de Oliveira (mfo)]

Hi Christian Ehrhardt,

Attaching PATCH v2 for Xenial.
It only checks/sets the UID for libvirt-qemu user; no other actions/changes performed.

[Mauricio Faria de Oliveira (mfo)]

For reference,
Debian bug for libvirt is #844339

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "xenial_libvirt_uidgid_v2.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[bugproxy (bugproxy)]

------- Comment on attachment From <email address hidden> 2016-11-14 12:42 EDT-------

Machine Details:
=============
Note:-

System is on a private network. Access the private network via SSH to "banner.isst.aus.stglabs.ibm.com" using your GSA ID and password.
(Banner itself is behind a BSO, so must authenticate through that first.)

Login details :
ssh banner.isst.aus.stglabs.ibm.com [debug/don2rry]

Source Host login:-
chikvm.isst.aus.stglabs.ibm.com [10.33.7.207]
[root/don2rry]

Destination Host Login:-
guskvm.isst.aus.stglabs.ibm.com has address 10.33.32.222

start Guest Login :-
virsh console chig6 --force

IPMI Login :-
-------------
From banner machine run the following :
ssh banner.isst.aus.stglabs.ibm.com [debug/don2rry ]

ipmitool -I lanplus -H fsp-chi -P don2rry sol deactivate
ipmitool -I lanplus -H fsp-chi -P don2rry sol activate

-----------------------------------------------------------------------------
                             TESTING INFORMATION
-----------------------------------------------------------------------------

SYSTEM INFORMATION
------------------
  HOST NAME or NETWORK ADDRESS: chikvm.isst.aus.stglabs.ibm.com
 [10.33.7.207]
guskvm..isst.aus.stglabs.ibm.com 10.33.32.222

  Sapphire FIRMWARE LEVEL: - FW860.00 (SV860_056)

-----------------------------------------------------------------------------
                            DEBUGGING INFORMATION
-----------------------------------------------------------------------------

DEBUG / LOGIN INFORMATION
-------------------------
  ERROR LOG: on local machine
  FSP LOGIN: see name above [dev/FipSdev]
RECENT SYSTEM CHANGES : none
-----------------------------------------
  none

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[Christian Ehrhardt  (paelzer)]

There was a lot of extra mirroring on this - another bugproxy issue?

[Christian Ehrhardt  (paelzer)]

Also while I know the maintainer scripts already have the same issue I today accidentally came by this https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=621833#174

I just wanted to mention if you think it would be worth to consider to modify your adduser/addgroup calls with that insight.

And as FYI I'm currently heads down on multiple issues, so sometimes replies can take a bit - I beg a pardon in advance.

[Mauricio Faria de Oliveira (mfo)]

> There was a lot of extra mirroring on this - another bugproxy issue?

Probably. Apologies for you getting that. Please disregard it. :/

> Also while I know the maintainer scripts already have the same issue
> I today accidentally came by this https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=621833#174

Very good accident, actually!

> I just wanted to mention if you think it would be worth to consider to
> modify your adduser/addgroup calls with that insight.

Yes, I'll; thanks for mentioning it.

Also, the Debian maintainer asked for changes (do not proceed silently if the allocated uid cannot be used, i.e., user already exists).
So I'll implement that and submit a v3 w/ both changes.

> And as FYI I'm currently heads down on multiple issues,
> so sometimes replies can take a bit - I beg a pardon in advance.

No problem at all. I know the drill. ;- )

[Mauricio Faria de Oliveira (mfo)]

Submitted patch v2 for Debian, addressing changes requested by Guido.
(but not yet the getent/adduser concern of debian bug 621833, which I decided to first ask him about, so not to mix the changes upfront)

[Christian Ehrhardt  (paelzer)]

FYI - @Mauricio - I'm tracking the Debian bug on my own and since a libvirt merge is about to happen some-when next time anyway. If timing works I intend to just do the merge after your change has been released in Debian.

[Mauricio Faria de Oliveira (mfo)]

Hi @Christian.
Cool. good news.

BTW, I have to post some status/progress updates in this bug for internal tracking by other people; not meant at you. :)

[Christian Ehrhardt  (paelzer)]

On Mon, Nov 21, 2016 at 12:31 PM, Mauricio Faria de Oliveira <
<email address hidden>> wrote:

> BTW, I have to post some status/progress updates in this bug for
> internal tracking by other people; not meant at you. :)
>

I still remember the internal process and its mirroring :-)
Thanks for the heads up.

[Mauricio Faria de Oliveira (mfo)]

> I still remember the internal process and its mirroring :-)

I got you now.. ;-) [1]

BTW, just checked in BTS how Guido would like to proceed w/ this one since the last patch (v3) submission ~1.5 weeks ago. Waiting.

[1] https://insights.ubuntu.com/author/paelzer/

[bugproxy (bugproxy)]

------- Comment on attachment From <email address hidden> 2016-11-14 12:42 EDT-------

Machine Details:
=============
Note:-

System is on a private network. Access the private network via SSH to "banner.isst.aus.stglabs.ibm.com" using your GSA ID and password.
(Banner itself is behind a BSO, so must authenticate through that first.)

Login details :
ssh banner.isst.aus.stglabs.ibm.com [debug/don2rry]

Source Host login:-
chikvm.isst.aus.stglabs.ibm.com [10.33.7.207]
[root/don2rry]

Destination Host Login:-
guskvm.isst.aus.stglabs.ibm.com has address 10.33.32.222

start Guest Login :-
virsh console chig6 --force

IPMI Login :-
-------------
From banner machine run the following :
ssh banner.isst.aus.stglabs.ibm.com [debug/don2rry ]

ipmitool -I lanplus -H fsp-chi -P don2rry sol deactivate
ipmitool -I lanplus -H fsp-chi -P don2rry sol activate

-----------------------------------------------------------------------------
                             TESTING INFORMATION
-----------------------------------------------------------------------------

SYSTEM INFORMATION
------------------
  HOST NAME or NETWORK ADDRESS: chikvm.isst.aus.stglabs.ibm.com
 [10.33.7.207]
guskvm..isst.aus.stglabs.ibm.com 10.33.32.222

  Sapphire FIRMWARE LEVEL: - FW860.00 (SV860_056)

-----------------------------------------------------------------------------
                            DEBUGGING INFORMATION
-----------------------------------------------------------------------------

DEBUG / LOGIN INFORMATION
-------------------------
  ERROR LOG: on local machine
  FSP LOGIN: see name above [dev/FipSdev]
RECENT SYSTEM CHANGES : none
-----------------------------------------
  none

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 2.5.0-3ubuntu1

---------------
libvirt (2.5.0-3ubuntu1) zesty; urgency=medium

  * Merged with Debian unstable
    - this picks up a fix for migrations using NFS mounts (LP: #1637601).
  * Remaining changes:
    - Disable sheepdog (universe dependency)
    - Disable libssh2 support (universe dependency)
    - Disable firewalld support (universe dependency)
    - Disable selinux
    - Enable esx support
      - Add build-dep to libcurl4-gnutls-dev (required for esx)
    - Set qemu-group to kvm (for compat with older ubuntu)
    - Added changes to use the upstream apparmor profiles with added
      delta (configurable via apparmor profiles version).
      * d/p/u/000[1-6]-apparmor-*
    - Regularly clear AppArmor profiles for vms that no longer exist
    - Fix name resolution calls from virt-aa-helper profile (LP 1546674).
    - Add missing apparmor rule for debug-threads feature (LP 1615550).
    - Add new block device types to virt-aa-helpers profile (LP 1641618)
    - Additional apport package-hook
    - d/rules: debhelper start virtlogd.socket
    - Add sysvinit script for virtlockd
    - Additional debian/bug-presubj
    - Modifications to adapt for our delayed switch away from libvirt-bin (can
      be dropped after 18.04).
      - d/p/ubuntu/libvirtd-service-add-bin-alias.patch: alias to old
        libvirt-bin name.
      - d/p/ubuntu/libvirtd-init-add-bin-alias.patch: provides for the old
        libvirt-bin name.
    - Wait on socket in sysvinit script
    - Backwards compatible handling of groups (can be dropped after 18.04).
    - config details and autostart of default bridged network. Creating that is
      now the default in general, yet our solution provides the following on
      top as of today:
      - nat only on some ports <port start='1024' end='65535'/>
      - autostart the default network by default
      - do not autostart if 192.168.122.0 is already taken (e.g. in containers)
    - d/p/ubuntu/Allow-libvirt-group-to-access-the-socket.patch: This is
      the group based access to libvirt functions as it was used in Ubuntu
      for quite long.
      - d/p/ubuntu/daemon-augeas-fix-expected.patch fix some related tests
        due to the group access change.
    - d/p/ubuntu/dnsmasq-as-priv-user: configuration to run as extra user
      - d/p/ubuntu/disable-network-test.patch: disable test failing due to
        dnsmasq changes.
    - ubuntu/parallel-shutdown.patch: set parallel shutdown by default.
    - d/p/ubuntu/enable-kvm-spice.patch: compat with older Ubuntu qemu/kvm
      which provided a separate kvm-spice.
    - d/p/ubuntu/storage-default-permission-mode-to-0711: safer default perms
      for storage dirs like /var/lib/libvirt/images.
    - d/p/ubuntu/storage-disable-gluster-test: gluster not enabled, skip test
    - d/p/ubuntu/9031-enable-lxc-apparmor: enable apparmor confinement of
      containers by default.
    - d/p/ubuntu/9032-lxc-allow-no-security-driver: allow empty sec driver for
      libvirt-lxc.
    - d/p/ubuntu/ubuntu_machine_type.patch: accept ubuntu types as pci440fx
    - d/p/ubuntu/ubuntu-libxl-qemu-path.patch: Set common qemu path to match
      Deb...

[Christian Ehrhardt  (paelzer)]

Hi Mauricio,
this being in zesty due to the merge of latest Debian as planned by Debian picking up your changes as discussed.

That said, it would be very kind if you did a check if it works for you as intended.

It does for me, but an ack would be nice
$ id libvirt-qemu
uid=64055(libvirt-qemu) gid=116(kvm) groups=116(kvm),64055(libvirt-qemu)

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-01-12 18:07 EDT-------
Hi Christian,

> That said, it would be very kind if you did a check if it works for you as
> intended.
>
> It does for me, but an ack would be nice

Sure. Sorry for the delay. Catching up from vacation. Should be able to test this soon! Thanks.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-01-16 14:45 EDT-------
Christian,

> That said, it would be very kind if you did a check if it works for you as
> intended.
>
> It does for me, but an ack would be nice
> $ id libvirt-qemu
> uid=64055(libvirt-qemu) gid=116(kvm) groups=116(kvm),64055(libvirt-qemu)

Yes, it does work OK on Zesty for me too.

How would you prefer to handle this for Xenial -- myself or yourself to provide a backport of changes in Debian/Zesty?

Thanks!

Details
----------

Distro and Package version
---

# lsb_release -d
Description: Ubuntu Zesty Zapus (development branch)

# apt-cache madison libvirt-daemon-system
libvirt-daemon-system | 2.5.0-3ubuntu1 | http://us.ports.ubuntu.com/ubuntu-ports zesty/main ppc64el Packages

Install without existing user/group (set UID/GID)
---

# id libvirt-qemu
id: ?libvirt-qemu?: no such user

# apt-get install --yes --no-install-recommends libvirt-daemon-system

# id libvirt-qemu
uid=64055(libvirt-qemu) gid=117(kvm) groups=117(kvm),64055(libvirt-qemu)

Install with existing user/group (keep UID/GID)
---

# apt-get purge --yes libvirt-daemon-system

# id libvirt-qemu
id: ?libvirt-qemu?: no such user

# useradd --system --no-create-home --uid 1704 libvirt-qemu

# id libvirt-qemu
uid=1704(libvirt-qemu) gid=999(libvirt-qemu) groups=999(libvirt-qemu)

# apt-get install --yes --no-install-recommends libvirt-daemon-system

# id libvirt-qemu
uid=1704(libvirt-qemu) gid=999(libvirt-qemu) groups=999(libvirt-qemu)

[Christian Ehrhardt  (paelzer)]

On Mon, Jan 16, 2017 at 9:58 PM, bugproxy <email address hidden> wrote:

> Yes, it does work OK on Zesty for me too.
>

Great to hear that.

> How would you prefer to handle this for Xenial -- myself or yourself to
> provide a backport of changes in Debian/Zesty?
>

You did almost all the work so far, no need to take things away from you.
The steps to go for the SRU would not need to go via Debian again since
Xenial is released already.
To to provide this for xenial follow
https://wiki.ubuntu.com/StableReleaseUpdates.

I can review and upload the libvirt part, but we again need help for
baser-passwd.
The SRU Teamplate will help the SRU Team to make sure we considered
regressions and that severity is worth an update into a released Ubuntu
version.

So in some Detail:
1. you provide a debdiff for Xenial base-passwd
2. you provide a debdiff for Xenial libvirt
3. you fill the SRU Template and add it to the bug description here.
4. Together we will find a sponsor for base-passwd
5. SRU Team will evaluate and accept base-passwd
6. One of us can Test if base-passwd in proposed is as it should
7. I can upload libvirt for you
8. SRU Team will evaluate and accept libvirt
9. We check proposed for the full solution
Done, finally :-)

Since the code in libvirt is written defensively (not fatal error if
base-passwd update isn't there yet) both can actually progress at the same
time safely.
So 4-6 and 7-9 can run concurrently.

For 1&2 please we should start with you confirming if your attached:
https://launchpadlibrarian.net/293402521/xenial_libvirt_uidgid_v2.debdiff
https://launchpadlibrarian.net/291206174/xenial_base-passwd_libvirt.debdiff
are the latest and most up to date that shall be reviewed by me and then
the SRU Team?

E.g. the version numbers are not following what they should be yet (see
https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging
).
Also since it went a bit back and forth you might want changes in them, so
I would like to have a confirmation of bing up-to-date first.

If you need more help let me know, but you handled all so well until now,
there is no need to fully take over that IMHO.

[Mauricio Faria de Oliveira (mfo)]

Christian,

Alright!
I'll provide the up-to-date backport for libvirt in Xenial and the SRU template.

Per Colin's comment #21 it's not required to SRU base-passwd:
> [snip] you should definitely not waste time SRUing base-passwd [snip]

Thanks.

[Christian Ehrhardt  (paelzer)]

On Tue, Jan 17, 2017 at 8:45 PM, Mauricio Faria de Oliveira <
<email address hidden>> wrote:

> Per Colin's comment #21 it's not required to SRU base-passwd:
> > [snip] you should definitely not waste time SRUing base-passwd [snip]
>

Glad that you memorized that part better than I did :-)
Ok - libvirt SRU only then.

[Mauricio Faria de Oliveira (mfo)]

Hi Christian,

Here's the patch for libvirt in Xenial.
It's the very minimal changes required from Zesty/Debian in order to set the UID (and document the change in the NEWS file).
I didn't backport the debconf warning stuff as it's not essentially required.
Hopefully this is simple/conservative enough for the SRU to occur more safely. :)

I'll now add the SRU template and testing with this patch.

[Mauricio Faria de Oliveira (mfo)]

Verification steps with just attached patch.

----------------------------------------
Test 1) new user, uid available
Result) libvirt-qemu uid is set to 64055
----------------------------------------

# id libvirt-qemu
id: ‘libvirt-qemu’: no such user

# getent passwd 64055
#

# dpkg -i libvirt0_*.deb libvirt-bin_*.deb

# id libvirt-qemu
uid=64055(libvirt-qemu) gid=117(kvm) groups=117(kvm)

----------------------------------------------
Test 2) new user, uid not available
Result) libvirt-qemu uid is _not_ set to 64055
----------------------------------------------

# apt-get purge libvirt-bin

# userdel libvirt-qemu

# id libvirt-qemu
id: ‘libvirt-qemu’: no such user

# useradd --uid 64055 uidtaken

# getent passwd 64055
uidtaken:x:64055:1001::/home/uidtaken:

# dpkg -i libvirt0_*.deb libvirt-bin_*.deb

# id libvirt-qemu
uid=113(libvirt-qemu) gid=117(kvm) groups=117(kvm)

------------------------------------------
Test 3) old user, uid available
Result) libvirt-qemu uid is _not_ changed
------------------------------------------

# apt-get purge libvirt-bin

# userdel uidtaken

# getent passwd 64055
#

# id libvirt-qemu
uid=113(libvirt-qemu) gid=117(kvm) groups=117(kvm)

# dpkg -i libvirt0_*.deb libvirt-bin_*.deb

# id libvirt-qemu
uid=113(libvirt-qemu) gid=117(kvm) groups=117(kvm)

[Mauricio Faria de Oliveira (mfo)]

Christian,

For your consideration...
Since this patch is mainly targeted at first-time installations of libvirt (so that the uid assignment can occur), it would be great if it can make 16.04.2 now that the date slipped 2 weeks (per the public release schedule), if at all possible... so that more users can benefit from it.

Thanks!

[Christian Ehrhardt  (paelzer)]

Sure, I added that the simplified test is just the user check after fresh install.
I can only do as fast as possible - not direct effect to the point release by me.
I can ask a few other people thou to advise on it after I put it into the queue.

And thanks for the paperwork on the SRU - if the requester does that it puts some extra weight by confirming it is not just something I want :-)

Reviewing the change now (but I don't expect major surprises).

[Christian Ehrhardt  (paelzer)]

Slightly adapted the versioning, other than that great work IMHO!
I also agree to keep the "LIBVIRT_QEMU_GID=64055 # not used in Xenial." around.
It doesn't hurt and makes it more clear why it is NOT here.

SRU always being extra cautious - Building/Testing from ppa now.
I need to run now, but very likely sponsoring after I'm back later today.

[Mauricio Faria de Oliveira (mfo)]

Okay, great! Thank you.

[Christian Ehrhardt  (paelzer)]

Ok, test succeeded - sponsoring the upload.

It is in the SRU queue now and I pinged the Release Team for you if there is any chance to slip into 16.04.2

[Mauricio Faria de Oliveira (mfo)]

Okay, thanks again for all you (excellent) support on all this, Christian.

[bugproxy (bugproxy)]

Default Comment by Bridge

[Łukasz Zemczak (sil2100)]

Hello bugproxy, or anyone else affected,

Accepted libvirt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.3.1-1ubuntu10.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

------- Comment on attachment From <email address hidden> 2016-11-14 12:42 EDT-------

Machine Details:
=============
Note:-

System is on a private network. Access the private network via SSH to "banner.isst.aus.stglabs.ibm.com" using your GSA ID and password.
(Banner itself is behind a BSO, so must authenticate through that first.)

Login details :
ssh banner.isst.aus.stglabs.ibm.com [debug/don2rry]

Source Host login:-
chikvm.isst.aus.stglabs.ibm.com [10.33.7.207]
[root/don2rry]

Destination Host Login:-
guskvm.isst.aus.stglabs.ibm.com has address 10.33.32.222

start Guest Login :-
virsh console chig6 --force

IPMI Login :-
-------------
From banner machine run the following :
ssh banner.isst.aus.stglabs.ibm.com [debug/don2rry ]

ipmitool -I lanplus -H fsp-chi -P don2rry sol deactivate
ipmitool -I lanplus -H fsp-chi -P don2rry sol activate

-----------------------------------------------------------------------------
                             TESTING INFORMATION
-----------------------------------------------------------------------------

SYSTEM INFORMATION
------------------
  HOST NAME or NETWORK ADDRESS: chikvm.isst.aus.stglabs.ibm.com
 [10.33.7.207]
guskvm..isst.aus.stglabs.ibm.com 10.33.32.222

  Sapphire FIRMWARE LEVEL: - FW860.00 (SV860_056)

-----------------------------------------------------------------------------
                            DEBUGGING INFORMATION
-----------------------------------------------------------------------------

DEBUG / LOGIN INFORMATION
-------------------------
  ERROR LOG: on local machine
  FSP LOGIN: see name above [dev/FipSdev]
RECENT SYSTEM CHANGES : none
-----------------------------------------
  none

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-01-20 08:47 EDT-------
Verification done with Xenial. All good!

# lsb_release -d
Description: Ubuntu 16.04.1 LTS

Previous Package:
----------------

0) No existing user/uid:

# id libvirt-qemu
id: ?libvirt-qemu?: no such user

# getent passwd 64055
#

User is created w/ the next available uid

# apt-get install --yes --no-install-recommends libvirt{0,-bin}=1.3.1-1ubuntu10.6

# id libvirt-qemu
uid=110(libvirt-qemu) gid=117(kvm) groups=117(kvm)

# getent passwd 64055
#

Proposed Package:
----------------

1) With existing user (e.g., package update)

# id libvirt-qemu
uid=110(libvirt-qemu) gid=117(kvm) groups=117(kvm)

User uid remains the same:

# apt-get install --yes --no-install-recommends libvirt{0,-bin}=1.3.1-1ubuntu10.7

# id libvirt-qemu
uid=110(libvirt-qemu) gid=117(kvm) groups=117(kvm)

# getent passwd 64055
#

2) Without existing user (e.g., new install)

# apt-get purge --yes libvirt{0,-bin}
Reading package lists... Done

# userdel libvirt-qemu

# id libvirt-qemu
id: ?libvirt-qemu?: no such user

# getent passwd 64055
#

User uid is set to the reserved uid:

# apt-get install --yes --no-install-recommends libvirt{0,-bin}=1.3.1-1ubuntu10.7

# id libvirt-qemu
uid=64055(libvirt-qemu) gid=117(kvm) groups=117(kvm)

# getent passwd 64055
libvirt-qemu:x:64055:117:Libvirt Qemu,,,:/var/lib/libvirt:/bin/false

3) Without existing user (e.g., new install) _but_ with uid taken

# apt-get purge --yes libvirt{0,-bin}

# id libvirt-qemu
id: ?libvirt-qemu?: no such user

# getent passwd 64055
uidtaken:x:64055:1001::/home/uidtaken:

# apt-get install --yes --no-install-recommends libvirt{0,-bin}=1.3.1-1ubuntu10.7

User uid is set to the next available uid:

# id libvirt-qemu
uid=110(libvirt-qemu) gid=117(kvm) groups=117(kvm)

# getent passwd 64055
uidtaken:x:64055:1001::/home/uidtaken:

------- Comment From <email address hidden> 2017-01-20 08:49 EDT-------
BTW,
*again*, sorry for all the noise/attachments from the bug mirroring.
This problem has already been passed along to its development team.
I have no idea why this happens.

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

------- Comment on attachment From <email address hidden> 2016-11-14 12:42 EDT-------

Machine Details:
=============
Note:-

System is on a private network. Access the private network via SSH to "banner.isst.aus.stglabs.ibm.com" using your GSA ID and password.
(Banner itself is behind a BSO, so must authenticate through that first.)

Login details :
ssh banner.isst.aus.stglabs.ibm.com [debug/don2rry]

Source Host login:-
chikvm.isst.aus.stglabs.ibm.com [10.33.7.207]
[root/don2rry]

Destination Host Login:-
guskvm.isst.aus.stglabs.ibm.com has address 10.33.32.222

start Guest Login :-
virsh console chig6 --force

IPMI Login :-
-------------
From banner machine run the following :
ssh banner.isst.aus.stglabs.ibm.com [debug/don2rry ]

ipmitool -I lanplus -H fsp-chi -P don2rry sol deactivate
ipmitool -I lanplus -H fsp-chi -P don2rry sol activate

-----------------------------------------------------------------------------
                             TESTING INFORMATION
-----------------------------------------------------------------------------

SYSTEM INFORMATION
------------------
  HOST NAME or NETWORK ADDRESS: chikvm.isst.aus.stglabs.ibm.com
 [10.33.7.207]
guskvm..isst.aus.stglabs.ibm.com 10.33.32.222

  Sapphire FIRMWARE LEVEL: - FW860.00 (SV860_056)

-----------------------------------------------------------------------------
                            DEBUGGING INFORMATION
-----------------------------------------------------------------------------

DEBUG / LOGIN INFORMATION
-------------------------
  ERROR LOG: on local machine
  FSP LOGIN: see name above [dev/FipSdev]
RECENT SYSTEM CHANGES : none
-----------------------------------------
  none

[bugproxy (bugproxy)]

Default Comment by Bridge

[Christian Ehrhardt  (paelzer)]

Verification was done 17 days ago, I think that should be ok to migrate right?

Also we have another set of SRU in queue waiting for this to get out of the way.

[Christian Ehrhardt  (paelzer)]

I found (thank apw) that this triggers an unrelated dep8 test fail which is the reason it stalled.
Investigating ...

[Christian Ehrhardt  (paelzer)]

Issue did no more show up on two retries of the test, also what showed up was totally unrelated to the upload we made.
That said should be safe to migrate now.

[Christian Ehrhardt  (paelzer)]

FYI - I pinged internally again to consider it for the point release as you requested it.
Quoting you from commend 57 - "Since this patch is mainly targeted at first-time installations of libvirt (so that the uid assignment can occur)".

But since it is not auto-installed an "apt update" prior to install should save you no matter if on the iso or not.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 1.3.1-1ubuntu10.7

---------------
libvirt (1.3.1-1ubuntu10.7) xenial; urgency=medium

  [ Mauricio Faria de Oliveira ]
  * Use a fixed uid/gid for libvirt-qemu user to avoid permission
    issues on network filesystems like NFS (backport) (LP: #1637601).

 -- Christian Ehrhardt <email address hidden> Wed, 18 Jan 2017 16:15:34 +0100

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Mauricio Faria de Oliveira (mfo)]

Hi Christian,

Thanks for handling it to xenial-updates.

BTW, that point about the package to make the point release ISO is so that people who use the ISO as package repository (which happens in some scenarios like xCAT clusters), and not always get more updates from the internet, can get it.

cheers

[Christian Ehrhardt  (paelzer)]

I picked up a report today on IRC which I wanted to discuss here with the audience that created the change.

The report was that with 16.04.2 the libvirt user now shows up on e.g. gdm dropdown when selecting users.

I thought it filters system users, but then --system (=get a low number) and providing the number from base-passwd kind of oppose each other.
Yet I seem not to have puzzled together the full picture, there are many more UIDs in base-passwd yet I never heard about those in gdb.
Would there be a different option that one has to set to "create" them differently?