broken postinst test for uvtool-libvirt

[Mike McCracken]

Installing uvtool-libvirt *inside an lxc container* on utopic fails due to a test in the postinst script.

It uses socat on the libvirt socket, which fails, despite libvirt being installed correctly.

ubuntu@uoi-bootstrap:~$ sudo apt-get install -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libfreetype6 os-prober
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up uvtool-libvirt (0~bzr92-0ubuntu2) ...
2015/01/08 13:01:34 socat[10184] E read(3, 0x13b2a30, 8192): Permission denied
libvirtd does not appear to be listening on "/var/run/libvirt/libvirt-sock".
On Ubuntu, libvirtd is managed with the "libvirt-bin" upstart job.
Repair libvirtd, then reconfigure uvtool-libvirt with:
    sudo apt-get -f install
dpkg: error processing package uvtool-libvirt (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 uvtool-libvirt
E: Sub-process /usr/bin/dpkg returned an error code (1)

ubuntu@uoi-bootstrap:~$ ps -ef | grep libvirt
libvirt+ 9556 1 0 09:52 ? 00:00:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
root 9557 9556 0 09:52 ? 00:00:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
root 9854 1 0 10:24 ? 00:00:00 /usr/sbin/libvirtd -d
ubuntu 10155 10120 0 12:56 pts/0 00:00:00 grep libvirt

ubuntu@uoi-bootstrap:~$ groups
ubuntu adm dialout cdrom floppy sudo audio dip video plugdev netdev libvirtd

ubuntu@uoi-bootstrap:~$ virsh list
 Id Name State
----------------------------------------------------

for a little more context, there are notes here:
https://gist.github.com/mikemccracken/53c665e6094db21efc03

[Serge Hallyn (serge-hallyn)]

uvtool could work around this issue by not using socat to verify that libvirt is running.

apparmor doesn't appear to be reporting any denials, even in complain mode. So i don't know what is actually causing the denial. However, running the container with lxc.aa_profile = unconfined does allow it to succeed, so it does appear to be apparmor causing it.

Once libvirt is installed in the container, you can test by hand by simply doing:

sudo socat UNIX-CONNECT:/var/run/libvirt/libvirt-sock -

Until uvtool or apparmor packages are updated, you can edit the postinst file to remove the socat check by hand to work around this, or put the container into unconfined mode.

[Serge Hallyn (serge-hallyn)]

(Marked confirmed because at least Mike and I have reproduced this).

i've reproduced this on both 3.13 and 3.18 kernels - all with utopic host, so utopic apparmor package.

[Robie Basak (racb)]

> uvtool could work around this issue by not using socat to verify that libvirt is running.

What should it do instead? The reason for the check is that various users were getting unexplained postinst failures because libvirtd was broken on their system in some way. This was for interactive/developer uvtool use. The check was to provide a better error message.

So I'm not sure how to proceed right now.

[Serge Hallyn (serge-hallyn)]

An apparmor policy file to trigger container policies on a testcase

[Serge Hallyn (serge-hallyn)]

A trivial testcase.

[Serge Hallyn (serge-hallyn)]

To most easily reproduce this,

1. install lxc and libvirt-bin on your host

2. Grab the testcase from comment #5, compile it twice, once as $HOME/test/aasocat and once as $HOME/test/aasocat2

3. Grab the attachmen tfrom comment #4. call it /etc/apparmor.d/home.$user.test.aa_socat, and change the path listed in the file to /home/$user/test/aa_socat

4. load the policy (apparmor_parser -r /etc/apparmor/home.$user.test.aa_socat)

Now when you run $HOME/test/aa_socat and type "hi there <return>", you'll see the failure. When you run $HOME/test/aa_socat2, you'll see "success". Nothing shows up in /var/log/audit/audit.log however.

Here is my terminal output:

0 ✓ serge@sl ~/test $ sudo ./aasocat2
[sudo] password for serge:
hi there
system: Success
0 ✓ serge@sl ~/test $ sudo ./aasocat
hi there
2015/01/09 14:23:47 socat[10526] E read(3, 0x1fdb9b0, 8192): Permission denied
system: Success

[Serge Hallyn (serge-hallyn)]

Here is a simpler apparmor policy (that is, no tincluding lxc abstractions):

#include <tunables/global>

/home/serge/test/aasocat {
  #include <abstractions/base>
  #include <abstractions/dbus>
  /home/serge/test/aasocat ixr,
  capability,
  network,
  file,
  /var/** rw,
  /usr/bin/** ixr,
  /bin/** ixr,
}

[Christian Boltz (cboltz)]

> apparmor doesn't appear to be reporting any denials, even in complain mode.

Does the profile or one of the abstractions you use contain "deny" rules? (Hint: deny rules are enforced even in complain mode.)

[John Johansen (jjohansen)]

So I can verify that this failure is due to how apparmor is handling a unix domain sockets that are deleted/shutdown. I have a kernel with a tentative fix building.

[John Johansen (jjohansen)]

please test with http://people.canonical.com/~jj/linux-image-3.18.0-8-generic_3.18.0-8.9+lp1408833_amd64.deb and verify that it fixes the problem.

[Mike McCracken (mikemc)]

I can confirm that the linked kernel fixes the original issue for me. I can create a container and install uvtool, and cloud-init finishes successfully.

ubuntu@uoi-bootstrap:/$ uname -a
Linux uoi-bootstrap 3.18.0-8-generic #9+lp1408833 SMP Mon Jan 12 17:08:53 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@uoi-bootstrap:/$ sudo apt-cache policy uvtool
uvtool:
  Installed: 0~bzr92-0ubuntu2
  Candidate: 0~bzr92-0ubuntu2
  Version table:
 *** 0~bzr92-0ubuntu2 0
        500 http://archive.ubuntu.com/ubuntu/ utopic/universe amd64 Packages
        100 /var/lib/dpkg/status

ubuntu@uoi-bootstrap:/$ sudo cat /run/cloud-init/result.json
{
 "v1": {
  "errors": [],
  "datasource": "DataSourceNoCloudNet [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net]"
 }
}

[John Johansen (jjohansen)]

okay thanks, I'll get this patch submitted

[John Johansen (jjohansen)]

Alright so after further looking and working on this, the original patch was insufficient for all the cases. I have a new kernel available for testing at
http://people.canonical.com/~jj/linux-image-3.18.0-14-generic_3.18.0-14.15+lp1408833_amd64.deb

It passes for me but I would like it verified if possible before submitting

[Mike McCracken (mikemc)]

Hi, my apologies for the slow reply. The second patch also works for me.
Thanks!

[Maarten (mthibaut-f)]

I've tried installing this kernel on my amd64 utopic box, but the system fails to boot with it. Should I run unstable or testing to try it out?

My current setup is 3.16.0-31-generic

Thanks!

[Maarten (mthibaut-f)]

I was able to run this kernel inside a VMware environment, but when I ran it on bare metal I got a busybox prompt as it wasn't able to mount the root directory. I think there's something seriously wrong with this kernel.

[Maarten (mthibaut-f)]

This is the erros I'm getting. Is the same for LVM and regular ext4:

[ 1.835313] i8042: No controller found
  Gave up waiting for root device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/mapper/sj--il--ucs--b--1--vg-root does not exist. Dropping to a shell!

BusyBox v1.22.1 (Ubuntu 1:1.22.0-8ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-utopic' to 'verification-done-utopic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.16.0-34.45

---------------
linux (3.16.0-34.45) utopic; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1435400

  [ Andy Whitcroft ]

  * [Packaging] generate live watchdog blacklists
    - LP: #1432837

  [ Chris J Arges ]

  * [Config] Add ibmvfc to d-i
    - LP: #1416001

  [ John Johansen ]

  * SAUCE: (no-up): apparmor: fix mediation of fs unix sockets
    - LP: #1408833

  [ Seth Forshee ]

  * [Config] updateconfigs - enable X86_UP_APIC_MSI

  [ Upstream Kernel Changes ]

  * cdc-acm: add sanity checks
    - LP: #1413992
  * x86: thinkpad_acpi.c: fixed spacing coding style issue
    - LP: #1417915
  * thinkpad_acpi: support new BIOS version string pattern
    - LP: #1417915
  * powernv: Use _GLOBAL_TOC for opal wrappers
    - LP: #1431196
  * Btrfs: clear compress-force when remounting with compress option
    - LP: #1434183
  * Btrfs: send, don't delay dir move if there's a new parent inode
    - LP: #1434223
  * [media] em28xx: fix em28xx-input removal
    - LP: #1434595
  * [media] em28xx: ensure "closing" messages terminate with a newline
    - LP: #1434595
  * [media] em28xx-input: fix missing newlines
    - LP: #1434595
  * [media] em28xx-core: fix missing newlines
    - LP: #1434595
  * [media] em28xx-audio: fix missing newlines
    - LP: #1434595
  * [media] em28xx-audio: fix missing newlines
    - LP: #1434595
  * [media] em28xx-dvb: fix missing newlines
    - LP: #1434595
  * [media] em28xx-video: fix missing newlines
    - LP: #1434595
  * ARM: pxa: add regulator_has_full_constraints to corgi board file
    - LP: #1434595
  * ARM: pxa: add regulator_has_full_constraints to poodle board file
    - LP: #1434595
  * ARM: pxa: add regulator_has_full_constraints to spitz board file
    - LP: #1434595
  * hx4700: regulator: declare full constraints
    - LP: #1434595
  * HID: input: fix confusion on conflicting mappings
    - LP: #1434595
  * HID: fixup the conflicting keyboard mappings quirk
    - LP: #1434595
  * ARM: dts: tegra20: fix GR3D, DSI unit and reg base addresses
    - LP: #1434595
  * megaraid_sas: disable interrupt_mask before enabling hardware
    interrupts
    - LP: #1434595
  * PCI: Generate uppercase hex for modalias var in uevent
    - LP: #1434595
  * usb: core: buffer: smallest buffer should start at ARCH_DMA_MINALIGN
    - LP: #1434595
  * tty/serial: at91: enable peripheral clock before accessing I/O
    registers
    - LP: #1434595
  * tty/serial: at91: fix error handling in atmel_serial_probe()
    - LP: #1434595
  * axonram: Fix bug in direct_access
    - LP: #1434595
  * btrfs: fix leak of path in btrfs_find_item
    - LP: #1434595
  * ksoftirqd: Enable IRQs and call cond_resched() before poking RCU
    - LP: #1434595
  * TPM: Add new TPMs to the tail of the list to prevent inadvertent change
    of dev
    - LP: #1434595
  * char: tpm: Add missing error check for devm_kzalloc
    - LP: #1434595
  * tpm_tis: verify interrupt during init
    - LP: #1434595
  * tpm: Fix NULL return in tpm_ibmvtpm_get_desired_dma
    - LP: #1434595
  * tpm/tpm_i2c_stm_st33: Fix potential bug in tpm_stm_i2c_send
    - LP: #1434595
  * tpm/tpm_i2c_stm_st33:...

[Adam Stokes (adam-stokes)]

The problem exists in kernel 3.16.0-30-generic on trusty as well. Can we get this fix backported to trusty?

[Enki (y-b)]

Hi all

Please when this issue will be solved? We are completely stuck and no workaround available.

Many thanks for your help.

[Adam Stokes (adam-stokes)]

Enki,

Looks like you just need the enablement kernel: See here for more information:

https://wiki.ubuntu.com/TrustyTahr/ReleaseNotes#LTS_Hardware_Enablement_Stack

You'll want to make sure you're on a 3.16-0-38.52 or higher

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.3.0-1.10

---------------
linux (4.3.0-1.10) xenial; urgency=low

  [ Andy Whitcroft ]

  * [Config] make IBMVETH consistent on powerpc/ppc64el
    - LP: #1521712
  * [Config] follow ibmvscsi name change
    - LP: #1521712
  * [Config] move ibm disk and ethernet drivers to linux-image
    - LP: #1521712
  * [Config] include ibmveth in nic-modules for ppc64el
    - LP: #1521712
  * [Config] s390x -- disable abi/module checks for s390x

  [ Tim Gardner ]

  * [Config] Add spl/zfs provides to generic and powerpc64-smp
  * [Config] Add zfs to d-i fs-core-modules

  [ Upstream Kernel Changes ]

  * KVM: x86: work around infinite loop in microcode when #AC is delivered
  * KVM: svm: unconditionally intercept #DB
  * Btrfs: fix truncation of compressed and inlined extents
  * staging/dgnc: fix info leak in ioctl
  * [media] media/vivid-osd: fix info leak in ioctl
  * crypto: asymmetric_keys - remove always false comparison
  * X.509: Fix the time validation [ver #2]
  * isdn_ppp: Add checks for allocation failure in isdn_ppp_open()
  * ppp, slip: Validate VJ compression slot parameters completely

 -- Andy Whitcroft <email address hidden> Tue, 01 Dec 2015 21:37:13 +0000