migrate lxc android container to lxd

[You-Sheng Yang]

Several things to do here:
* include lxd, lxd-client in default installation,
* writable path for lxd: lxd resides at /var/lib/lxd and it takes write permission to fully function,
* migrate lxc tmpfs-based rootfs to lxd image, lxc configs to lxd profiles, etc.,
* migrate systemd service definitions, etc.

[Steps to reproduce current setup]

0. rebuild avila kernel with additional patches from https://code.launchpad.net/~vicamo/avila/+git/kernel-3.10/+merge/310849 , or just use the prebuilt one in the attachments.

1. follow bug 1641505 and https://bugs.launchpad.net/canonical-devices-system-image/+bug/1641505/comments/2 to enable adbd emergency mode and disable current lxc-based android container.

2. add /var/lib/lxd to writable paths, install lxd and lxd-client, and then reboot the device so that writable path may take effect.

3. initialize lxd. This can take a while (2 minutes for me).

  $ sudo lxd init --auto

4. apply changes from https://code.launchpad.net/~vicamo/lxc-android-config/lxd-migration/+merge/311103 . This cause the system to automatically import android initrd as an lxd image and to start it up on boot.

[You-Sheng Yang (vicamo)]

Again, avila's kernel is pretty outdated, and its user namespace support is so poor that it's actually mis-behave on a certain security related operations.

In security/commoncap.c, function cap_prctl_drop, it should check `ns_capable(...)`, not just `capable(...)`. See https://github.com/lxc/lxd/issues/2039 . At lease commit 160da84 "userns: Allow PR_CAPBSET_DROP in a user namespace" has to be cherry-picked, but besides that, I also found a lot similar places to be fixed.

[You-Sheng Yang (vicamo)]

With comment #1 fixed with a patched kernel, it seems we bumped into bug 1625916 again. Need to run lxc in the background.

[You-Sheng Yang (vicamo)]

Again, the fundermental reason init dies is a critical core service healthd dies, so it's not really about background/foreground here. And healthd dies because it fails to open /dev/binder. It follows in lxd, ueventd is not correctly running yet.

[You-Sheng Yang (vicamo)]

Prebuilt boot image with https://code.launchpad.net/~vicamo/avila/+git/kernel-3.10/+merge/310849 applied.

[You-Sheng Yang (vicamo)]

Script to generate android lxd image from /android/system/boot/android-ramdisk.img

[You-Sheng Yang (vicamo)]

Current status: with following three lines added to the profile, ueventd processes devices as usual. But I can't still found any daemon process with executable located in /system running. /proc/$(pidof init)/mounts shows /system and all other partitions have been correctly mounted.

  lxc.cgroup.devices.allow = a
  lxc.mount.auto=
  lxc.mount.auto=sys:rw proc:mixed cgroup:mixed

[You-Sheng Yang (vicamo)]

Script to generate android lxd image from /android/system/boot/android-ramdisk.img

[You-Sheng Yang (vicamo)]

With some hacking with systemd, now lxd image boots and GUI is available. Some refinements are needed. Known issues:

1. lxc tries to write client.srt to /root/.config/lxc, which should be read-only by default. Add --force-local to every lxc command invoked fixes this issue.

2. Currently I'm launching lxd container via `lxc launch -e ...`, but it would be nice if that's somehow covered by lxd daemon. Don't know if it's possible to create an ephemeral yet autostart container in a formal way.

3. wifi/bluetooth is not available at boot because urfkill starts before the container. Need a manual restart to urfkill.

[You-Sheng Yang (vicamo)]

Another issue seems to relate to the kernel version again. In logcat messages I found logd restarts over and over again due to failure when sending audit netlink messages. It seems lxd fails to setup some cgroups: cgroups controllers for blkio/memory/network/pids/... not found.

I/UpstartPropertyWatcher( 1833): Property changed: init.svc.logd=running
E/libaudit(30123): Error sending data over the netlink socket: Unknown error -111
E/libaudit(30123): Could net set pid for audit events, error: Connection refused
I/UpstartPropertyWatcher( 1833): Property changed: init.svc.logd=restarting

[You-Sheng Yang (vicamo)]

Add audit=0 to kernel command line and logd ceases to try kernel audit, and the rest of the system (lxd/snap) doesn't seem to be affected.

[You-Sheng Yang (vicamo)]

AppArmor is not functioning this way .... Hmmm

[You-Sheng Yang (vicamo)]

audit namespace issue fixed in upstream https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f9441639e6319f0c0e12bd63fa2f58990af0a9d2 and https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=33faba7fa7f2288d2f8aaea95958b2c97bf9ebfb . Already included in my kernel merge proposal.

[You-Sheng Yang (vicamo)]

Update prebuilt boot.img with current kernel changes in https://code.launchpad.net/~vicamo/avila/+git/kernel-3.10/+merge/310849

[You-Sheng Yang (vicamo)]

All known issues in comment 8 addressed, so I think that's all we need for lxd porting.