Comment 9 for bug 1387214

After a lot of deep digging into the bind mount, loop driver, and buffer cache and tracking the corrupt pages back down the layers of the stack we've sanity checked this down to the image. The smoking gun was the kernel message:

Nov 6 12:15:16 ubuntu-phablet kernel: [ 3.940485] do_mount: /dev/loop0 -> /root [<null>]
Nov 6 12:15:16 ubuntu-phablet kernel: [ 3.941095] EXT2-fs (loop0): warning: mounting unchecked fs, running e2fsck is recommended
Nov 6 12:15:16 ubuntu-phablet kernel: [ 3.941431] do_mount return -> 0

(apologies for my extra debug).

So it appears that /dev/loop0 is being mounted and it is corrupted. I ran fsck on /userdata/system.img and /userdata/ubuntu.img only to find that the system.img needed some fixing:

fsck /userdata/system.img
fsck from util-linux 2.25
e2fsck 1.42.10 (18-May-2014)
/userdata/system.img was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Unattached inode 3225
Connect to /lost+found<y>? yes
Inode 3225 ref count is 2, should be 1. Fix<y>? yes
Unattached inode 3709
Connect to /lost+found<y>? yes
Inode 3709 ref count is 2, should be 1. Fix<y>? yes
Unattached inode 3808
Connect to /lost+found<y>? yes
Inode 3808 ref count is 2, should be 1. Fix<y>? yes
Unattached inode 4427
Connect to /lost+found<y>? yes
Inode 4427 ref count is 2, should be 1. Fix<y>? yes
Unattached inode 4485
Connect to /lost+found<y>? yes
Inode 4485 ref count is 2, should be 1. Fix<y>? yes
Unattached inode 5889
Connect to /lost+found<y>? yes
Inode 5889 ref count is 2, should be 1. Fix<y>? yes
Unattached inode 5943
Connect to /lost+found<y>? yes
Inode 5943 ref count is 2, should be 1. Fix<y>? yes
Unattached inode 7853
Connect to /lost+found<y>? yes
Inode 7853 ref count is 2, should be 1. Fix<y>? yes
yyyPass 5: Checking group summary information
Block bitmap differences: -70903 -71144 -71201 -(71674--71675) -71727 -71852 -72689 -72757 -(74519--74520) -74869 -74961 +(92082--92087) +(92089--92092) -92102 +92104 +92114 +y92119 +(92121--92131)
Fix<y>? yes
Free blocks count wrong for group #13 (8813, counted=8820).
Fix<y>? yes
Free blocks count wrong (133222, counted=133229).
Fix<y>? yes
Inode bitmap differences: +(19989--20010) +(20013--20014) -(20545--20549)y -(20551--20569)
Fix<y>? yes
Free inodes count wrong for group #13 (3225, counted=3232).
Fix<y>? yes
Directories count wrong for group #13 (761, counted=760).
Fix<y>? yes
Free inodes count wrong (81946, counted=81953).
Fix<y>? yes

/userdata/system.img: ***** FILE SYSTEM WAS MODIFIED *****
/userdata/system.img: ***** REBOOT LINUX *****

So, there are two big issues outstanding, most probably in the user space shutdown and initrd stages:

1. The file system is not being flushed and unmounted properly.
2. The file system is not being fsck'd before mounting - this is a cardinal sin IMHO

The end result is mounting a corrupt file system that is causing the garbage in the apparmor files.