ERROR processing policydb rules for profile lxc-container-default, failed to load

Bug #1295774 reported by Florian Engelmann on 2014-03-21
32
This bug affects 7 people
Affects Status Importance Assigned to Milestone
apparmor (Ubuntu)
Undecided
Tyler Hicks

Bug Description

OS: Ubuntu Trusty 14.04 64bit
Kernel: Linux global04-jobs2 3.13.0-18-generic #38-Ubuntu SMP Mon Mar 17 21:40:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

ii apparmor 2.8.95~2430-0ubuntu2 amd64 User-space parser utility for AppArmor
ii apparmor-utils 2.8.95~2430-0ubuntu2 amd64 Utilities for controlling AppArmor

# service apparmor restart
 * Reloading AppArmor profiles Enocoding of mount rule failed
ERROR processing policydb rules for profile lxc-container-default, failed to load

John Johansen (jjohansen) wrote :

Can you please tar up the /etc/apparmor.d/ directory and attach it to the bug

Jamie Strandboge (jdstrand) wrote :

FYI, I was not able to reproduce this in a trusty VM and the lxc autopkgtests passed: https://jenkins.qa.ubuntu.com/view/Trusty/view/AutoPkgTest/job/trusty-adt-lxc/

Changed in apparmor (Ubuntu):
status: New → Incomplete
Florian Engelmann (engelmann) wrote :

This happened on two of six servers. All of the same hardware, firmware versions and same ubuntu installation. I am very confused.

I tried to purge apparmor and lxc - reboot and reinstall them but this did not fix the issue even if both directories (/etc/apparmor and /etc/apparmor.d) were deleted.

Florian Engelmann (engelmann) wrote :
Download full text (5.2 KiB)

root@xxxxxxxxx:~# apparmor_parser -vd /etc/apparmor.d/lxc-containers
----- Debugging built structures -----
Name: lxc-container-default
Profile Mode: Enforce
Capabilities: chown dac_override dac_read_search fowner fsetid kill setgid setuid setpcap linux_immutable net_bind_service net_broadcast net_admin net_raw ipc_lock ipc_owner sys_module sys_rawio sys_chroot sys_ptrace sys_pacct sys_admin sys_boot sys_nice sys_resource sys_time sys_tty_config mknod lease audit_write audit_control setfcap mac_overridesyslog
Quiet Caps: chown dac_override dac_read_search fowner fsetid kill setgid setuid setpcap linux_immutable net_bind_service net_broadcast net_admin net_raw ipc_lock ipc_owner sys_module sys_rawio sys_chroot sys_ptrace sys_pacct sys_admin sys_boot sys_nice sys_resource sys_time sys_tty_config mknod lease audit_write audit_control setfcap mac_overridesyslog
Network: <all>
--- Entries ---
Mode: rwalkmx:rwalkmx Name: (/{**,})
 link: (/**)
Mode: rwalkx:rwalkx Name: (/proc//kmem)
 link: (/**)
Mode: rwalkx:rwalkx Name: (/proc//mem)
 link: (/**)
Mode: walkx:walkx Name: (/proc//sys/fs/**)
 link: (/**)
Mode: walkx:walkx Name: (/proc//sys/kernel/*/**)
 link: (/**)
Mode: walkx:walkx Name: (/proc//sys/kernel/[^s][^h][^m]*)
 link: (/**)
Mode: rwalkx:rwalkx Name: (/proc//sysrq-trigger)
 link: (/**)
Mode: walkx:walkx Name: (/sys/[^f]*/**)
 link: (/**)
Mode: walkx:walkx Name: (/sys/f[^s]*/**)
 link: (/**)
Mode: rwalkx:rwalkx Name: (/sys/firmware/efi/efivars/**)
 link: (/**)
Mode: walkx:walkx Name: (/sys/fs/[^c]*/**)
 link: (/**)
Mode: walkx:walkx Name: (/sys/fs/c[^g]*/**)
 link: (/**)
Mode: walkx:walkx Name: (/sys/fs/cg[^r]*/**)
 link: (/**)
Mode: rwalkx:rwalkx Name: (/sys/kernel/security/**)
 link: (/**)

Name: lxc-container-default-with-mounting
Profile Mode: Enforce
Capabilities: chown dac_override dac_read_search fowner fsetid kill setgid setuid setpcap linux_immutable net_bind_service net_broadcast net_admin net_raw ipc_lock ipc_owner sys_module sys_rawio sys_chroot sys_ptrace sys_pacct sys_admin sys_boot sys_nice sys_resource sys_time sys_tty_config mknod lease audit_write audit_control setfcap mac_overridesyslog
Quiet Caps: chown dac_override dac_read_search fowner fsetid kill setgid setuid setpcap linux_immutable net_bind_service net_broadcast net_admin net_raw ipc_lock ipc_owner sys_module sys_rawio sys_chroot sys_ptrace sys_pacct sys_admin sys_boot sys_nice sys_resource sys_time sys_tty_config mknod lease audit_write audit_control setfcap mac_overridesyslog
Network: <all>
--- Entries ---
Mode: rwalkmx:rwalkmx Name: (/{**,})
 link: (/**)
Mode: rwalkx:rwalkx Name: (/proc//kmem)
 link: (/**)
Mode: rwalkx:rwalkx Name: (/proc//mem)
 link: (/**)
Mode: walkx:walkx Name: (/proc//sys/fs/**)
 link: (/**)
Mode: walkx:walkx Name: (/proc//sys/kernel/*/**)
 link: (/**)
Mode: walkx:walkx Name: (/proc//sys/kernel/[^s][^h][^m]*)
 link: (/**)
Mode: rwalkx:rwalkx Name: (/proc//sysrq-trigger)
 link: (/**)
Mode: walkx:walkx Name: (/sys/[^f]*/**)
 link: (/**)
Mode: walkx:walkx Name: (/sys/f[^s]*/**)
 link: (/**)
Mode: rwalkx:rwalkx Name: (/sys/firmware/efi/efivars/**)
 link: (/**)
Mode: walkx:walkx Name: (/sys/fs/[^c]*/**)
 link: (/**)
Mode: walkx:walkx Name: (/sys/f...

Read more...

Florian Engelmann (engelmann) wrote :

starting lxc containers with

lxc.aa_profile = unconfined

is working but without I get these errors:

      lxc-start 1395510515.495 ERROR lxc_apparmor - No such file or directory - failed to change apparmor profile to lxc-container-default
      lxc-start 1395510515.495 ERROR lxc_sync - invalid sequence number 1. expected 4
      lxc-start 1395510515.495 ERROR lxc_start - failed to spawn 'xxxxxx'
      lxc-start 1395510515.496 ERROR lxc_cgfs - Device or resource busy - cgroup_rmdir: failed to delete /sys/fs/cgroup/hugetlb/lxc/xxxxxxx-2
      lxc-start 1395510515.496 ERROR lxc_cgfs - Device or resource busy - cgroup_rmdir: failed to delete /sys/fs/cgroup/perf_event/lxc/xxxxxx-2
      lxc-start 1395510515.496 ERROR lxc_cgfs - Device or resource busy - cgroup_rmdir: failed to delete /sys/fs/cgroup/blkio/lxc/xxxxxxxx-2
      lxc-start 1395510515.496 ERROR lxc_cgfs - Device or resource busy - cgroup_rmdir: failed to delete /sys/fs/cgroup/freezer/lxc/xxxxxxxxxxxxx-2
      lxc-start 1395510515.496 ERROR lxc_cgfs - Device or resource busy - cgroup_rmdir: failed to delete /sys/fs/cgroup/devices/lxc/xxxxxxxxxxxx-2
      lxc-start 1395510515.496 ERROR lxc_cgfs - Device or resource busy - cgroup_rmdir: failed to delete /sys/fs/cgroup/memory/lxc/xxxxxx-2
      lxc-start 1395510515.496 ERROR lxc_cgfs - Device or resource busy - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpuacct/lxc/xxxxxx-2
      lxc-start 1395510515.496 ERROR lxc_cgfs - Device or resource busy - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpu/lxc/xxxxxxxx-2
      lxc-start 1395510515.497 ERROR lxc_cgfs - Device or resource busy - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpuset/lxc/xxxxxxx-2
      lxc-start 1395510515.557 ERROR lxc_commands - command get_cgroup failed to receive response

Florian Engelmann (engelmann) wrote :
Download full text (7.1 KiB)

root@global05:~# rsync -r --no-dirs -I -v -c --no-times -n /etc/ global04:/etc/ | grep -v ^skipping
sending incremental file list
aliases.db
hostname
hosts
mailname
mtab
popularity-contest.conf
shadow
shadow-
apparmor.d/sbin.dhclient
apparmor.d/usr.sbin.rsyslogd
apparmor.d/usr.sbin.tcpdump
apparmor.d/cache/lxc-containers
apparmor.d/cache/sbin.dhclient
apparmor.d/cache/usr.sbin.tcpdump
apparmor.d/local/sbin.dhclient
apparmor.d/local/usr.sbin.rsyslogd
apparmor.d/local/usr.sbin.tcpdump
apparmor/init/
apparmor/init/network-interface-security/
console-setup/cached.kmap.gz
lvm/archive/lxc1_00000-1857182547.vg
lvm/archive/lxc1_00001-53930154.vg
lvm/archive/lxc1_00002-541588005.vg
lvm/archive/lxc1_00003-141333383.vg
lvm/archive/lxc1_00004-399699798.vg
lvm/archive/lxc1_00005-1503595767.vg
lvm/archive/lxc1_00006-1107661694.vg
lvm/archive/lxc1_00007-2040287556.vg
lvm/archive/lxc1_00008-1604263375.vg
lvm/archive/lxc1_00009-330752173.vg
lvm/archive/lxc1_00010-763042882.vg
lvm/archive/lxc1_00011-596258448.vg
lvm/archive/lxc1_00012-1603213455.vg
lvm/archive/lxc1_00013-169959787.vg
lvm/archive/lxc1_00014-1107499866.vg
lvm/archive/lxc1_00015-579308241.vg
lvm/archive/lxc1_00016-2009102655.vg
lvm/archive/lxc1_00017-1238494450.vg
lvm/archive/lxc1_00018-1600929065.vg
lvm/archive/lxc1_00019-692732338.vg
lvm/archive/lxc1_00020-218351101.vg
lvm/archive/lxc1_00021-1764864896.vg
lvm/archive/lxc1_00022-646456308.vg
lvm/archive/lxc1_00023-366512818.vg
lvm/archive/lxc1_00024-2009751024.vg
lvm/archive/lxc1_00025-351826666.vg
lvm/archive/lxc1_00026-1024113898.vg
lvm/archive/lxc1_00027-760861153.vg
lvm/archive/lxc1_00028-1025681260.vg
lvm/archive/lxc1_00029-613892203.vg
lvm/archive/lxc1_00030-2091926349.vg
lvm/archive/lxc1_00031-706594095.vg
lvm/archive/lxc1_00032-1679912731.vg
lvm/archive/lxc1_00033-1341836069.vg
lvm/archive/lxc1_00034-104626252.vg
lvm/archive/lxc1_00035-889249066.vg
lvm/archive/lxc1_00036-1726344104.vg
lvm/archive/lxc1_00037-1211692719.vg
lvm/archive/lxc1_00038-1557917309.vg
lvm/archive/lxc1_00039-1127047191.vg
lvm/archive/lxc1_00040-1131108888.vg
lvm/archive/lxc1_00041-1500497243.vg
lvm/archive/lxc1_00042-551157204.vg
lvm/archive/lxc1_00043-776381470.vg
lvm/archive/lxc1_00044-73417476.vg
lvm/archive/lxc1_00045-719573418.vg
lvm/archive/lxc1_00046-1119133349.vg
lvm/archive/lxc1_00047-1831045492.vg
lvm/archive/lxc1_00048-14742525.vg
lvm/archive/lxc1_00049-1784772140.vg
lvm/archive/lxc1_00050-1671727915.vg
lvm/archive/lxc1_00051-1986768746.vg
lvm/archive/lxc1_00052-90850912.vg
lvm/archive/lxc1_00053-2030773492.vg
lvm/archive/lxc1_00054-605380818.vg
lvm/archive/lxc1_00055-662852896.vg
lvm/archive/lxc1_00056-1563912899.vg
lvm/archive/lxc1_00057-2021188712.vg
lvm/archive/lxc1_00058-2143104283.vg
lvm/archive/lxc1_00059-1037728114.vg
lvm/archive/lxc1_00060-1941522894.vg
lvm/archive/lxc1_00061-1401878871.vg
lvm/archive/lxc1_00062-1502314018.vg
lvm/archive/lxc1_00063-1838288952.vg
lvm/archive/lxc1_00064-1407083719.vg
lvm/archive/lxc1_00065-887677401.vg
lvm/archive/lxc1_00066-1239195323.vg
lvm/archive/lxc1_00067-1012081319.vg
lvm/archive/lxc1_00068-1452178278.vg
lvm/archive/lxc1_00069-385273101.vg
lvm/archive/lxc1_00070-1948198084.vg
lvm/archive/lxc1_00071-126370694...

Read more...

Florian Engelmann (engelmann) wrote :

I tried deleting
/etc/apparmor.d/cache/*

and restarting apparmor

service apparmor restart

but this failes:

root@global04:/# rm -rf /etc/apparmor.d/cache/*
root@global04:/# /etc/init.d/apparmor restart
 * Reloading AppArmor profiles Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Enocoding of mount rule failed
ERROR processing policydb rules for profile lxc-container-default, failed to load
                                                                                                                               [ OK ]
root@global04:/# ls -la /etc/apparmor.d/cache/
total 124
drwxr-xr-x 2 root root 4096 Mar 24 13:30 .
drwxr-xr-x 9 root root 4096 Mar 21 12:27 ..
-rw-r--r-- 1 root root 1095 Mar 24 13:30 .features
-rw------- 1 root root 0 Mar 24 13:30 lxc-containers-uOYqHQ
-rw------- 1 root root 58275 Mar 24 13:30 sbin.dhclient
-rw------- 1 root root 6033 Mar 24 13:30 usr.bin.lxc-start
-rw------- 1 root root 41193 Mar 24 13:30 usr.sbin.tcpdump

aa-status:

apparmor module is loaded.
5 profiles are loaded.
5 profiles are in enforce mode.
   /sbin/dhclient
   /usr/bin/lxc-start
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/connman/scripts/dhclient-script
   /usr/sbin/tcpdump
0 profiles are in complain mode.
5 processes have profiles defined.
5 processes are in enforce mode.
   /usr/bin/lxc-start (8341)
   /usr/bin/lxc-start (8363)
   /usr/bin/lxc-start (8527)
   /usr/bin/lxc-start (8814)
   /usr/bin/lxc-start (8900)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

Florian Engelmann (engelmann) wrote :

The only thing I can remember global04 and global01 are different from global0{2,3,5,6} is the kernel "history" I patched global01 and global04 more often than the others. Might this have caused the problem?

Casey Marshall (cmars) wrote :

I'm getting the same "Enocoding of mount rule failed" when I restart apparmor. I'm running Xubuntu 14.04 on my laptop, doing a dist-upgrade at least daily. I was using LXC user-space containers to isolate non-free apps, and it was working quite well until a few days ago. Is there anything I can try or collect that might help?

Changed in apparmor (Ubuntu):
assignee: nobody → Tyler Hicks (tyhicks)
Florian Engelmann (engelmann) wrote :

I also used meld to check the /etc directory of global02 vs. global04 and only those files like ssh host key, hostname, postfix/main.cf, network/interfaces ... are different. The rest seems to be the same.

Should I check /usr or /var also? does that make any sense?

Looking at the constructor mnt_rule::mnt_rule() in apparmor-2.8.95~2430/parser/mount.c:394, I cannot find where flags and inv_flags are initialized. So depending on the heap - in particular when not zero-initialized anymore - apparmor loading fails randomly.

John Johansen (jjohansen) wrote :

This is fixed in apparmor-2.8.95~2430

the initialization happens as part of the constructor in mount.c

mnt_rule::mnt_rule(struct cond_entry *src_conds, char *device_p,
     struct cond_entry *dst_conds __unused, char *mnt_point_p,
     int allow_p):
 mnt_point(mnt_point_p), device(device_p), trans(NULL), opts(NULL),
 flags(0), inv_flags(0), audit(0), deny(0)
                 ^ ^
             initialization is here

Casey Marshall (cmars) wrote :

This fix resolved the issue for me. Thanks!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers