apparmor profiles are not loaded properly in setup-hosts.yml

[Qin Wang]

In an AIO deployment in ubuntu14.04.
"openstack-ansible setup-hosts.yml" failed at starting up containers.
The error log shows:
stderr: lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 346 To get more details, run the container in foreground mode.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.
fatal: [aio1_heat_engine_container-bc8fdf5a -> None]: FAILED! => {"changed": true, "cmd": ["lxc-start", "--daemon", "--name", "aio1_heat_engine_container-bc8fdf5a", "--logfile", "/var/log/lxc/lxc-aio1_heat_engine_container-bc8fdf5a.log", "--logpriority", "DEBUG"], "delta": "0:00:06.229740", "end": "2016-09-02 16:4
1:24.955376", "failed": true, "rc": 1, "start": "2016-09-02 16:41:18.725636", "stderr": "lxc-start: lxc_start.c: main: 344 The container failed to start.\nlxc-start: lxc_start.c: main: 346 To get more details, run the container in foreground mode.\nlxc-start: lxc_start.c: main: 348 Additional information can be ob
tained by setting the --logfile and --logpriority options.", "stdout": "", "stdout_lines": [], "warnings": []}
cmd:
lxc-start
--daemon
--name
aio1_heat_engine_container-bc8fdf5a
--logfile
/var/log/lxc/lxc-aio1_heat_engine_container-bc8fdf5a.log
--logpriority
DEBUG

The container log shows:
      lxc-start 20160902164119.443 ERROR lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:234 - No such file or directory - failed to change apparmor profile to lxc-openstack
      lxc-start 20160902164119.443 ERROR lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 5)
      lxc-start 20160902164119.443 ERROR lxc_start - start.c:__lxc_start:1353 - failed to spawn 'aio1_heat_engine_container-bc8fdf5a'
      lxc-start 20160902164119.443 INFO lxc_conf - conf.c:run_script_argv:367 - Executing script '/usr/share/lxcfs/lxc.reboot.hook' for container 'aio1_heat_engine_container-bc8fdf5a', config section 'lxc'
      lxc-start 20160902164119.945 INFO lxc_conf - conf.c:run_script_argv:367 - Executing script '/var/lib/lxc/aio1_heat_engine_container-bc8fdf5a/veth-cleanup.sh' for container 'aio1_heat_engine_container-bc8fdf5a', config section 'lxc'
      lxc-start 20160902164119.954 WARN lxc_commands - commands.c:lxc_cmd_rsp_recv:172 - command get_cgroup failed to receive response
      lxc-start 20160902164124.954 ERROR lxc_start_ui - lxc_start.c:main:344 - The container failed to start.
      lxc-start 20160902164124.955 ERROR lxc_start_ui - lxc_start.c:main:346 - To get more details, run the container in foreground mode.
      lxc-start 20160902164124.955 ERROR lxc_start_ui - lxc_start.c:main:348 - Additional information can be obtained by setting the --logfile and --logpriority options.

apparmor_status:
apparmor module is loaded.
45 profiles are loaded.
12 profiles are in enforce mode.
   /sbin/dhclient
   /usr/bin/lxc-start
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/chromium-browser/chromium-browser//browser_java
   /usr/lib/chromium-browser/chromium-browser//browser_openjdk
   /usr/lib/chromium-browser/chromium-browser//sanitized_helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/sbin/tcpdump
   lxc-container-default
   lxc-container-default-cgns
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting

After running /etc/init.d/apparmor reload, apparmor_status shows:
46 profiles are loaded.
13 profiles are in enforce mode.
   /sbin/dhclient
   /usr/bin/lxc-start
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/chromium-browser/chromium-browser//browser_java
   /usr/lib/chromium-browser/chromium-browser//browser_openjdk
   /usr/lib/chromium-browser/chromium-browser//sanitized_helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/sbin/tcpdump
   lxc-container-default
   lxc-container-default-cgns
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting
   lxc-openstack

[Qin Wang (qwang)]

could be related to:
https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1432683

[Qin Wang (qwang)]

The bug in the previous link is unrelated.

The real reason for this issue is that, after the apparmor profile is dropped to the apparmor.d/lxc directory, two handlers are suppose to be notified and one of them load the lxc-openstack profile and the other restart apparmor service. However, when the playbook is interrupted/failed before it executes to the end, and then it is re-executed, the triggering task's state will be okay(instead of changed) so that the two handlers won't be notified. As a result, the lxc-openstack profile is not loaded to apparmor properly.

I think we'd better keep these kinds of scenarios in minds and change some "notify" to "register" if necessary.

[Qin Wang (qwang)]

PS:
The two related handlers:

- name: Load lxc-openstack apparmor profile
  command: apparmor_parser -Kr /etc/apparmor.d/lxc-openstack ----- wrong path

- name: Restart apparmor
  service:
    name: "apparmor"
    state: "restarted"
                         ----- with the second handler, the first handler is not needed. restart the service will reload all available profiles

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/367612

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/367612
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/commit/?id=75d32df1dcf17e65adf6d5738c624b62748de1d5
Submitter: Jenkins
Branch: master

commit 75d32df1dcf17e65adf6d5738c624b62748de1d5
Author: Qin Wang <email address hidden>
Date: Thu Sep 8 19:01:23 2016 +0000

    removed redundant handler and flushed handler right away

    loading lxc-openstack profile into apparmor is done with service reloading,
    so the redundant loading handler of lxc-openstack is removed.
    The reloading handler is flushed right away in case of interrupted execution.

    Change-Id: I7a0e9d886808e0949a0e8301c6a5ea2994c6cd49
    closes-bug: 1620757