OpenStack Neutron Gateway Charm

neutron-lbaasv2-agent.log flooded with Error while connecting to stats socket: [Errno 13] EACCES

Bug #1718768 reported by Nobuto Murata
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Neutron Gateway Charm
Fix Released
Medium
James Page
OpenStack Neutron Gateway Charm 17.11

Bug Description

OpenStack origin: cloud:xenial-ocata
Charm release: 17.08

When using LBaaS v2, I got 503 from the load balancer.

<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

When I manually connect to the backing servers, those returned 200, so status monitoring should work. However, /var/log/neutron/neutron-lbaasv2-agent.log is flooded with socket connection error and apparmor denied messages can be also observed.

[/var/log/neutron/neutron-lbaasv2-agent.log]
2017-09-21 19:44:44.850 736613 WARNING neutron_lbaas.drivers.haproxy.namespace_driver [-] Error while connecting to stats socket: [Errno 13] EACCES
2017-09-21 19:44:54.853 736613 WARNING neutron_lbaas.drivers.haproxy.namespace_driver [-] Error while connecting to stats socket: [Errno 13] EACCES
2017-09-21 19:45:04.851 736613 WARNING neutron_lbaas.drivers.haproxy.namespace_driver [-] Error while connecting to stats socket: [Errno 13] EACCES
2017-09-21 19:45:14.855 736613 WARNING neutron_lbaas.drivers.haproxy.namespace_driver [-] Error while connecting to stats socket: [Errno 13] EACCES
2017-09-21 19:45:24.860 736613 WARNING neutron_lbaas.drivers.haproxy.namespace_driver [-] Error while connecting to stats socket: [Errno 13] EACCES

[kernel log]
Sep 21 19:46:44 HOSTNAME kernel: audit: type=1400 audit(1506023204.857:304): apparmor="DENIED" operation="connect" info="Failed name lookup - disconnected path" error=-13 profile="/usr/bin/neutron-lbaasv2-agent" name="var/lib/neutron/lbaas/v2/496d6d2b-8bf7-42b7-822f-c3f31d8db43f/haproxy_stats.sock" pid=736613 comm="neutron-lbaasv2" requested_mask="wr" denied_mask="wr" fsuid=115 ouid=0
Sep 21 19:46:54 HOSTNAME kernel: audit: type=1400 audit(1506023214.857:305): apparmor="DENIED" operation="connect" info="Failed name lookup - disconnected path" error=-13 profile="/usr/bin/neutron-lbaasv2-agent" name="var/lib/neutron/lbaas/v2/496d6d2b-8bf7-42b7-822f-c3f31d8db43f/haproxy_stats.sock" pid=736613 comm="neutron-lbaasv2" requested_mask="wr" denied_mask="wr" fsuid=115 ouid=0
Sep 21 19:47:04 HOSTNAME kernel: audit: type=1400 audit(1506023224.857:306): apparmor="DENIED" operation="connect" info="Failed name lookup - disconnected path" error=-13 profile="/usr/bin/neutron-lbaasv2-agent" name="var/lib/neutron/lbaas/v2/496d6d2b-8bf7-42b7-822f-c3f31d8db43f/haproxy_stats.sock" pid=736613 comm="neutron-lbaasv2" requested_mask="wr" denied_mask="wr" fsuid=115 ouid=0
Sep 21 19:47:14 HOSTNAME kernel: audit: type=1400 audit(1506023234.861:307): apparmor="DENIED" operation="connect" info="Failed name lookup - disconnected path" error=-13 profile="/usr/bin/neutron-lbaasv2-agent" name="var/lib/neutron/lbaas/v2/496d6d2b-8bf7-42b7-822f-c3f31d8db43f/haproxy_stats.sock" pid=736613 comm="neutron-lbaasv2" requested_mask="wr" denied_mask="wr" fsuid=115 ouid=0
Sep 21 19:47:24 HOSTNAME kernel: audit: type=1400 audit(1506023244.865:308): apparmor="DENIED" operation="connect" info="Failed name lookup - disconnected path" error=-13 profile="/usr/bin/neutron-lbaasv2-agent" name="var/lib/neutron/lbaas/v2/496d6d2b-8bf7-42b7-822f-c3f31d8db43f/haproxy_stats.sock" pid=736613 comm="neutron-lbaasv2" requested_mask="wr" denied_mask="wr" fsuid=115 ouid=0

It may be an apparmor policy issue, but starting from filing a bug to charm-neutron-gateway since it's charm-deployed environment.

Revision history for this message
Nobuto Murata (nobuto) wrote :

Hmm, it looks like /etc/apparmor.d/usr.bin.neutron-lbaasv2-agent comes from the charm, but I didn't enable aa-profile-mode at all, but the profile is activated?

$ juju config neutron-gateway aa-profile-mode
disable

Revision history for this message
Nobuto Murata (nobuto) wrote :

According to the unit log, the charm wrote 7 templates, but disabled 6 profiles?

unit-neutron-gateway-2: 19:38:42 INFO unit.neutron-gateway/2.juju-log Wrote template /etc/apparmor.d/usr.bin.neutron-dhcp-agent.
unit-neutron-gateway-2: 19:38:43 INFO unit.neutron-gateway/2.juju-log Wrote template /etc/apparmor.d/usr.bin.neutron-openvswitch-agent.
unit-neutron-gateway-2: 19:38:45 INFO unit.neutron-gateway/2.juju-log Wrote template /etc/apparmor.d/usr.bin.neutron-lbaasv2-agent.
unit-neutron-gateway-2: 19:38:45 INFO unit.neutron-gateway/2.juju-log Wrote template /etc/apparmor.d/usr.bin.neutron-metering-agent.
unit-neutron-gateway-2: 19:38:45 INFO unit.neutron-gateway/2.juju-log Wrote template /etc/apparmor.d/usr.bin.neutron-l3-agent.
unit-neutron-gateway-2: 19:38:45 INFO unit.neutron-gateway/2.juju-log Wrote template /etc/apparmor.d/usr.bin.neutron-metadata-agent.
unit-neutron-gateway-2: 19:38:45 INFO unit.neutron-gateway/2.juju-log Wrote template /etc/apparmor.d/usr.bin.nova-api-metadata.
unit-neutron-gateway-2: 19:38:50 DEBUG unit.neutron-gateway/2.config-changed Disabling /etc/apparmor.d/usr.bin.neutron-dhcp-agent.
unit-neutron-gateway-2: 19:38:52 DEBUG unit.neutron-gateway/2.config-changed Disabling /etc/apparmor.d/usr.bin.neutron-l3-agent.
unit-neutron-gateway-2: 19:38:54 DEBUG unit.neutron-gateway/2.config-changed Disabling /etc/apparmor.d/usr.bin.neutron-metadata-agent.
unit-neutron-gateway-2: 19:38:56 DEBUG unit.neutron-gateway/2.config-changed Disabling /etc/apparmor.d/usr.bin.neutron-metering-agent.
unit-neutron-gateway-2: 19:38:57 DEBUG unit.neutron-gateway/2.config-changed Disabling /etc/apparmor.d/usr.bin.nova-api-metadata.
unit-neutron-gateway-2: 19:38:59 DEBUG unit.neutron-gateway/2.config-changed Disabling /etc/apparmor.d/usr.bin.neutron-openvswitch-agent.

Revision history for this message
James Page (james-page) wrote :

OK so on install, the apparmor profile does not get enabled; however after a reboot of the unit:

$ sudo aa-status
apparmor module is loaded.
14 profiles are loaded.
14 profiles are in enforce mode.
   /sbin/dhclient
   /usr/bin/lxc-start
   /usr/bin/neutron-lbaasv2-agent
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/lxd/lxd-bridge-proxy
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/sbin/tcpdump
   lxc-container-default
   lxc-container-default-cgns
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting
0 profiles are in complain mode.
3 processes have profiles defined.
3 processes are in enforce mode.
   /sbin/dhclient (872)
   /usr/bin/neutron-lbaasv2-agent (1112)
   /usr/lib/lxd/lxd-bridge-proxy (1625)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

You can see that the profile is enabled; the code that manages the profile status currently does not deal with the switch to lbaasv2.

Changed in charm-neutron-gateway:
status: New → Confirmed
importance: Undecided → Medium
milestone: none → 17.11
status: Confirmed → Triaged
James Page (james-page)
Changed in charm-neutron-gateway:
assignee: nobody → James Page (james-page)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-neutron-gateway (master)

Fix proposed to branch: master
Review: https://review.openstack.org/506368

Revision history for this message
James Page (james-page) wrote :

Nobuto - cs:~james-page/neutron-gateway-3 has the proposed fix if you want to test it out.

We'll also need to update the profile to actually work - but for the time being it should get correctly disabled on default install.

Revision history for this message
Nobuto Murata (nobuto) wrote :

@James,

Brilliant, thanks!

tags: added: canonical-bootstack cpe-onsite
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-gateway (master)

Reviewed: https://review.openstack.org/506368
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=d7ccd2bfba17a8cc000eb588804acd80a5d62723
Submitter: Jenkins
Branch: master

commit d7ccd2bfba17a8cc000eb588804acd80a5d62723
Author: James Page <email address hidden>
Date: Thu Sep 21 21:53:51 2017 +0100

    apparmor: manage lbaasv2 profile >= newton

    Ensure that the LBaaS v2 profile is managed for OpenStack Newton
    or later, in preference to the removed LBaaS v1 profile.

    Change-Id: I2510e55a1bb14ee5771c0991d8257faa321b7621
    Closes-Bug: 1718768

Changed in charm-neutron-gateway:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-neutron-gateway (stable/17.08)

Fix proposed to branch: stable/17.08
Review: https://review.openstack.org/510345

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-gateway (stable/17.08)

Reviewed: https://review.openstack.org/510345
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=036634cfa9efe506ed19e7948479defc8350d4fb
Submitter: Jenkins
Branch: stable/17.08

commit 036634cfa9efe506ed19e7948479defc8350d4fb
Author: James Page <email address hidden>
Date: Thu Sep 21 21:53:51 2017 +0100

    apparmor: manage lbaasv2 profile >= newton

    Ensure that the LBaaS v2 profile is managed for OpenStack Newton
    or later, in preference to the removed LBaaS v1 profile.

    Change-Id: I2510e55a1bb14ee5771c0991d8257faa321b7621
    Closes-Bug: 1718768
    (cherry picked from commit d7ccd2bfba17a8cc000eb588804acd80a5d62723)

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

Sorry, need to switch back to fix-committed for 17.11 and add fix-released for 17.08.

Changed in charm-neutron-gateway:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.