Charm is making sysctl calls in LXD containers (unsupported)

Bug #1829047 reported by Ryan Farrell on 2019-05-14
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack neutron-gateway charm
High
Chris MacNaughton

Bug Description

I am following the guide outlined here: https://docs.openstack.org/charm-guide/latest/openstack-on-lxd.html

Steps
-----
# install deps <not shown>
# get openstack-on-lxd source configs
git clone https://github.com/openstack-charmers/openstack-on-lxd

# Configure LXD on Host
echo fs.inotify.max_queued_events=1048576 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_instances=1048576 | sudo tee -a /etc/sysctl.conf
echo fs.inotify.max_user_watches=1048576 | sudo tee -a /etc/sysctl.conf
echo vm.max_map_count=262144 | sudo tee -a /etc/sysctl.conf
echo vm.swappiness=1 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
sudo lxd init
lxc profile device set default eth0 mtu 9000

# Bootstrap Juju
juju bootstrap --config config.yaml localhost lxd
cat lxd-profile.yaml | lxc profile edit juju-default

# Install Openstack
juju deploy bundle-bionic-queens.yaml
# Get coffee...
-----

Once deployment stabilizes'juju status' will show \
    neutron-gateway/0* error idle 15 10.0.8.251 hook failed: "config-changed"

From Log
--------
1879 2019-05-13 21:52:18 DEBUG juju-log Updating sysctl_file: /etc/sysctl.d/50-quantum-gateway.conf values: {'net.ipv4.neigh.default.gc_thresh1': 128, 'net.ipv4.neigh.default.gc_thresh2': 28672, 'net.ipv 4.neigh.default.gc_thresh3': 32768, 'net.ipv6.neigh.default.gc_thresh1': 128, 'net.ipv6.neigh.default.gc_thresh2': 28672, 'net.ipv6.neigh.default.gc_thresh3': 32768, 'net.nf_conntrack_max': 1000000, 'net.netfilter.nf_conntrack_buckets': 204800, 'net.netfilter.nf_conntrack_max': 1000000}
 1880 2019-05-13 21:52:18 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh1: No such file or directory
 1881 2019-05-13 21:52:19 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh2: No such file or directory
 1882 2019-05-13 21:52:19 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh3: No such file or directory
 1883 2019-05-13 21:52:19 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh1: No such file or directory
 1884 2019-05-13 21:52:19 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh2: No such file or directory
 1885 2019-05-13 21:52:19 DEBUG config-changed sysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh3: No such file or directory
 1886 2019-05-13 21:52:19 DEBUG config-changed sysctl: cannot stat /proc/sys/net/nf_conntrack_max: No such file or directory
 1887 2019-05-13 21:52:19 DEBUG config-changed sysctl: setting key "net.netfilter.nf_conntrack_buckets": No such file or directory
 1888 2019-05-13 21:52:19 DEBUG config-changed net.netfilter.nf_conntrack_max = 1000000
 1889 2019-05-13 21:52:19 DEBUG config-changed Traceback (most recent call last):
 1890 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/config-changed", line 362, in <module>
 1891 2019-05-13 21:52:19 DEBUG config-changed hooks.execute(sys.argv)
 1892 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/core/hookenv.py", line 914, in execute
 1893 2019-05-13 21:52:19 DEBUG config-changed self._hooks[hook_name]()
 1894 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1546, in wrapped_f
 1895 2019-05-13 21:52:19 DEBUG config-changed stopstart, restart_functions)
 1896 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/core/host.py", line 741, in restart_on_change_helper
 1897 2019-05-13 21:52:19 DEBUG config-changed r = lambda_f()
 1898 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1545, in <lambda>
 1899 2019-05-13 21:52:19 DEBUG config-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
 1900 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 93, in _harden_inner2
 1901 2019-05-13 21:52:19 DEBUG config-changed return f(*args, **kwargs)
 1902 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/config-changed", line 128, in config_changed
 1903 2019-05-13 21:52:19 DEBUG config-changed '/etc/sysctl.d/50-quantum-gateway.conf')
 1904 2019-05-13 21:52:19 DEBUG config-changed File "/var/lib/juju/agents/unit-neutron-gateway-0/charm/hooks/charmhelpers/core/sysctl.py", line 65, in create
 1905 2019-05-13 21:52:19 DEBUG config-changed check_call(call)
 1906 2019-05-13 21:52:19 DEBUG config-changed File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
 1907 2019-05-13 21:52:19 DEBUG config-changed raise CalledProcessError(retcode, cmd)
 1908 2019-05-13 21:52:19 DEBUG config-changed subprocess.CalledProcessError: Command '['sysctl', '-p', '/etc/sysctl.d/50-quantum-gateway.conf']' returned non-zero exit status 255.
 1909 2019-05-13 21:52:19 ERROR juju.worker.uniter.operation runhook.go:132 hook "config-changed" failed: exit status 1

Full Logs attached

Ryan Farrell (whereisrysmind) wrote :
David Ames (thedac) on 2019-05-14
Changed in charm-neutron-gateway:
status: New → Triaged
importance: Undecided → High
milestone: none → 19.07
David Ames (thedac) wrote :

TRIAGE:

The sysctl commands fail when neutron-gateway is deployed in an LXC.

We have patterns in other charms to handle this. See hardening. We simply need a try/except for the sysctl -p or we need to gate based detecting containerization.

Jamon Camisso (jamon) wrote :

I ran into this bug when upgrading from 18.11 to 19.04 charms. I juju set sysctl="" and that resolved the issues.

Checking for container=lxc in /proc/1/environ seems straightforward enough. If possible a backport to 19.04 would save (me) some work and I suspect problems for others who haven't upgraded yet.

Changed in charm-neutron-gateway:
assignee: nobody → Chris MacNaughton (chris.macnaughton)

Fix proposed to branch: master
Review: https://review.opendev.org/659722

Changed in charm-neutron-gateway:
status: Triaged → In Progress

Reviewed: https://review.opendev.org/659722
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-gateway/commit/?id=a47f400e1513a1ca37ae9adfd84006c95acc2fb4
Submitter: Zuul
Branch: master

commit a47f400e1513a1ca37ae9adfd84006c95acc2fb4
Author: Chris MacNaughton <email address hidden>
Date: Fri May 17 07:56:31 2019 +0200

    Ensure we only try to set sysctls outside of a container

    THe sysctls are not available inside of a container, so we
    should not set them.

    Change-Id: I93e1052958a8c8fb44c0589caf7cb5d6a6019ab8
    Closes-Bug: #1829047

Changed in charm-neutron-gateway:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers