tripleo

neutron's dockers are unhealthy

Bug #1823527 reported by Nawar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Incomplete
High
Unassigned
tripleo victoria-3 "tripleo victoria"

Bug Description

Description
===========

after installing undercloud all neutron's docker are unhealthy

Steps to reproduce
==================
installing undercloud using
- tripleoclient 11.3.1

Expected result
===============
all docker containers are healthy

Actual result
=============
3 neutron containers are unhealthy:(docker container ps)
ocker.io/tripleomaster/centos-binary-neutron-openvswitch-agent:current-tripleo "dumb-init --singl..." 10 days ago Up 7 minutes (unhealthy) neutron_ovs_agent
6148271e4411 docker.io/tripleomaster/centos-binary-neutron-l3-agent:current-tripleo "dumb-init --singl..." 10 days ago Up 8 minutes (unhealthy) neutron_l3_agent
d30698710596 docker.io/tripleomaster/centos-binary-neutron-dhcp-agent:current-tripleo

Environment
===========

1 undercloud server (centos 7)

Logs & Configs
==============

The only error I found in log files is in (/var/log/containers/neutron/dhcp-agent.log):

Cannot connect to the Docker daemon at unix:///var/lib/openstack/docker.sock. Is the docker daemon running?
+ LIST=
++ printf '%s\n' ''
++ grep -E ':(Exited|Created)'
+ ORPHANTS=
+ '[' -n '' ']'
+ printf '%s\n' ''
+ grep -q 'neutron-dnsmasq-qdhcp-31b5db27-7423-4b04-8a00-c4fa154da8e4$'
+ echo 'Starting a new child container neutron-dnsmasq-qdhcp-31b5db27-7423-4b04-8a00-c4fa154da8e4'
+ docker run --detach -v /var/lib/config-data/puppet-generated/neutron/etc/neutron:/etc/neutron:ro -v /run/netns:/run/netns:shared -v /var/lib/neutron:/var/lib/neutron:z,shared -v /dev/log:/dev/log --net host --pid host --privileged -u root --name neutron-dnsmasq-qdhcp-31b5db27-7423-4b04-8a00-c4fa154da8e4 docker.io/tripleomaster/centos-binary-neutron-dhcp-agent:current-tripleo ip netns exec qdhcp-31b5db27-7423-4b04-8a00-c4fa154da8e4 /usr/sbin/dnsmasq -k --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/host --addn-hosts=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/opts --dhcp-leasefile=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-interfaces --dhcp-range=set:tag0,10.100.0.0,static,255.255.0.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=65536 --conf-file= --domain=hiast.edu.sy
/usr/bin/docker-current: Cannot connect to the Docker daemon at unix:///var/lib/openstack/docker.sock. Is the docker daemon running?.
See '/usr/bin/docker-current run --help'.
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 152, in call_driver
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs)
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 226, in enable
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent self.spawn_process()
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 450, in spawn_process
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent self._spawn_or_reload_process(reload_with_HUP=False)
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 464, in _spawn_or_reload_process
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent pm.enable(reload_cfg=reload_with_HUP)
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 88, in enable
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent run_as_root=self.run_as_root)
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 788, in execute
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent run_as_root=run_as_root)
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 147, in execute
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent returncode=returncode)
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent ProcessExecutionError: Exit code: 125; Stdin: ; Stdout: Starting a new child container neutron-dnsmasq-qdhcp-31b5db27-7423-4b04-8a00-c4fa154da8e4
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent ; Stderr: + export DOCKER_HOST=unix:///var/lib/openstack/docker.sock
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + DOCKER_HOST=unix:///var/lib/openstack/docker.sock
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + ARGS='--no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/host --addn-hosts=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/opts --dhcp-leasefile=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-interfaces --dhcp-range=set:tag0,10.100.0.0,static,255.255.0.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=65536 --conf-file= --domain=hiast.edu.sy'
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent ++ ip netns identify
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + NETNS=qdhcp-31b5db27-7423-4b04-8a00-c4fa154da8e4
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + NAME=neutron-dnsmasq-qdhcp-31b5db27-7423-4b04-8a00-c4fa154da8e4
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + CLI=docker
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + CMD='ip netns exec qdhcp-31b5db27-7423-4b04-8a00-c4fa154da8e4 /usr/sbin/dnsmasq -k'
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent ++ docker ps -a --filter name=neutron-dnsmasq- --format '{{.ID}}:{{.Names}}:{{.Status}}'
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent ++ awk '{print $1}'
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent Cannot connect to the Docker daemon at unix:///var/lib/openstack/docker.sock. Is the docker daemon running?
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + LIST=
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent ++ printf '%s\n' ''
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent ++ grep -E ':(Exited|Created)'
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + ORPHANTS=
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + '[' -n '' ']'
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + printf '%s\n' ''
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + grep -q 'neutron-dnsmasq-qdhcp-31b5db27-7423-4b04-8a00-c4fa154da8e4$'
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + echo 'Starting a new child container neutron-dnsmasq-qdhcp-31b5db27-7423-4b04-8a00-c4fa154da8e4'
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent + docker run --detach -v /var/lib/config-data/puppet-generated/neutron/etc/neutron:/etc/neutron:ro -v /run/netns:/run/netns:shared -v /var/lib/neutron:/var/lib/neutron:z,shared -v /dev/log:/dev/log --net host --pid host --privileged -u root --name neutron-dnsmasq-qdhcp-31b5db27-7423-4b04-8a00-c4fa154da8e4 docker.io/tripleomaster/centos-binary-neutron-dhcp-agent:current-tripleo ip netns exec qdhcp-31b5db27-7423-4b04-8a00-c4fa154da8e4 /usr/sbin/dnsmasq -k --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/host --addn-hosts=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/opts --dhcp-leasefile=/var/lib/neutron/dhcp/31b5db27-7423-4b04-8a00-c4fa154da8e4/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-interfaces --dhcp-range=set:tag0,10.100.0.0,static,255.255.0.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=65536 --conf-file= --domain=hiast.edu.sy
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent /usr/bin/docker-current: Cannot connect to the Docker daemon at unix:///var/lib/openstack/docker.sock. Is the docker daemon running?.
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent See '/usr/bin/docker-current run --help'.
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent
2019-04-07 13:34:57.399 8532 ERROR neutron.agent.dhcp.agent
2019-04-07 13:34:57.400 8532 INFO neutron.agent.dhcp.agent [-] Finished network 31b5db27-7423-4b04-8a00-c4fa154da8e4 dhcp configuration

Nawar (nawarix)
tags: added: dhcp-agent neutron unhealthy
Alex Schultz (alex-schultz)
Changed in tripleo:
milestone: none → train-1
importance: Undecided → High
status: New → Triaged
Alex Schultz (alex-schultz)
Changed in tripleo:
milestone: train-1 → train-2
Alex Schultz (alex-schultz)
Changed in tripleo:
milestone: train-2 → train-3
Revision history for this message
Trevor Vardeman (tvardeman) wrote :

I've been casually looking into this but have a question: Is the requirement to check with python-tripleoclient 11.3.1 a hard requirement for this, because the current version of python-tripleoclient is 12.1.0?

Alex Schultz (alex-schultz)
Changed in tripleo:
milestone: train-3 → ussuri-1
Emilien Macchi (emilienm)
Changed in tripleo:
milestone: ussuri-1 → ussuri-2
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-2 → ussuri-3
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-3 → ussuri-rc3
wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Incomplete
wes hayutin (weshayutin)
Changed in tripleo:
milestone: ussuri-rc3 → victoria-1
Emilien Macchi (emilienm)
Changed in tripleo:
milestone: victoria-1 → victoria-3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.