DHCP: no indication in API that DHCP service is not running

Bug #1544548 reported by Gary Kotton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Gary Kotton

Bug Description

Even if DHCP namespace creation fails at the network node due to some reason, neutron API still returns success to the user.

2016-01-18 02:51:12.661 ^[[00;32mDEBUG neutron.agent.dhcp.agent [^[[01;36mreq-f6d7a436-b9ff-45ca-9cfc-0f147b97effb ^[[00;36mctx_rally_bbaa10b4eb2749b3a09b375682b6cb6e_user_0 bbaa10b4eb2749b3a09b375682b6cb6e^[[00;32m] ^[[01;35m^[[00;32mCalling driver for network: 351d9017-6e92-4310-ae6d-cf1d0bce0b14 action: enable^[[00m ^[[00;33mfrom (pid=26547) call_driver /opt/stack/neutron/neutron/agent/dhcp/agent.py:104

2016-01-18 02:51:12.662 ^[[00;32mDEBUG neutron.agent.linux.dhcp [^[[01;36mreq-f6d7a436-b9ff-45ca-9cfc-0f147b97effb ^[[00;36mctx_rally_bbaa10b4eb2749b3a09b375682b6cb6e_user_0 bbaa10b4eb2749b3a09b375682b6cb6e^[[00;32m] ^[[01;35m^[[00;32mDHCP port dhcpa382383f-19b6-5ca7-94ec-5ec1e62dc705-351d9017-6e92-4310-ae6d-cf1d0bce0b14 on network 351d9017-6e92-4310-ae6d-cf1d0bce0b14 does not yet exist. Checking for a reserved port.^[[00m ^[[00;33mfrom (pid=26547) _setup_reserved_dhcp_port /opt/stack/neutron/neutron/agent/linux/dhcp.py:1098^[[00m
2016-01-18 02:51:12.663 ^[[00;32mDEBUG neutron.agent.linux.dhcp [^[[01;36mreq-f6d7a436-b9ff-45ca-9cfc-0f147b97effb ^[[00;36mctx_rally_bbaa10b4eb2749b3a09b375682b6cb6e_user_0 bbaa10b4eb2749b3a09b375682b6cb6e^[[00;32m] ^[[01;35m^[[00;32mDHCP port dhcpa382383f-19b6-5ca7-94ec-5ec1e62dc705-351d9017-6e92-4310-ae6d-cf1d0bce0b14 on network 351d9017-6e92-4310-ae6d-cf1d0bce0b14 does not yet exist. Creating new one.^[[00m ^[[00;33mfrom (pid=26547) _setup_new_dhcp_port /opt/stack/neutron/neutron/agent/linux/dhcp.py:1119

2016-01-18 02:51:13.000 ^[[01;31mERROR neutron.agent.dhcp.agent [^[[01;36mreq-f6d7a436-b9ff-45ca-9cfc-0f147b97effb ^[[00;36mctx_rally_bbaa10b4eb2749b3a09b375682b6cb6e_user_0 bbaa10b4eb2749b3a09b375682b6cb6e^[[01;31m] ^[[01;35m^[[01;31mUnable to enable dhcp for 351d9017-6e92-4310-ae6d-cf1d0bce0b14.^[[00m
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00mTraceback (most recent call last):
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/dhcp/agent.py", line 113, in call_driver
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m getattr(driver, action)(**action_kwargs)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 206, in enable
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m interface_name = self.device_manager.setup(self.network)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 1206, in setup
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m namespace=network.namespace)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 243, in plug
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m bridge, namespace, prefix)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 311, in plug_new
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m self.check_bridge_exists(bridge)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/interface.py", line 220, in check_bridge_exists
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m if not ip_lib.device_exists(bridge):
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 908, in device_exists
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m return IPDevice(device_name, namespace=namespace).exists()
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 265, in exists
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m return bool(self.link.address)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 482, in address
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m return self.attributes.get('link/ether')
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 506, in attributes
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m return self._parse_line(self._run(['o'], ('show', self.name)))
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 315, in _run
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m return self._parent._run(options, self.COMMAND, args)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 87, in _run
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m log_fail_as_error=self.log_fail_as_error)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 103, in _execute
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m log_fail_as_error=log_fail_as_error)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 119, in execute
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m addl_env=addl_env)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 88, in create_process
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m stderr=subprocess.PIPE)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/opt/stack/neutron/neutron/common/utils.py", line 206, in subprocess_popen
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m close_fds=close_fds, env=env)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/eventlet/green/subprocess.py", line 53, in __init__
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m errread, errwrite)
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m File "/usr/lib/python2.7/subprocess.py", line 1231, in _execute_child
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m self.pid = os.fork()
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00mOSError: [Errno 12] Cannot allocate memory
^[[01;31m2016-01-18 02:51:13.000 TRACE neutron.agent.dhcp.agent ^[[01;35m^[[00m
2016-01-18 02:51:13.016 ^[[00;32mDEBUG oslo_concurrency.lockutils [^[[01;36mreq-f6d7a436-b9ff-45ca-9cfc-0f147b97effb ^[[00;36mctx_rally_bbaa10b4eb2749b3a09b375682b6cb6e_user_0 bbaa10b4eb2749b3a09b375682b6cb6e^[[00;32m] ^[[01;35m^[[00;32mLock "dhcp-agent" released by "neutron.agent.dhcp.agent.network_create_end" :: held 0.521s^[[00m ^[[00;33mfrom (pid=26547) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282

Gary Kotton (garyk)
Changed in neutron:
assignee: nobody → Gary Kotton (garyk)
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/279081

Changed in neutron:
status: New → In Progress
Changed in neutron:
milestone: none → mitaka-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/279081
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=80cfec66259e05e6914abf0ee443b7c280de21a9
Submitter: Jenkins
Branch: master

commit 80cfec66259e05e6914abf0ee443b7c280de21a9
Author: Gary Kotton <email address hidden>
Date: Thu Feb 11 05:28:45 2016 -0800

    DHCP: release DHCP port if not enough memory

    When the DHCP agent fails to create a namespace for the DHCP
    service we will release the DHCP port instead of failing silently.

    This will at least give the user an indication that there is no DHCP
    service. No DHCP port will exist.

    Change-Id: I59af745d3991e6deb424ecd9b916b03f146c246a
    Closes-bug: #1544548

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/281112

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/liberty)

Reviewed: https://review.openstack.org/281112
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e573352bb18d0b33e91bfbe37ff4e05f6d664459
Submitter: Jenkins
Branch: stable/liberty

commit e573352bb18d0b33e91bfbe37ff4e05f6d664459
Author: Gary Kotton <email address hidden>
Date: Thu Feb 11 05:28:45 2016 -0800

    DHCP: release DHCP port if not enough memory

    When the DHCP agent fails to create a namespace for the DHCP
    service we will release the DHCP port instead of failing silently.

    This will at least give the user an indication that there is no DHCP
    service. No DHCP port will exist.

    Change-Id: I59af745d3991e6deb424ecd9b916b03f146c246a
    Closes-bug: #1544548
    (cherry picked from commit 80cfec66259e05e6914abf0ee443b7c280de21a9)

tags: added: in-stable-liberty
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.