unable to reload allocations neutron-dhcp-agent

Bug #1407618 reported by Miguel Angel Ajo
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Miguel Angel Ajo

Bug Description

In some cases dhcp agent dumps an exception because it's trying to use replace_file over a directory
which was never created.

http://logstash.openstack.org/#eyJzZWFyY2giOiJcIlVuYWJsZSB0byByZWxvYWRfYWxsb2NhdGlvbnMgZGhjcCBmb3JcIiIsImZpZWxkcyI6WyJsb2dfdXJsIl0sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyMDQ1MDU0OTY2M30=

[req-84e8f66a-5401-424c-82d5-068267d81c7b None] Unable to reload_allocations dhcp for c2a866cf-a1db-4e50-9fc8-5da8b7320669.
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent Traceback (most recent call last):
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent File "/opt/stack/new/neutron/neutron/agent/dhcp_agent.py", line 129, in call_driver
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent getattr(driver, action)(**action_kwargs)
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 445, in reload_allocations
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent self._output_hosts_file()
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent File "/opt/stack/new/neutron/neutron/agent/linux/dhcp.py", line 525, in _output_hosts_file
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent utils.replace_file(filename, buf.getvalue())
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 112, in replace_file
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent tmp_file = tempfile.NamedTemporaryFile('w+', dir=base_dir, delete=False)
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.7/tempfile.py", line 462, in NamedTemporaryFile
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.7/tempfile.py", line 239, in _mkstemp_inner
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent fd = _os.open(file, flags, 0600)
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent File "/usr/local/lib/python2.7/dist-packages/eventlet/green/os.py", line 109, in open
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent fd = __original_open__(file, flags, mode)
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent OSError: [Errno 2] No such file or directory: '/opt/stack/data/neutron/dhcp/c2a866cf-a1db-4e50-9fc8-5da8b7320669/tmpJnpUfM'
2015-01-05 09:06:38.554 2489 TRACE neutron.agent.dhcp_agent

Proper directory creation must always happen before any attempt to "replace_file"

Changed in neutron:
assignee: nobody → Miguel Angel Ajo (mangelajo)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/144977

Changed in neutron:
status: New → In Progress
Changed in neutron:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/144977
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=057e540087150d57a2e628b5f08dcdaa8b43a307
Submitter: Jenkins
Branch: master

commit 057e540087150d57a2e628b5f08dcdaa8b43a307
Author: Miguel Angel Ajo <email address hidden>
Date: Mon Jan 5 12:34:53 2015 +0000

    Ensure config directory created before updating leases

    Under high load conditions dhcp-agent could try to start the
    dhcp local process via reload_allocations. But it will fail since
    the dhcp config directory for the specific network is not
    created yet.

    We ensure its creation with this patch.

    Closes-Bug: 1407618

    Change-Id: Ib37651f7f802debd472ab292b148c2a2496063a3

Changed in neutron:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in neutron:
milestone: none → kilo-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: kilo-2 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.