| 2014-01-21 22:13:43 |
Robert Collins |
bug |
|
|
added bug |
| 2014-01-21 22:14:05 |
Robert Collins |
bug task added |
|
tripleo |
|
| 2014-01-21 22:14:15 |
Robert Collins |
tripleo: importance |
Undecided |
Critical |
|
| 2014-01-21 22:14:18 |
Robert Collins |
tripleo: status |
New |
Triaged |
|
| 2014-01-25 18:57:03 |
Robert Collins |
description |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped updating DHCP entries - new VMs don't get IP addresses until neutron-dhcp-agent is restarted.
Haven't seen anything obvious in the logs yet, happy to set specific log levels or whatever to try and debug this. |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted.
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem. |
|
| 2014-01-25 18:59:36 |
Robert Collins |
description |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted.
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem. |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem. |
|
| 2014-01-29 03:21:01 |
Robert Collins |
description |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem. |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem.
kill -HUPing the dnsmasqs doesn't seem to fix things |
|
| 2014-01-29 03:39:01 |
Robert Collins |
summary |
neutron-dhcp-agent not getting updates after ~24h running |
neutron-dhcp-agent doesn't hand out leases for recently used addresses |
|
| 2014-01-29 03:42:03 |
Robert Collins |
description |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem.
kill -HUPing the dnsmasqs doesn't seem to fix things |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem.
kill -HUPing the dnsmasqs doesn't seem to fix things
-> The issue is that a prior VM gets (say ) 192.168.1.25, then is shutdown.
-> we then issue 1.25 to a new VM, but the dnsmasq process still has 1.25 in its in-memory leases table as being for the other adress.
# POSSIBLE SOLUTIONS
- change dnsmasq to discard leases that were statically allocated and no longer have that MAC address (e.g. to trust the config more)
- change neutron to only recycle port addresses when the discarded MAC has been deleted more than (lease time). |
|
| 2014-01-29 03:43:54 |
Robert Collins |
description |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem.
kill -HUPing the dnsmasqs doesn't seem to fix things
-> The issue is that a prior VM gets (say ) 192.168.1.25, then is shutdown.
-> we then issue 1.25 to a new VM, but the dnsmasq process still has 1.25 in its in-memory leases table as being for the other adress.
# POSSIBLE SOLUTIONS
- change dnsmasq to discard leases that were statically allocated and no longer have that MAC address (e.g. to trust the config more)
- change neutron to only recycle port addresses when the discarded MAC has been deleted more than (lease time). |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem.
kill -HUPing the dnsmasqs doesn't seem to fix things
-> The issue is that a prior VM gets (say ) 192.168.1.25, then is shutdown.
-> we then issue 1.25 to a new VM, but the dnsmasq process still has 1.25 in its in-memory leases table as being for the other adress.
# POSSIBLE SOLUTIONS
- change dnsmasq to discard leases that were statically allocated and no longer have that MAC address (e.g. to trust the config more)
- change neutron to only recycle port addresses when the discarded MAC has been deleted more than (lease time).
- use dnsmasqs multiple-MAC feature to tell dnsmasq that a handover is happening.[1]
[1] " As a special case, in DHCPv4, it is possible to include more than one hardware address. eg:
--dhcp-host=11:22:33:44:55:66,12:34:56:78:90:12,192.168.0.2 This allows an IP address to be associated with multiple hardware addresses, and gives dnsmasq permission to abandon a DHCP lease to one of the hardware addresses when another one asks for a lease. Beware that this is a dangerous thing to do, it will only work reliably if only one of the hardware addresses is active at any time and there is no way for dnsmasq to enforce this. It is, for instance, useful to allocate a stable IP address to a laptop which has both wired and wireless interfaces.
" |
|
| 2014-01-29 03:47:54 |
Robert Collins |
description |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem.
kill -HUPing the dnsmasqs doesn't seem to fix things
-> The issue is that a prior VM gets (say ) 192.168.1.25, then is shutdown.
-> we then issue 1.25 to a new VM, but the dnsmasq process still has 1.25 in its in-memory leases table as being for the other adress.
# POSSIBLE SOLUTIONS
- change dnsmasq to discard leases that were statically allocated and no longer have that MAC address (e.g. to trust the config more)
- change neutron to only recycle port addresses when the discarded MAC has been deleted more than (lease time).
- use dnsmasqs multiple-MAC feature to tell dnsmasq that a handover is happening.[1]
[1] " As a special case, in DHCPv4, it is possible to include more than one hardware address. eg:
--dhcp-host=11:22:33:44:55:66,12:34:56:78:90:12,192.168.0.2 This allows an IP address to be associated with multiple hardware addresses, and gives dnsmasq permission to abandon a DHCP lease to one of the hardware addresses when another one asks for a lease. Beware that this is a dangerous thing to do, it will only work reliably if only one of the hardware addresses is active at any time and there is no way for dnsmasq to enforce this. It is, for instance, useful to allocate a stable IP address to a laptop which has both wired and wireless interfaces.
" |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem.
kill -HUPing the dnsmasqs doesn't seem to fix things
-> The issue is that a prior VM gets (say ) 192.168.1.25, then is shutdown.
-> we then issue 1.25 to a new VM, but the dnsmasq process still has 1.25 in its in-memory leases table as being for the other adress.
# POSSIBLE SOLUTIONS
- change dnsmasq to discard leases that were statically allocated and no longer have that MAC address (e.g. to trust the config more)
- change neutron to only recycle port addresses when the discarded MAC has been deleted more than (lease time).
- use dnsmasqs multiple-MAC feature to tell dnsmasq that a handover is happening.[1]
- the lease change script is apparently called for all leases when SIGHUP is received, so maybe we can terminate dead leases there? [2]
[1] " As a special case, in DHCPv4, it is possible to include more than one hardware address. eg:
--dhcp-host=11:22:33:44:55:66,12:34:56:78:90:12,192.168.0.2 This allows an IP address to be associated with multiple hardware addresses, and gives dnsmasq permission to abandon a DHCP lease to one of the hardware addresses when another one asks for a lease. Beware that this is a dangerous thing to do, it will only work reliably if only one of the hardware addresses is active at any time and there is no way for dnsmasq to enforce this. It is, for instance, useful to allocate a stable IP address to a laptop which has both wired and wireless interfaces.
"
[2] "When it receives a SIGHUP, dnsmasq clears its cache and then re-loads /etc/hosts and /etc/ethers and any file given by --dhcp-hostsfile, --dhcp-optsfile or --addn-hosts. The dhcp lease change script is called for all existing DHCP leases." |
|
| 2014-01-29 04:00:14 |
Robert Collins |
description |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem.
kill -HUPing the dnsmasqs doesn't seem to fix things
-> The issue is that a prior VM gets (say ) 192.168.1.25, then is shutdown.
-> we then issue 1.25 to a new VM, but the dnsmasq process still has 1.25 in its in-memory leases table as being for the other adress.
# POSSIBLE SOLUTIONS
- change dnsmasq to discard leases that were statically allocated and no longer have that MAC address (e.g. to trust the config more)
- change neutron to only recycle port addresses when the discarded MAC has been deleted more than (lease time).
- use dnsmasqs multiple-MAC feature to tell dnsmasq that a handover is happening.[1]
- the lease change script is apparently called for all leases when SIGHUP is received, so maybe we can terminate dead leases there? [2]
[1] " As a special case, in DHCPv4, it is possible to include more than one hardware address. eg:
--dhcp-host=11:22:33:44:55:66,12:34:56:78:90:12,192.168.0.2 This allows an IP address to be associated with multiple hardware addresses, and gives dnsmasq permission to abandon a DHCP lease to one of the hardware addresses when another one asks for a lease. Beware that this is a dangerous thing to do, it will only work reliably if only one of the hardware addresses is active at any time and there is no way for dnsmasq to enforce this. It is, for instance, useful to allocate a stable IP address to a laptop which has both wired and wireless interfaces.
"
[2] "When it receives a SIGHUP, dnsmasq clears its cache and then re-loads /etc/hosts and /etc/ethers and any file given by --dhcp-hostsfile, --dhcp-optsfile or --addn-hosts. The dhcp lease change script is called for all existing DHCP leases." |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem.
kill -HUPing the dnsmasqs doesn't seem to fix things
-> The issue is that a prior VM gets (say ) 192.168.1.25, then is shutdown.
-> we then issue 1.25 to a new VM, but the dnsmasq process still has 1.25 in its in-memory leases table as being for the other adress.
# WORKAROUNDS:
- restart the agent in a cronjob
- change reload_allocations to return self.restart(), avoiding the cache in dnsmasq but introducing short interrupts in DHCP on each new VM deployed.
# POSSIBLE SOLUTIONS
- change dnsmasq to discard leases that were statically allocated and no longer have that MAC address (e.g. to trust the config more)
- change neutron to only recycle port addresses when the discarded MAC has been deleted more than (lease time).
- use dnsmasqs multiple-MAC feature to tell dnsmasq that a handover is happening.[1]
- the lease change script is apparently called for all leases when SIGHUP is received, so maybe we can terminate dead leases there? [2]
[1] " As a special case, in DHCPv4, it is possible to include more than one hardware address. eg:
--dhcp-host=11:22:33:44:55:66,12:34:56:78:90:12,192.168.0.2 This allows an IP address to be associated with multiple hardware addresses, and gives dnsmasq permission to abandon a DHCP lease to one of the hardware addresses when another one asks for a lease. Beware that this is a dangerous thing to do, it will only work reliably if only one of the hardware addresses is active at any time and there is no way for dnsmasq to enforce this. It is, for instance, useful to allocate a stable IP address to a laptop which has both wired and wireless interfaces.
"
[2] "When it receives a SIGHUP, dnsmasq clears its cache and then re-loads /etc/hosts and /etc/ethers and any file given by --dhcp-hostsfile, --dhcp-optsfile or --addn-hosts. The dhcp lease change script is called for all existing DHCP leases." |
|
| 2014-01-29 08:30:06 |
Salvatore Orlando |
neutron: assignee |
|
Aaron Rosen (arosen) |
|
| 2014-01-29 08:47:44 |
Robert Collins |
description |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem.
kill -HUPing the dnsmasqs doesn't seem to fix things
-> The issue is that a prior VM gets (say ) 192.168.1.25, then is shutdown.
-> we then issue 1.25 to a new VM, but the dnsmasq process still has 1.25 in its in-memory leases table as being for the other adress.
# WORKAROUNDS:
- restart the agent in a cronjob
- change reload_allocations to return self.restart(), avoiding the cache in dnsmasq but introducing short interrupts in DHCP on each new VM deployed.
# POSSIBLE SOLUTIONS
- change dnsmasq to discard leases that were statically allocated and no longer have that MAC address (e.g. to trust the config more)
- change neutron to only recycle port addresses when the discarded MAC has been deleted more than (lease time).
- use dnsmasqs multiple-MAC feature to tell dnsmasq that a handover is happening.[1]
- the lease change script is apparently called for all leases when SIGHUP is received, so maybe we can terminate dead leases there? [2]
[1] " As a special case, in DHCPv4, it is possible to include more than one hardware address. eg:
--dhcp-host=11:22:33:44:55:66,12:34:56:78:90:12,192.168.0.2 This allows an IP address to be associated with multiple hardware addresses, and gives dnsmasq permission to abandon a DHCP lease to one of the hardware addresses when another one asks for a lease. Beware that this is a dangerous thing to do, it will only work reliably if only one of the hardware addresses is active at any time and there is no way for dnsmasq to enforce this. It is, for instance, useful to allocate a stable IP address to a laptop which has both wired and wireless interfaces.
"
[2] "When it receives a SIGHUP, dnsmasq clears its cache and then re-loads /etc/hosts and /etc/ethers and any file given by --dhcp-hostsfile, --dhcp-optsfile or --addn-hosts. The dhcp lease change script is called for all existing DHCP leases." |
Hi, last two days on ci-overcloud.tripleo.org, the neutron-dhcp-agent has stopped answering properly for new VMs - new VMs don't get IP addresses until neutron-dhcp-agent is restarted (we see cloud-init waiting in nova console-log)
There is a bunch of rabbit disconnect messages in the log, but they are not correlated with the fault.
Existing VMs can DHCP successfully, and the overlay network seems fine.
I've checked the dhcp agent host file and it is being updated.
restarting the neutron-dhcp-agent 'fixes' the problem.
kill -HUPing the dnsmasqs doesn't seem to fix things
-> The issue is that a prior VM gets (say ) 192.168.1.25, then is shutdown.
-> we then issue 1.25 to a new VM, but the dnsmasq process still has 1.25 in its in-memory leases table as being for the other adress.
- DHCPRELEASE *is* being issued, perhaps sporadic failures or some other condition/
# WORKAROUNDS:
- restart the agent in a cronjob
- change reload_allocations to return self.restart(), avoiding the cache in dnsmasq but introducing short interrupts in DHCP on each new VM deployed.
# POSSIBLE SOLUTIONS
- change dnsmasq to discard leases that were statically allocated and no longer have that MAC address (e.g. to trust the config more)
- change neutron to only recycle port addresses when the discarded MAC has been deleted more than (lease time).
- use dnsmasqs multiple-MAC feature to tell dnsmasq that a handover is happening.[1]
- the lease change script is apparently called for all leases when SIGHUP is received, so maybe we can terminate dead leases there? [2]
[1] " As a special case, in DHCPv4, it is possible to include more than one hardware address. eg:
--dhcp-host=11:22:33:44:55:66,12:34:56:78:90:12,192.168.0.2 This allows an IP address to be associated with multiple hardware addresses, and gives dnsmasq permission to abandon a DHCP lease to one of the hardware addresses when another one asks for a lease. Beware that this is a dangerous thing to do, it will only work reliably if only one of the hardware addresses is active at any time and there is no way for dnsmasq to enforce this. It is, for instance, useful to allocate a stable IP address to a laptop which has both wired and wireless interfaces.
"
[2] "When it receives a SIGHUP, dnsmasq clears its cache and then re-loads /etc/hosts and /etc/ethers and any file given by --dhcp-hostsfile, --dhcp-optsfile or --addn-hosts. The dhcp lease change script is called for all existing DHCP leases." |
|
| 2014-01-29 16:04:53 |
Ashok kumaran B |
bug |
|
|
added subscriber Ashok kumaran B |
| 2014-03-13 11:53:27 |
stephen mulcahy |
bug |
|
|
added subscriber stephen mulcahy |
| 2014-03-25 19:34:23 |
Robert Collins |
tripleo: importance |
Critical |
High |
|
| 2014-03-25 19:37:12 |
Aaron Rosen |
neutron: importance |
Undecided |
High |
|
| 2014-03-26 02:05:39 |
Sudheendra Murthy |
bug |
|
|
added subscriber Sudheendra Murthy |
| 2014-03-27 00:53:25 |
Duncan Idaho |
bug |
|
|
added subscriber Duncan Idaho |
| 2014-04-01 06:46:08 |
Haw Loeung |
bug |
|
|
added subscriber Haw Loeung |
| 2014-04-01 07:37:36 |
Haw Loeung |
bug |
|
|
added subscriber The Canonical Sysadmins |
| 2014-04-07 11:09:18 |
Salvatore Orlando |
neutron: milestone |
|
juno-1 |
|
| 2014-04-25 21:11:47 |
KaZeR |
bug |
|
|
added subscriber KaZeR |
| 2014-04-29 15:04:59 |
Kyle Mestery |
neutron: status |
New |
Confirmed |
|
| 2014-06-11 04:24:38 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto MURATA |
| 2014-06-12 14:26:50 |
Kyle Mestery |
neutron: milestone |
juno-1 |
juno-2 |
|
| 2014-06-23 09:46:33 |
Sudhakar Gariganti |
bug |
|
|
added subscriber Sudhakar Gariganti |
| 2014-07-22 13:05:13 |
Kyle Mestery |
neutron: status |
Confirmed |
Incomplete |
|
| 2014-07-22 13:06:02 |
Kyle Mestery |
neutron: milestone |
juno-2 |
|
|
| 2014-10-29 13:24:12 |
Aaron Rosen |
neutron: status |
Incomplete |
Invalid |
|
| 2014-10-29 13:24:20 |
Aaron Rosen |
neutron: status |
Invalid |
Fix Released |
|
| 2014-10-29 13:24:22 |
Aaron Rosen |
tripleo: status |
Triaged |
Fix Released |
|
| 2014-12-01 07:10:29 |
gustavo panizzo |
bug |
|
|
added subscriber gustavo panizzo |
