| 2015-07-02 21:24:17 |
Edward Hope-Morley |
bug |
|
|
added bug |
| 2015-07-02 21:24:34 |
Edward Hope-Morley |
affects |
nova (Ubuntu) |
neutron (Ubuntu) |
|
| 2015-07-03 08:11:55 |
Louis Bouchard |
nominated for series |
|
Ubuntu Utopic |
|
| 2015-07-03 08:11:55 |
Louis Bouchard |
bug task added |
|
neutron (Ubuntu Utopic) |
|
| 2015-07-03 08:11:55 |
Louis Bouchard |
nominated for series |
|
Ubuntu Vivid |
|
| 2015-07-03 08:11:55 |
Louis Bouchard |
bug task added |
|
neutron (Ubuntu Vivid) |
|
| 2015-07-03 08:11:55 |
Louis Bouchard |
nominated for series |
|
Ubuntu Trusty |
|
| 2015-07-03 08:11:55 |
Louis Bouchard |
bug task added |
|
neutron (Ubuntu Trusty) |
|
| 2015-07-03 08:17:57 |
Edward Hope-Morley |
description |
This issue appears to be a consequence of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572 where we added a 'wait-for-state running' to the nova-compute upstart so as to ensure that neutron-ovs-cleanup has finished before nova-compute starts.
I have started to spot, however, that on some hosts (metal only) there is now a race between the two whereby nova-compute fails to start or boot/reboot with the following in /var/log/upstart/nova-compute.log:
...
libvirt-bin stop/waiting
wait-for-state stop/waiting
neutron-ovs-cleanup start/pre-start, process 3084
start: Job failed to start
If I manually restart nova-compute all is fine. So this looks like a race between nova-compute's wait-for-state and neutron-ovs-cleanup's pre-start -> start/running. |
This issue appears to be a consequence of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572 where we added a 'wait-for-state running' to the nova-compute upstart so as to ensure that neutron-ovs-cleanup has finished before nova-compute starts.
I have started to spot, however, that on some hosts (metal only) there is now a race between the two whereby nova-compute sometimes fails to start or boot/reboot with the following in /var/log/upstart/nova-compute.log:
...
libvirt-bin stop/waiting
wait-for-state stop/waiting
neutron-ovs-cleanup start/pre-start, process 3084
start: Job failed to start
If I manually restart nova-compute all is fine. So this looks like a race between nova-compute's wait-for-state and neutron-ovs-cleanup's pre-start -> start/running. |
|
| 2015-07-03 08:18:19 |
Edward Hope-Morley |
description |
This issue appears to be a consequence of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572 where we added a 'wait-for-state running' to the nova-compute upstart so as to ensure that neutron-ovs-cleanup has finished before nova-compute starts.
I have started to spot, however, that on some hosts (metal only) there is now a race between the two whereby nova-compute sometimes fails to start or boot/reboot with the following in /var/log/upstart/nova-compute.log:
...
libvirt-bin stop/waiting
wait-for-state stop/waiting
neutron-ovs-cleanup start/pre-start, process 3084
start: Job failed to start
If I manually restart nova-compute all is fine. So this looks like a race between nova-compute's wait-for-state and neutron-ovs-cleanup's pre-start -> start/running. |
This issue appears to be a consequence of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572 where we added a 'wait-for-state running' to the nova-compute upstart so as to ensure that neutron-ovs-cleanup has finished before nova-compute starts.
I have started to spot, however, that on some hosts (metal only) there is now a race between the two whereby nova-compute sometimes fails to start on system boot/reboot with the following in /var/log/upstart/nova-compute.log:
...
libvirt-bin stop/waiting
wait-for-state stop/waiting
neutron-ovs-cleanup start/pre-start, process 3084
start: Job failed to start
If I manually restart nova-compute all is fine. So this looks like a race between nova-compute's wait-for-state and neutron-ovs-cleanup's pre-start -> start/running. |
|
| 2015-07-03 08:19:13 |
Edward Hope-Morley |
neutron (Ubuntu Trusty): importance |
Undecided |
High |
|
| 2015-07-03 08:19:14 |
Edward Hope-Morley |
neutron (Ubuntu Utopic): importance |
Undecided |
High |
|
| 2015-07-03 08:19:17 |
Edward Hope-Morley |
neutron (Ubuntu Vivid): importance |
Undecided |
High |
|
| 2015-07-03 08:19:20 |
Edward Hope-Morley |
neutron (Ubuntu): status |
New |
In Progress |
|
| 2015-07-03 08:19:23 |
Edward Hope-Morley |
neutron (Ubuntu): assignee |
|
Edward Hope-Morley (hopem) |
|
| 2015-07-03 11:52:23 |
Edward Hope-Morley |
affects |
neutron (Ubuntu) |
nova (Ubuntu) |
|
| 2015-07-03 18:27:37 |
Edward Hope-Morley |
attachment added |
|
trusty.debdiff https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1471022/+attachment/4423971/+files/trusty.debdiff |
|
| 2015-07-03 18:27:52 |
Edward Hope-Morley |
attachment added |
|
utopic.debdiff https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1471022/+attachment/4423972/+files/utopic.debdiff |
|
| 2015-07-03 18:28:06 |
Edward Hope-Morley |
attachment added |
|
vivid.debdiff https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1471022/+attachment/4423974/+files/vivid.debdiff |
|
| 2015-07-03 18:42:21 |
Edward Hope-Morley |
description |
This issue appears to be a consequence of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572 where we added a 'wait-for-state running' to the nova-compute upstart so as to ensure that neutron-ovs-cleanup has finished before nova-compute starts.
I have started to spot, however, that on some hosts (metal only) there is now a race between the two whereby nova-compute sometimes fails to start on system boot/reboot with the following in /var/log/upstart/nova-compute.log:
...
libvirt-bin stop/waiting
wait-for-state stop/waiting
neutron-ovs-cleanup start/pre-start, process 3084
start: Job failed to start
If I manually restart nova-compute all is fine. So this looks like a race between nova-compute's wait-for-state and neutron-ovs-cleanup's pre-start -> start/running. |
[Impact]
This issue appears to be a consequence of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572 where we added a 'wait-for-state running' to the nova-compute upstart so as to ensure that neutron-ovs-cleanup has finished before nova-compute starts.
I have started to spot, however, that on some hosts (metal only) there is now a race between the two whereby nova-compute sometimes fails to start on system boot/reboot with the following in /var/log/upstart/nova-compute.log:
...
libvirt-bin stop/waiting
wait-for-state stop/waiting
neutron-ovs-cleanup start/pre-start, process 3084
start: Job failed to start
If I manually restart nova-compute all is fine. So this looks like a race between nova-compute's wait-for-state and neutron-ovs-cleanup's pre-start -> start/running.
The proposed solution here is add some retry logic to nova-compute upstart job to tolerate neutron-ovs-cleanup not being able to start yet. We, therefore, allow a certain number of retries, every other with an incremented delay, before giving up and allowing nova-compute to start anyway. If ovs-cleanup failed to start after what is a failry liberal retry period, it is assumed to have failed altogether this making is safe(ish) to start nova-compute.
[Test Case]
In one terminal (as root) do:
service neutron-ovs-cleanup stop; service openvswitch-switch stop; service nova-compute restart
In another do:
sudo tail -F /var/log/upstart/nova-compute.log
Observe the retries occurring
Then do 'sudo service openvswitch-switch start' and observe nova-compute retry and succeed.
[Regression Potential]
* If openvswitch-switch does not start within the max retries and intervals nova-compute will start anyway and of ovs-cleanup were at some point to run one would see the behaviour that LP 1420572 was intended to resolve. It does not seem to make sense to wait indefinitely for ovs-cleanup to be up and the coded interval is pretty liberal and should be plenty enough. |
|
| 2015-07-03 18:42:30 |
Edward Hope-Morley |
nova (Ubuntu Trusty): status |
New |
In Progress |
|
| 2015-07-03 18:42:32 |
Edward Hope-Morley |
nova (Ubuntu Utopic): status |
New |
In Progress |
|
| 2015-07-03 18:42:35 |
Edward Hope-Morley |
nova (Ubuntu Vivid): status |
New |
In Progress |
|
| 2015-07-03 18:42:38 |
Edward Hope-Morley |
nova (Ubuntu Trusty): assignee |
|
Edward Hope-Morley (hopem) |
|
| 2015-07-03 18:42:39 |
Edward Hope-Morley |
nova (Ubuntu Utopic): assignee |
|
Edward Hope-Morley (hopem) |
|
| 2015-07-03 18:42:41 |
Edward Hope-Morley |
nova (Ubuntu Vivid): assignee |
|
Edward Hope-Morley (hopem) |
|
| 2015-07-03 18:43:12 |
Edward Hope-Morley |
description |
[Impact]
This issue appears to be a consequence of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572 where we added a 'wait-for-state running' to the nova-compute upstart so as to ensure that neutron-ovs-cleanup has finished before nova-compute starts.
I have started to spot, however, that on some hosts (metal only) there is now a race between the two whereby nova-compute sometimes fails to start on system boot/reboot with the following in /var/log/upstart/nova-compute.log:
...
libvirt-bin stop/waiting
wait-for-state stop/waiting
neutron-ovs-cleanup start/pre-start, process 3084
start: Job failed to start
If I manually restart nova-compute all is fine. So this looks like a race between nova-compute's wait-for-state and neutron-ovs-cleanup's pre-start -> start/running.
The proposed solution here is add some retry logic to nova-compute upstart job to tolerate neutron-ovs-cleanup not being able to start yet. We, therefore, allow a certain number of retries, every other with an incremented delay, before giving up and allowing nova-compute to start anyway. If ovs-cleanup failed to start after what is a failry liberal retry period, it is assumed to have failed altogether this making is safe(ish) to start nova-compute.
[Test Case]
In one terminal (as root) do:
service neutron-ovs-cleanup stop; service openvswitch-switch stop; service nova-compute restart
In another do:
sudo tail -F /var/log/upstart/nova-compute.log
Observe the retries occurring
Then do 'sudo service openvswitch-switch start' and observe nova-compute retry and succeed.
[Regression Potential]
* If openvswitch-switch does not start within the max retries and intervals nova-compute will start anyway and of ovs-cleanup were at some point to run one would see the behaviour that LP 1420572 was intended to resolve. It does not seem to make sense to wait indefinitely for ovs-cleanup to be up and the coded interval is pretty liberal and should be plenty enough. |
[Impact]
This issue appears to be a consequence of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572 where we added a 'wait-for-state running' to the nova-compute upstart so as to ensure that neutron-ovs-cleanup has finished before nova-compute starts.
I have started to spot, however, that on some hosts (metal only) there is now a race between the two whereby nova-compute sometimes fails to start on system boot/reboot with the following in /var/log/upstart/nova-compute.log:
...
libvirt-bin stop/waiting
wait-for-state stop/waiting
neutron-ovs-cleanup start/pre-start, process 3084
start: Job failed to start
If I manually restart nova-compute all is fine. So this looks like a race between nova-compute's wait-for-state and neutron-ovs-cleanup's pre-start -> start/running.
The proposed solution here is add some retry logic to nova-compute upstart job to tolerate neutron-ovs-cleanup not being able to start yet. We, therefore, allow a certain number of retries, every other with an incremented delay, before giving up and allowing nova-compute to start anyway. If ovs-cleanup failed to start after what is a failry liberal retry period, it is assumed to have failed altogether thus making is safe(ish) to start nova-compute.
[Test Case]
In one terminal (as root) do:
service neutron-ovs-cleanup stop; service openvswitch-switch stop; service nova-compute restart
In another do:
sudo tail -F /var/log/upstart/nova-compute.log
Observe the retries occurring
Then do 'sudo service openvswitch-switch start' and observe nova-compute retry and succeed.
[Regression Potential]
* If openvswitch-switch does not start within the max retries and intervals nova-compute will start anyway and of ovs-cleanup were at some point to run one would see the behaviour that LP 1420572 was intended to resolve. It does not seem to make sense to wait indefinitely for ovs-cleanup to be up and the coded interval is pretty liberal and should be plenty enough. |
|
| 2015-07-03 18:43:34 |
Edward Hope-Morley |
description |
[Impact]
This issue appears to be a consequence of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572 where we added a 'wait-for-state running' to the nova-compute upstart so as to ensure that neutron-ovs-cleanup has finished before nova-compute starts.
I have started to spot, however, that on some hosts (metal only) there is now a race between the two whereby nova-compute sometimes fails to start on system boot/reboot with the following in /var/log/upstart/nova-compute.log:
...
libvirt-bin stop/waiting
wait-for-state stop/waiting
neutron-ovs-cleanup start/pre-start, process 3084
start: Job failed to start
If I manually restart nova-compute all is fine. So this looks like a race between nova-compute's wait-for-state and neutron-ovs-cleanup's pre-start -> start/running.
The proposed solution here is add some retry logic to nova-compute upstart job to tolerate neutron-ovs-cleanup not being able to start yet. We, therefore, allow a certain number of retries, every other with an incremented delay, before giving up and allowing nova-compute to start anyway. If ovs-cleanup failed to start after what is a failry liberal retry period, it is assumed to have failed altogether thus making is safe(ish) to start nova-compute.
[Test Case]
In one terminal (as root) do:
service neutron-ovs-cleanup stop; service openvswitch-switch stop; service nova-compute restart
In another do:
sudo tail -F /var/log/upstart/nova-compute.log
Observe the retries occurring
Then do 'sudo service openvswitch-switch start' and observe nova-compute retry and succeed.
[Regression Potential]
* If openvswitch-switch does not start within the max retries and intervals nova-compute will start anyway and of ovs-cleanup were at some point to run one would see the behaviour that LP 1420572 was intended to resolve. It does not seem to make sense to wait indefinitely for ovs-cleanup to be up and the coded interval is pretty liberal and should be plenty enough. |
[Impact]
This issue appears to be a consequence of https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572 where we added a 'wait-for-state running' to the nova-compute upstart so as to ensure that neutron-ovs-cleanup has finished before nova-compute starts.
I have started to spot, however, that on some hosts (metal only) there is now a race between the two whereby nova-compute sometimes fails to start on system boot/reboot with the following in /var/log/upstart/nova-compute.log:
...
libvirt-bin stop/waiting
wait-for-state stop/waiting
neutron-ovs-cleanup start/pre-start, process 3084
start: Job failed to start
If I manually restart nova-compute all is fine. So this looks like a race between nova-compute's wait-for-state and neutron-ovs-cleanup's pre-start -> start/running.
The proposed solution here is add some retry logic to nova-compute upstart job to tolerate neutron-ovs-cleanup not being able to start yet. We, therefore, allow a certain number of retries, every other with an incremented delay, before giving up and allowing nova-compute to start anyway. If ovs-cleanup failed to start after what is a failry liberal retry period, it is assumed to have failed altogether thus making is safe(ish) to start nova-compute.
[Test Case]
In one terminal (as root) do:
service neutron-ovs-cleanup stop; service openvswitch-switch stop; service nova-compute restart
In another do:
sudo tail -F /var/log/upstart/nova-compute.log
Observe the retries occurring
Then do 'sudo service openvswitch-switch start' and observe nova-compute retry and succeed.
[Regression Potential]
If openvswitch-switch does not start within the max retries and intervals nova-compute will start anyway and of ovs-cleanup were at some point to run one would see the behaviour that LP 1420572 was intended to resolve. It does not seem to make sense to wait indefinitely for ovs-cleanup to be up and the coded interval is pretty liberal and should be plenty enough. |
|
| 2015-07-03 20:17:48 |
Ubuntu Foundations Team Bug Bot |
tags |
|
patch |
|
| 2015-07-03 20:17:55 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2015-07-04 11:36:34 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2015-07-06 10:50:39 |
Edward Hope-Morley |
attachment added |
|
wily.debdiff https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1471022/+attachment/4425020/+files/wily.debdiff |
|
| 2015-07-06 10:58:24 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-server-dev/nova/kilo |
|
| 2015-07-06 11:06:32 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-server-dev/nova/juno |
|
| 2015-07-06 11:48:21 |
Launchpad Janitor |
nova (Ubuntu): status |
In Progress |
Fix Released |
|
| 2015-07-08 14:41:16 |
Edward Hope-Morley |
attachment removed |
trusty.debdiff https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1471022/+attachment/4423971/+files/trusty.debdiff |
|
|
| 2015-07-08 14:41:35 |
Edward Hope-Morley |
attachment added |
|
trusty.debdiff https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1471022/+attachment/4426408/+files/trusty.debdiff |
|
| 2015-07-08 16:40:34 |
Chris J Arges |
nova (Ubuntu Trusty): status |
In Progress |
Fix Committed |
|
| 2015-07-08 16:40:37 |
Chris J Arges |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2015-07-08 16:40:39 |
Chris J Arges |
bug |
|
|
added subscriber SRU Verification |
| 2015-07-08 16:40:42 |
Chris J Arges |
tags |
patch |
patch verification-needed |
|
| 2015-07-08 16:40:47 |
Chris J Arges |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2015-07-08 16:44:39 |
Chris J Arges |
nova (Ubuntu Utopic): status |
In Progress |
Fix Committed |
|
| 2015-07-08 16:51:11 |
Chris J Arges |
nova (Ubuntu Vivid): status |
In Progress |
Fix Committed |
|
| 2015-07-15 13:40:43 |
Edward Hope-Morley |
tags |
patch verification-needed |
patch verification-done |
|
| 2015-07-15 13:52:35 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-server-dev/nova/icehouse |
|
| 2015-07-15 18:58:51 |
Launchpad Janitor |
nova (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2015-07-15 18:58:55 |
Chris J Arges |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2015-07-15 18:59:09 |
Launchpad Janitor |
nova (Ubuntu Utopic): status |
Fix Committed |
Fix Released |
|
| 2015-07-15 18:59:19 |
Launchpad Janitor |
nova (Ubuntu Vivid): status |
Fix Committed |
Fix Released |
|
| 2015-07-15 18:59:35 |
Chris J Arges |
removed subscriber SRU Verification |
|
|
|
| 2016-11-04 15:01:00 |
Nick |
bug |
|
|
added subscriber Nick |
