service neutron-l3-agent restart triggers ovs-cleanup on computes

Bug #1562565 reported by Eugene Nikanorov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
High
Sergey Kolekonov
7.0.x
Fix Released
Critical
Eugene Nikanorov
8.0.x
Invalid
High
Sergey Kolekonov
9.x
Invalid
High
Sergey Kolekonov

Bug Description

service neutron-l3-agent restart triggers ovs-cleanup on computes on ubuntu.
As a result, VM ports could be wiped out, which would not only break network connectivity for the instances, but also require hard rebooting them.

THe suspect is /etc/init/neutron-l3-agent.conf

Revision history for this message
Alexander Ignatov (aignatov) wrote :

Proposed fix for 7.0 maintenance branch is here:
https://review.fuel-infra.org/#/c/18789/

no longer affects: mos/8.0.x
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

I suggest to reevaluate a fix... I'm not sure the fix is correct

Revision history for this message
Sergey Kolekonov (skolekonov) wrote :

ovs-cleanup script is not mentioned in neutron-related init scripts for 8.0/9.0, so these releases should not be affected

Revision history for this message
Sergey Kolekonov (skolekonov) wrote :

I was unable to reproduce this bug - neutron-l3-agent init script doesn't trigger ovs-cleanup. Moving the bug to incomplete as steps to reproduce this bug are not clear

Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

We don't have much evidence for this bug.
Only a line in init.d logs saying that the service was started, but no proof.
Moving to Invalid because of lack of data.

Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

Reopening the bug.
I was able to reproduce the problem with both 'initctl neutron-l3-agent restart' and 'service neutron-l3-agent restart'
It seems that it may be caused by remaining pid file

Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Eugene, is https://review.fuel-infra.org/#/c/18789/ still valid fix for this issue?

Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

I think it is valid.
The issue hardly reproducible, but the removed lines is the only thing that connects l3 agent startup with ovs cleanup.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack-build/neutron-build (openstack-ci/fuel-7.0/2015.1.0)

Reviewed: https://review.fuel-infra.org/18789
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 16b57842e9f401ec107973898286eb8fe4de6212
Author: Sergey Kolekonov <email address hidden>
Date: Mon May 16 17:57:41 2016

Do not check ovs-cleanup job status on l3/dhcp agents start

Closes-Bug: #1562565

Change-Id: I5536fc6f20ea05c415ccf8be4717174f8cda5de7

tags: added: on-verification
Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

I tried to reproduce it with the following steps:
1. Deploy HA env with enabled DVR
2. Create instance
3. Open /var/log/neutron/ovs-cleanup.log on compute node (on which instance is hosted)
4. Restart services on compute node (on which instance is hosted):
root@node-2:~# service neutron-l3-agent restart
neutron-l3-agent stop/waiting
neutron-l3-agent start/running, process 22543
root@node-2:~# initctl restart neutron-l3-agent
neutron-l3-agent start/running, process 22457
5. Check /var/log/neutron/ovs-cleanup.log, other logs in /var/log/neutron folder

I couldn't reproduce this bug on MOS 7.0 without updates.
After applying MU4 updates the problem also couldn't be reproduced with steps above (appropriate changes in code are present).

Also please see email thread "Bug 1562565 is not reproduced" which confirms that there are no exact steps to reproduce for this issue, but it is required to get this fix merged.

tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.