Few Vm's not getting IP due to missing security group rules in scale scenarios

Bug #1591114 reported by Krishna Kanth
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-hyperv
Fix Released
Undecided
Krishna Kanth

Bug Description

Few Vm's are not getting IP due to missing security group rules(outbound rules and sometimes inbound dhcp allow rules in scale scenarios of 1000)

Steps to reproduce:

1. Boot 1000 vm's on a scale setup of 20 computes or more
2. Ping the VM's after boot
3. Some fail to ping as they failed to get ip due to missing outbound allow rules and some times missing inbound udp rules for dhcp ports.

Root cause: Rules are getting missed due to synchronization issues while adding and removing rules.

Changed in networking-hyperv:
assignee: nobody → Krishna Kanth (krishna-kanth-mallela)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-hyperv (master)

Fix proposed to branch: master
Review: https://review.openstack.org/328210

Changed in networking-hyperv:
status: New → In Progress
Revision history for this message
Krishna Kanth (krishna-kanth-mallela) wrote :
Revision history for this message
Krishna Kanth (krishna-kanth-mallela) wrote :

Please note that https://review.openstack.org/328210 and https://review.openstack.org/#/c/328218/ are required to solve this issue.
greenthread.sleep is no more required reason being:
1. currently all wmi rules are applied at one shot and port binding is much faster and reporting is not a problem. verified that there is no issue with reporting threads.
2. Also we observed that sleep results in giving away the lock when there are wmi exceptions while applying rules hence causing sync problems in scale.
3. Also this solves the problem of wmi exception file already exists while applying rule problem which is also a sync problem which https://review.openstack.org/328210 resolves.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/os-win 1.1.0

This issue was fixed in the openstack/os-win 1.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-hyperv (master)

Reviewed: https://review.openstack.org/328210
Committed: https://git.openstack.org/cgit/openstack/networking-hyperv/commit/?id=ff663bc8f3aa136a3c6b8c32a85e76d5327bfc8b
Submitter: Jenkins
Branch: master

commit ff663bc8f3aa136a3c6b8c32a85e76d5327bfc8b
Author: krishna kanth <email address hidden>
Date: Fri Jun 10 15:08:05 2016 +0530

    Fix for VM's not getting IP due to missing SG rules

    This patch set fixes missing outbound/inbound rules due
    to missing SG rules due to synchronizatin issues while
    adding and removing rules seen on a scale environment

    Change-Id: I716f64a74add80e11e3ce7ac8641578d63eec69b
    Co-Authored-By: sonu <email address hidden>
    Co-Authored-By: ShivaKumar M <email address hidden>
    Co-Authored-By: Vinod Kumar <email address hidden>
    Closes-Bug: 1591114

Changed in networking-hyperv:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-hyperv 3.0.0.0rc1

This issue was fixed in the openstack/networking-hyperv 3.0.0.0rc1 release candidate.

Revision history for this message
Krishna Kanth (krishna-kanth-mallela) wrote :

Cheery-picking the changes from master to stable/mitaka for os-win repo.
https://review.openstack.org/#/c/413508/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-hyperv (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/413519

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on networking-hyperv (stable/mitaka)

Change abandoned by Claudiu Belu (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/413519
Reason: Patch breaks stable/mitaka, and was not updated since January. We are also approaching Mitaka EOL.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-win 0.5.0

This issue was fixed in the openstack/os-win 0.5.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.