multicast network verification fails with >11 nodes

Bug #1350007 reported by Andrew Woodward
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Medium
Fuel Python (Deprecated)

Bug Description

On HA configurations.
With more than 9 nodes it may occur some of the time.
With 12 or more nodes verify networks fails 100% of the time with message similar to

"
Verification failed.
Not received ids set([u'8']) for node 11.
Not received ids set([u'8']) for node 10.
Not received ids set([u'8']) for node 13.
Not received ids set([u'8']) for node 12.
Not received ids set([u'8']) for node 14.
Not received ids set([u'8']) for node 1.
Not received ids set([u'8']) for node 5.
Not received ids set([u'8']) for node 4.
Not received ids set([u'8']) for node 7.
Not received ids set([u'8']) for node 6.
Not received ids set([u'8']) for node 9.
Not received ids set([u'8']) for node 8.
"

Note the ID in the set will change every time and sometimes a few nodes may miss messages from other nodes.

"Not received ids" string is from https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/rpc/receiver.py#L866 (multicast_verification_resp)

Deselecting "Corosync\Need to pass network verification" does not prevent this issue from occurring

Revision history for this message
Dima Shulyak (dshulyak) wrote :

We disabled multicast for 5.1, so this issue should not occur with newest master.
I think it is also related to problems with python libpcap bindings. I will debug it in 6.0

Changed in fuel:
milestone: 5.1 → 6.0
summary: - network verification fails with >11 nodes
+ multicast network verification fails with >11 nodes
Andrew Woodward (xarses)
tags: added: customer-found
Revision history for this message
Andrew Woodward (xarses) wrote :

Please also improve the message to something like "Failed validating multicast: node {id} didn't see a response from {missing nodes}"

Revision history for this message
Mike Scherbakov (mihgen) wrote :

Andrew, why does this bug has "customer-found" tag? We never had this multicast verification enabled in our GA releases.

Revision history for this message
Andrew Woodward (xarses) wrote :

Customer found because someone in IRC reported it on one of the community-master builds

tags: added: release-notes
Revision history for this message
Dima Shulyak (dshulyak) wrote :

We are not using multicast verification in 5.1, no release-notes needed

tags: removed: release-notes
tags: added: release-notes
tags: removed: release-notes
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

Multicast checks are disabled now

Changed in fuel:
importance: High → Wishlist
status: Confirmed → Won't Fix
status: Won't Fix → Invalid
importance: Wishlist → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.