[nailgun] Network verification fails on Cisco UCS 5108 & Cisco N2K-C2232PP

[Artem Panchenko]

api: '1.0'
astute_sha: a7eac46348dc77fc2723c6fcc3dbc66cc1a83152
build_id: 2014-05-27_05-51-41
build_number: '26'
fuellib_sha: 2f79c0415159651fc1978d99bd791079d1ae4a06
fuelmain_sha: 505741e4f431f85a8d0252fc42754d10c0326c1a
mirantis: 'yes'
nailgun_sha: bd09f89ef56176f64ad5decd4128933c96cb20f4
ostf_sha: a8b7660082a6f152794c610d6abe30d360fd577d
production: docker
release: '5.0'

While performing network verification on hardware cluster (Cisco UCS 5108 blades connected to the Cisco N2K-C2232PP-10GE switches) it always fails with the following errors:

http://paste.openstack.org/show/83928/

But after deploying of environment, network connectivity works fine and cluster passes all health checks.

Here is the environment network configuration:

http://paste.openstack.org/show/83929/

Configuration of network interfaces on the nodes and diagnostic snapshot you can find in attachments. Also, I tried to catch control packets on the nodes "problem" interfaces using tcpdump and all tagged packets were received except ones that were sent from the same node.

[Dima Shulyak (dshulyak)]

If anyone interested here is pcap capture of traffic at the moment of network verification

[Dima Shulyak (dshulyak)]

Seems like problem on side of python libpcap bindings.. because i can see all this traffic with tcpdump, and with pcap module
i see no traffic at all

[Dima Shulyak (dshulyak)]

So the problem is:
Every port is configured as trunk for each vlan, so any packet can end up in any eth interface,
but we are expecting to receive packet on exact interface, that is configured with nailgun

[Dima Shulyak (dshulyak)]

Second problem:

Sometimes node dont see its own broadcasted traffic, but receives all packets from other nodes

[Dima Shulyak (dshulyak)]

Well i am not sure.. should we consider this a bug?
Configuration in nailgun affects only eth0 and eth1, but trunks configured for any interface with all vlans,
so generated traffic can end up in any interface, even eth2 or eth3 that is not used in nailgun configuration

[Dima Shulyak (dshulyak)]

The problem was that all interfaces was configured as trunks for all vlans