"Some untagged networks are assigned to same physical interface" error after adding new hosts to already deployed env

[Marcin Iwinski]

Detailed bug description:
Error "Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces." is displayed (but only for some hosts) when trying to deploy additional nodes into env using advanced network template.

Steps to reproduce:
1) enable advanced feature in Fuel
2) Create new environment
3) create a new network dedicated for ceph
fuel --env 1 network-group --create --node-group 1 --name "cluster" --release 2 --cidr "172.31.5.0/24" --vlan 504
4) upload custom network template
fuel --env 1 network-template --upload --dir ./
5) add 3 physical hosts and assign them a virt role
6) define 3 VMs on each hypervisor
for i in 19 20 22;do fuel2 node create-vms-conf $i --conf '{"id":1,"mem":128,"cpu":6,"vda_size":"200G"}' \ '{"id":2,"mem":128,"cpu":10,"vda_size":"200G"}' '{"id":3,"mem":128,"cpu":6,"vda_size":"900G"}';done

#id1 = for mongo, id2 = for controller, id3 - for LMA installed after the env is deployed (plugin is hot-plugabble)
7) Set proper IP ranges on network tabs (VLANs are already specified after the template is uploaded)
8) privison VMs via UI or via cli (fuel2 env spawn-vms 1)
9) once the new VMs are discovered assign them with Controller, and Mongo roles; add physical compute nodes and Ceph OSDs
10) Deploy changes
11) Install LMA plugin
12) enable and configure LMA plugin
13) add 3 spare VMs created in step 8 with LMA specific roles
14) deploy changes

Expected result:
Deployment of 3 additional hosts for LMA finishes successfully.

Actual result:
Deployment in step 14 fails with the following Error:
Error
Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:
"management", "public", "private" networks at node "node-1 (53:34)"
"management", "public", "private" networks at node "node-2 (50:b4)"
"management", "public", "private" networks at node "node-6 (fe:58)"
"management", "public", "private" networks at node "node-4 (50:ec)"
"management", "public", "private" networks at node "node-5 (53:30)"
"management", "public", "private" networks at node "node-9 (fb:90)"

All nodes mentioned in above error are only physical hosts (computes and ceph OSDs) added in step 9.

When checking network settings in Fuel WebUI VLANs are not specified anymore.

Impact:
High, the deployment cannot be finished without installing additional hosts.

Description of the environment:
Operation system:
Ubuntu 14.04
Version of components:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"
Network model:
Neutron with VXLANs

Additional Information:
Diagnostic snapshot can be provided privately to an engineer working on this bug since it might contain customer specific data.
Network template uploaded in step 4:
https://drive.google.com/open?id=0ByFVngOTho04SFF2LWJoQUk5c0k

[1] reduced footprint feature: https://docs.mirantis.com/openstack/fuel/fuel-8.0/operations.html#reduced-footprint-ops

[Aleksey Kasatkin (alekseyk-ru)]

Please provide more info: diagnostic snapshot and network template would be helpful.

[Marcin Iwinski (iwi)]

@alekseyk-ru I updated the description and added a link to the template.
As per my comment in bug description - diagnostic snapshot contains Customer specific data.
It was however shared internally with adidenko.

[Marcin Iwinski (iwi)]

output from 'fuel --env 1 network download':
https://drive.google.com/open?id=0ByFVngOTho04N2t2NVo2Rm8zLVk

[Aleksandr Didenko (adidenko)]

OK, so the work-around is:
1) Download network coinfig:
fuel --env 1 network download
2) Fix VLAN tags for networks (update "vlan_start" for networks in "networks" hash)
3) Upload network config:
fuel --env 1 network upload

After that deployment worked. So lowering priority to medium for 8.0 and 9.0.
But we still need to figure out how vlans got changed.

[Aleksey Kasatkin (alekseyk-ru)]

Also, we need to fix validation of network parameters in Nailgun (NetworkCheck class) so that VLAN IDs from the template were checked when template is applied.

[Krzysztof Szukiełojć (kszukielojc)]

Is there somewhere updated network_template for 10.0? As it seems that this template doesn't validate anymore.

[Krzysztof Szukiełojć (kszukielojc)]

I was able to get to 9 step in virtual env.
This are steps I used

Steps to reproduce:
1) enable advanced feature in Fuel
1a) Use virtio driver for network instead of e1000
2) Create new environment
3) create a new network dedicated for ceph
fuel --env 1 network-group --create --node-group 1 --name "cluster" --release 2 --cidr "10.109.5.0/24" --vlan 222

4) upload custom network template
fuel --env 1 network-template --upload --dir ./
5) add 3 physical hosts and assign them a virt role
6) define 1 VMs on each hypervisor
for i in 1 2 3;do fuel2 node create-vms-conf $i --conf '{"id":1,"mem":2,"cpu":1,"vda_size":"200G"}';done

#id1 = for mongo, id2 = for controller, id3 - for LMA installed after the env is deployed (plugin is hot-plugabble)
7) Set proper IP ranges on network tabs (VLANs are already specified after the template is uploaded)
7a) Modify /etc/puppet/modules/osnailyfacter/templates/vm_libvirt.erb

Add
 <bootmenu enable='yes'/>

And replace interfaces interfaces to
    <interface type='bridge'>
      <source bridge='br-ex'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-database'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-ceph'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-fw-admin'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-ex'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-storage'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-mgmt'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-ex'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-messaging'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-ha'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-mongo'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-openstack'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-services'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <source bridge='br-prv'/>
      <model type='virtio'/><virtualport type='openvswitch'/>
    </interface>

8) privison VMs via UI or via cli (fuel2 env spawn-vms 1)
9) once the new VMs are discovered assign them with Controller, and Mongo roles; add physical compute nodes and Ceph OSDs

Correct problem is that interfaces aren't created according to /etc/puppet/modules/osnailyfacter/templates/vm_libvirt.erb.
I tried different template/interafaces, but it had not mongo/ceph roles and I failed.

[Aleksey Kasatkin (alekseyk-ru)]

Targeted to 9.0-updates as we have duplicate (https://bugs.launchpad.net/fuel/+bug/1592744) targeted to it.

[Alexander Kurenyshev (akurenyshev)]

I think this bug is related https://bugs.launchpad.net/fuel/mitaka/+bug/1613216 to this issue.
In both cases we use re-deploy mechanism

[Georgy Kibardin (gkibardin)]

Setting it to medium since we have a workaround.

[Alexey Stupnikov (astupnikov)]

Closing as won't fix for MOS9 for the following reasons:

- this is a bug in environments that use 'reduced footprint' feature. It is not a bug in production envs.
- this bug has workaround.
- there is no fix to backport and it will take much effort to fix this issue.