Network templates are not working for multiple nodegroups for VLAN segmentation

[Aleksey Chuprin]

Fuel 7.0 + MU2
Neutron OVS + VLAN

Steps to reproduce:
- create new fuel env with neutron VLAN
- create new nodegroup
- configure networks
- upload network template

Actual result:

- deployment has been failed

Nailgun Trace:

  File "/usr/lib/python2.6/site-packages/nailgun/task/manager.py", line 58, in _call_silently
    to_return = method(task, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nailgun/task/task.py", line 166, in message
    orchestrator_graph, task.cluster, nodes)
  File "/usr/lib/python2.6/site-packages/nailgun/orchestrator/deployment_serializers.py", line 575, in serialize
    cluster, nodes, ignore_customized=ignore_customized)
  File "/usr/lib/python2.6/site-packages/nailgun/orchestrator/deployment_serializers.py", line 82, in serialize
    cluster, node_group))
  File "/usr/lib/python2.6/site-packages/nailgun/orchestrator/deployment_serializers.py", line 91, in serialize_generated
    nodes = self.serialize_nodes(nodes)
  File "/usr/lib/python2.6/site-packages/nailgun/orchestrator/deployment_serializers.py", line 238, in serialize_nodes
    serialized_nodes.append(self.serialize_node(node, role))
  File "/usr/lib/python2.6/site-packages/nailgun/orchestrator/deployment_serializers.py", line 450, in serialize_node
    DeploymentHASerializer61, self).serialize_node(node, role)
  File "/usr/lib/python2.6/site-packages/nailgun/orchestrator/deployment_serializers.py", line 258, in serialize_node
    node_attrs.update(net_serializer.get_node_attrs(node))
  File "/usr/lib/python2.6/site-packages/nailgun/orchestrator/base_serializers.py", line 199, in get_node_attrs
    return cls.network_provider_node_attrs(node.cluster, node)
  File "/usr/lib/python2.6/site-packages/nailgun/orchestrator/neutron_serializers.py", line 64, in network_provider_node_attrs
    'network_scheme': cls.generate_network_scheme(node, networks),
  File "/usr/lib/python2.6/site-packages/nailgun/orchestrator/neutron_serializers.py", line 1224, in generate_network_scheme
    networks)
  File "/usr/lib/python2.6/site-packages/nailgun/orchestrator/neutron_serializers.py", line 579, in generate_routes
    netgroup = netgroups[ngname]
KeyError: u'private'

Part of the network template:
....
adv_net_template:
  group1:
...
    network_scheme:
      private:
        endpoints:
        - br-prv
        roles:
          neutron/mesh: null
          neutron/private: br-prv
        transformations:
        - action: add-br
          name: br-prv
          provider: ovs
        - action: add-patch
          bridges:
          - br-prv
          - br-ex
          mtu: 65000
          provider: ovs

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/292503

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/292503
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=24647834000c7ac9085f9230d77b6e35396d9a2f
Submitter: Jenkins
Branch: master

commit 24647834000c7ac9085f9230d77b6e35396d9a2f
Author: Aleksey Kasatkin <email address hidden>
Date: Mon Mar 14 18:42:55 2016 +0200

    Robust generating of routes

    'netgroups[ngname]' is replaced with 'netgroups.get(ngname, {})' as
    in other places in serializer.
    Problem was the 'br-prv' endpoint that didn't have IP in case of VLAN
    segmentation. The difference between 'netgroup_mapping' and 'netgroups'
    exists only in case of using network template. Otherwise, absence of IP
    for 'br-prv' did not lead to this issue.

    Also, there is a small refactoring of serialization tests was made here.

    Change-Id: I67616409017631023e707e533cb8e2faab943b34
    Closes-Bug: #1556917

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/297014

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/297044

[Alexey Stupnikov (astupnikov)]

Patch is tested and works for MOS 7.0. This issue has a big list of steps-to-reproduce, so please contact me in case you need them.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/297044
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=25a2392e86a48d55e81f0da1e92105f1668d64ec
Submitter: Jenkins
Branch: stable/7.0

commit 25a2392e86a48d55e81f0da1e92105f1668d64ec
Author: Aleksey Kasatkin <email address hidden>
Date: Mon Mar 14 18:42:55 2016 +0200

    Robust generating of routes

    'netgroups[ngname]' is replaced with 'netgroups.get(ngname, {})' as
    in other places in serializer.

    Problem was the 'br-prv' endpoint that didn't have IP in case of VLAN
    segmentation. The difference between 'netgroup_mapping' and 'netgroups'
    exists only in case of using network template. Otherwise, absence of IP
    for 'br-prv' did not lead to this issue.

    Also, there is a small refactoring of serialization tests was made here.

    Conflicts:
     nailgun/nailgun/orchestrator/neutron_serializers.py
     nailgun/nailgun/test/integration/test_orchestrator_serializer_70.py
     nailgun/nailgun/test/integration/test_orchestrator_serializer_80.py
     nailgun/nailgun/test/integration/test_orchestrator_serializer_90.py

    Change-Id: I67616409017631023e707e533cb8e2faab943b34
    Closes-Bug: #1556917

[Alexander Gromov (agromov)]

Verified on MOS 7.0 + mu3

Before updates:
Deployment failed

logs with ERROR:
http://paste.openstack.org/show/493754/

After updates:
Deployment is successfully finished.

logs at the beginning:
http://paste.openstack.org/show/493755/

[Alexandr Kostrikov (akostrikov-mirantis)]

I have created env with vlan network segmentation
and used template for two nodegroups[0]

At first run I have left interface assignments the same, for second run I have changed assignment
on second node-group.

    nic_mapping:
      default:
        if11: enp0s3 # admin
        if12: enp0s4 # public
        if13: enp0s7 # management
        if14: enp0s6 # private
        if15: enp0s5 # storage

Output on changed node:
root@node-5:~# brctl show
bridge name bridge id STP enabled interfaces
br-aux 8000.22395ad5aa05 no enp0s6
       p_e52381cd-0
br-ex 8000.64ae338b8958 no enp0s4
       p_ff798dba-0
br-fw-admin 8000.640331016a93 no enp0s3
br-mgmt 8000.64371e3cebb0 no enp0s7
br-storage 8000.6446974e3d3d no enp0s5
root@node-5:~# hiera roles
["cinder", "compute"]

Output on default nodegroup:

root@node-3:~# brctl show
bridge name bridge id STP enabled interfaces
br-aux 8000.6286ab901dee no enp0s6
       p_e52381cd-0
br-ex 8000.6472f7a55363 no enp0s4
       p_ff798dba-0
       v_public
br-fw-admin 8000.64c5e700c1b8 no enp0s3
br-mgmt 8000.64df8d5ce38c no enp0s5
       mgmt-conntrd
br-storage 8000.64104fc7e3c1 no enp0s7
root@node-3:~# hiera roles
["controller"]

The deployment has passed with both cases.

[0] https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/network_templates/two_nodegroups.yaml

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/297014
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=a3f21c65c0e25255d4bec47570384b154ef01040
Submitter: Jenkins
Branch: stable/8.0

commit a3f21c65c0e25255d4bec47570384b154ef01040
Author: Aleksey Kasatkin <email address hidden>
Date: Mon Mar 14 18:42:55 2016 +0200

    Robust generating of routes

    'netgroups[ngname]' is replaced with 'netgroups.get(ngname, {})' as
    in other places in serializer.
    Problem was the 'br-prv' endpoint that didn't have IP in case of VLAN
    segmentation. The difference between 'netgroup_mapping' and 'netgroups'
    exists only in case of using network template. Otherwise, absence of IP
    for 'br-prv' did not lead to this issue.

    Also, there is a small refactoring of serialization tests was made here.

    Change-Id: I67616409017631023e707e533cb8e2faab943b34
    Closes-Bug: #1556917

[Vladimir Jigulin (vjigulin)]

Reproduced on 8.0 without updates with error like in #1 post

But when verification on 8.0+(mu3-proposed) I got other error in same moment:
  File "/usr/lib/python2.7/site-packages/nailgun/orchestrator/neutron_serializers.py", line 64, in network_provider_node_attrs
    'network_scheme': cls.generate_network_scheme(node, networks),
  File "/usr/lib/python2.7/site-packages/nailgun/orchestrator/neutron_serializers.py", line 1255, in generate_network_scheme
    netgroups[public_net]['gateway']
KeyError: u'public'

Full trace: http://paste.openstack.org/show/rPXMKzzkeQCuYzfiMaKp/

Fail at line: https://github.com/openstack/fuel-web/blob/stable/8.0/nailgun/nailgun/orchestrator/neutron_serializers.py#L1254-L1255

In both cases used template: https://raw.githubusercontent.com/openstack/fuel-qa/stable/8.0/fuelweb_test/network_templates/two_nodegroups.yaml

[Alexey Stupnikov (astupnikov)]

Vladimir, this kind of error means that something wrong in your config. Lets try to review the environment.

[Vladimir Jigulin (vjigulin)]

Probably I makes mistake in network-group config.
I did verification from scratch and all was fine.