Networking is not working with OVS bridges

[Oleksiy Molchanov]

Network scheme

{"endpoints"=>
  {"br-ex"=>
    {"IP"=>["10.109.18.6/24"],
     "gateway"=>"10.109.18.1",
     "vendor_specific"=>{"provider_gateway"=>"10.109.18.1"}},
   "br-floating"=>{"IP"=>"none"},
   "br-fw-admin"=>
    {"IP"=>["10.109.15.3/24"],
     "vendor_specific"=>{"provider_gateway"=>"10.109.15.1"}},
   "br-mgmt"=>{"IP"=>["192.168.0.5/24"]},
   "br-prv"=>{"IP"=>"none"},
   "br-storage"=>{"IP"=>["192.168.1.3/24"]}},
 "interfaces"=>
  {"enp0s3"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:03.0", "driver"=>"e1000"}},
   "enp0s4"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:04.0", "driver"=>"e1000"}},
   "enp0s5"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:05.0", "driver"=>"e1000"}},
   "enp0s6"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:06.0", "driver"=>"e1000"}},
   "enp0s7"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:07.0", "driver"=>"e1000"}}},
 "provider"=>"lnx",
 "roles"=>
  {"admin/pxe"=>"br-fw-admin",
   "aodh/api"=>"br-mgmt",
   "ceilometer/api"=>"br-mgmt",
   "ceph/public"=>"br-storage",
   "ceph/radosgw"=>"br-ex",
   "ceph/replication"=>"br-storage",
   "cinder/api"=>"br-mgmt",
   "cinder/iscsi"=>"br-storage",
   "ex"=>"br-ex",
   "fw-admin"=>"br-fw-admin",
   "glance/api"=>"br-mgmt",
   "glance/glare"=>"br-mgmt",
   "heat/api"=>"br-mgmt",
   "horizon"=>"br-mgmt",
   "ironic/api"=>"br-mgmt",
   "keystone/api"=>"br-mgmt",
   "management"=>"br-mgmt",
   "mgmt/corosync"=>"br-mgmt",
   "mgmt/database"=>"br-mgmt",
   "mgmt/memcache"=>"br-mgmt",
   "mgmt/messaging"=>"br-mgmt",
   "mgmt/vip"=>"br-mgmt",
   "mongo/db"=>"br-mgmt",
   "murano/api"=>"br-mgmt",
   "murano/cfapi"=>"br-mgmt",
   "neutron/api"=>"br-mgmt",
   "neutron/floating"=>"br-floating",
   "neutron/private"=>"br-prv",
   "nova/api"=>"br-mgmt",
   "nova/migration"=>"br-mgmt",
   "public/vip"=>"br-ex",
   "sahara/api"=>"br-mgmt",
   "storage"=>"br-storage",
   "swift/api"=>"br-mgmt",
   "swift/replication"=>"br-storage"},
 "transformations"=>
  [{"action"=>"add-br", "name"=>"br-fw-admin", "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-mgmt", "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-storage", "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-ex", "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-floating", "provider"=>"ovs"},
   {"action"=>"add-patch",
    "bridges"=>["br-floating", "br-ex"],
    "mtu"=>65000,
    "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-prv", "provider"=>"ovs"},
   {"action"=>"add-patch",
    "bridges"=>["br-prv", "br-fw-admin"],
    "mtu"=>65000,
    "provider"=>"ovs"},
   {"action"=>"add-port", "bridge"=>"br-fw-admin", "name"=>"enp0s3"},
   {"action"=>"add-port", "bridge"=>"br-mgmt", "name"=>"enp0s3.101"},
   {"action"=>"add-port", "bridge"=>"br-storage", "name"=>"enp0s3.102"},
   {"action"=>"add-port", "bridge"=>"br-ex", "name"=>"enp0s4"}],
 "version"=>"1.1"}

ovs-vsctl show

root@node-6:~# ovs-vsctl show
bf8b8f19-0176-4fc0-8211-a8f2d9557c24
    Bridge br-prv
        Port br-prv
            Interface br-prv
                type: internal
        Port "p_eeee51a2-1"
            Interface "p_eeee51a2-1"
                type: patch
                options: {peer="p_eeee51a2-0"}
    Bridge br-mgmt
        Port "enp0s3.101"
            tag: 101
            Interface "enp0s3.101"
                type: internal
        Port v_management
            Interface v_management
        Port br-mgmt
            Interface br-mgmt
                type: internal
        Port v_vrouter
            Interface v_vrouter
    Bridge br-fw-admin
        Port br-fw-admin
            Interface br-fw-admin
                type: internal
        Port "enp0s3"
            Interface "enp0s3"
        Port "p_eeee51a2-0"
            Interface "p_eeee51a2-0"
                type: patch
                options: {peer="p_eeee51a2-1"}
    Bridge br-floating
        Port "p_ff798dba-1"
            Interface "p_ff798dba-1"
                type: patch
                options: {peer="p_ff798dba-0"}
        Port br-floating
            Interface br-floating
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port v_vrouter_pub
            Interface v_vrouter_pub
        Port "enp0s4"
            Interface "enp0s4"
        Port v_public
            Interface v_public
        Port "p_ff798dba-0"
            Interface "p_ff798dba-0"
                type: patch
                options: {peer="p_ff798dba-1"}
    Bridge br-storage
        Port br-storage
            Interface br-storage
                type: internal
        Port "enp0s3.102"
            tag: 102
            Interface "enp0s3.102"
                type: internal
    ovs_version: "2.6.1"

https://drive.google.com/open?id=0ByBwUBq5EbLEUjcxVlZ0R2JnQW8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/458851

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/458879

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/458880

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/458881

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergey Vasilenko (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/458879

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergey Vasilenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/458881

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergey Vasilenko (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/458880

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/458881
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d096bfe121cba47ae27f5f803d3d65b239910c1e
Submitter: Jenkins
Branch: master

commit d096bfe121cba47ae27f5f803d3d65b239910c1e
Author: Sergey Vasilenko <email address hidden>
Date: Fri Apr 21 16:27:34 2017 +0300

    Fix OVS-related issues with underlay networking

    * more intelligent calculation of default port provider
    * additional check of patchcord existing in the bridge

    OVS provider should be used for fake (OVS) ports, inserted
    into OVS bridge. OVS provider should be used for inserting
    native linux interface into OVS bridge.

    But if we try to add native linux subinterface into
    OVS bridge we should use lnx provider for creating
    such port correctly.

    Change-Id: Ib76b3340eca1ea22095da1cfffe7c224a139fb71
    Closes-bug: #1682835

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/458880
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f7466d00c406e0fac03d0e012b21107933574bdd
Submitter: Jenkins
Branch: stable/ocata

commit f7466d00c406e0fac03d0e012b21107933574bdd
Author: Sergey Vasilenko <email address hidden>
Date: Fri Apr 21 16:27:34 2017 +0300

    Fix OVS-related issues with underlay networking

    * more intelligent calculation of default port provider
    * additional check of patchcord existing in the bridge

    OVS provider should be used for fake (OVS) ports, inserted
    into OVS bridge. OVS provider should be used for inserting
    native linux interface into OVS bridge.

    But if we try to add native linux subinterface into
    OVS bridge we should use lnx provider for creating
    such port correctly.

    Change-Id: Ib76b3340eca1ea22095da1cfffe7c224a139fb71
    Closes-bug: #1682835

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/458851
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e6907520ae67ad35105ecbf609161c4bf0901f10
Submitter: Jenkins
Branch: stable/mitaka

commit e6907520ae67ad35105ecbf609161c4bf0901f10
Author: Sergey Vasilenko <email address hidden>
Date: Fri Apr 21 16:27:34 2017 +0300

    Fix OVS-related issues with underlay networking

    * more intelligent calculation of default port provider
    * additional check of patchcord existing in the bridge

    OVS provider should be used for fake (OVS) ports, inserted
    into OVS bridge. OVS provider should be used for inserting
    native linux interface into OVS bridge.

    But if we try to add native linux subinterface into
    OVS bridge we should use lnx provider for creating
    such port correctly.

    Change-Id: Ib76b3340eca1ea22095da1cfffe7c224a139fb71
    Closes-bug: #1682835

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/465591

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/465601

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/465602

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/465591
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=22e7dcb5396b503636c0a2f4ba14f100fd2850d1
Submitter: Jenkins
Branch: stable/mitaka

commit 22e7dcb5396b503636c0a2f4ba14f100fd2850d1
Author: Sergey Vasilenko <email address hidden>
Date: Wed May 17 17:02:03 2017 +0300

    Add more strict checking for veth pair into OVS bridge.

    Some time VETH pair may be self-disassempled or broken by 3d party action,
    but OVS doesn't know about it and does not remove port with broken interface.
    Such ports we can detect by 'ifindex' property. For really existing interfaces
    'ifindex' property should be equal to real ifindex. For broken interfaces
    'ifindex' mabe have zero or empty value.

    This patchset add checking for it.

    Also:
     * improove logging
     * monitor() function was refactored, because previous logic had no full
       coverage of failure cases.
     * set sysctl -w net.ipv4.ip_nonlocal_bind=1 into network namespace while creating

    Change-Id: I05062f84895e020b10a94ac09f8573598434b0e6
    Closes-Bug: #1682835

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: master
Review: https://review.opendev.org/465601
Reason: This repo is retired now, no further work will get merged.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/ocata
Review: https://review.opendev.org/465602
Reason: This repo is retired now, no further work will get merged.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/newton
Review: https://review.opendev.org/458879
Reason: This repo is retired now, no further work will get merged.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library ocata-eol release.