Networking is not working with OVS bridges

Bug #1682835 reported by Oleksiy Molchanov on 2017-04-14
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Critical
Sergey Vasilenko
Nominated for Ocata by Oleksiy Molchanov
Mitaka
Critical
Sergey Vasilenko
Newton
Critical
Sergii Golovatiuk

Bug Description

Network scheme

{"endpoints"=>
  {"br-ex"=>
    {"IP"=>["10.109.18.6/24"],
     "gateway"=>"10.109.18.1",
     "vendor_specific"=>{"provider_gateway"=>"10.109.18.1"}},
   "br-floating"=>{"IP"=>"none"},
   "br-fw-admin"=>
    {"IP"=>["10.109.15.3/24"],
     "vendor_specific"=>{"provider_gateway"=>"10.109.15.1"}},
   "br-mgmt"=>{"IP"=>["192.168.0.5/24"]},
   "br-prv"=>{"IP"=>"none"},
   "br-storage"=>{"IP"=>["192.168.1.3/24"]}},
 "interfaces"=>
  {"enp0s3"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:03.0", "driver"=>"e1000"}},
   "enp0s4"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:04.0", "driver"=>"e1000"}},
   "enp0s5"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:05.0", "driver"=>"e1000"}},
   "enp0s6"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:06.0", "driver"=>"e1000"}},
   "enp0s7"=>
    {"ethtool"=>{"offload"=>{}},
     "vendor_specific"=>{"bus_info"=>"0000:00:07.0", "driver"=>"e1000"}}},
 "provider"=>"lnx",
 "roles"=>
  {"admin/pxe"=>"br-fw-admin",
   "aodh/api"=>"br-mgmt",
   "ceilometer/api"=>"br-mgmt",
   "ceph/public"=>"br-storage",
   "ceph/radosgw"=>"br-ex",
   "ceph/replication"=>"br-storage",
   "cinder/api"=>"br-mgmt",
   "cinder/iscsi"=>"br-storage",
   "ex"=>"br-ex",
   "fw-admin"=>"br-fw-admin",
   "glance/api"=>"br-mgmt",
   "glance/glare"=>"br-mgmt",
   "heat/api"=>"br-mgmt",
   "horizon"=>"br-mgmt",
   "ironic/api"=>"br-mgmt",
   "keystone/api"=>"br-mgmt",
   "management"=>"br-mgmt",
   "mgmt/corosync"=>"br-mgmt",
   "mgmt/database"=>"br-mgmt",
   "mgmt/memcache"=>"br-mgmt",
   "mgmt/messaging"=>"br-mgmt",
   "mgmt/vip"=>"br-mgmt",
   "mongo/db"=>"br-mgmt",
   "murano/api"=>"br-mgmt",
   "murano/cfapi"=>"br-mgmt",
   "neutron/api"=>"br-mgmt",
   "neutron/floating"=>"br-floating",
   "neutron/private"=>"br-prv",
   "nova/api"=>"br-mgmt",
   "nova/migration"=>"br-mgmt",
   "public/vip"=>"br-ex",
   "sahara/api"=>"br-mgmt",
   "storage"=>"br-storage",
   "swift/api"=>"br-mgmt",
   "swift/replication"=>"br-storage"},
 "transformations"=>
  [{"action"=>"add-br", "name"=>"br-fw-admin", "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-mgmt", "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-storage", "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-ex", "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-floating", "provider"=>"ovs"},
   {"action"=>"add-patch",
    "bridges"=>["br-floating", "br-ex"],
    "mtu"=>65000,
    "provider"=>"ovs"},
   {"action"=>"add-br", "name"=>"br-prv", "provider"=>"ovs"},
   {"action"=>"add-patch",
    "bridges"=>["br-prv", "br-fw-admin"],
    "mtu"=>65000,
    "provider"=>"ovs"},
   {"action"=>"add-port", "bridge"=>"br-fw-admin", "name"=>"enp0s3"},
   {"action"=>"add-port", "bridge"=>"br-mgmt", "name"=>"enp0s3.101"},
   {"action"=>"add-port", "bridge"=>"br-storage", "name"=>"enp0s3.102"},
   {"action"=>"add-port", "bridge"=>"br-ex", "name"=>"enp0s4"}],
 "version"=>"1.1"}

ovs-vsctl show

root@node-6:~# ovs-vsctl show
bf8b8f19-0176-4fc0-8211-a8f2d9557c24
    Bridge br-prv
        Port br-prv
            Interface br-prv
                type: internal
        Port "p_eeee51a2-1"
            Interface "p_eeee51a2-1"
                type: patch
                options: {peer="p_eeee51a2-0"}
    Bridge br-mgmt
        Port "enp0s3.101"
            tag: 101
            Interface "enp0s3.101"
                type: internal
        Port v_management
            Interface v_management
        Port br-mgmt
            Interface br-mgmt
                type: internal
        Port v_vrouter
            Interface v_vrouter
    Bridge br-fw-admin
        Port br-fw-admin
            Interface br-fw-admin
                type: internal
        Port "enp0s3"
            Interface "enp0s3"
        Port "p_eeee51a2-0"
            Interface "p_eeee51a2-0"
                type: patch
                options: {peer="p_eeee51a2-1"}
    Bridge br-floating
        Port "p_ff798dba-1"
            Interface "p_ff798dba-1"
                type: patch
                options: {peer="p_ff798dba-0"}
        Port br-floating
            Interface br-floating
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port v_vrouter_pub
            Interface v_vrouter_pub
        Port "enp0s4"
            Interface "enp0s4"
        Port v_public
            Interface v_public
        Port "p_ff798dba-0"
            Interface "p_ff798dba-0"
                type: patch
                options: {peer="p_ff798dba-1"}
    Bridge br-storage
        Port br-storage
            Interface br-storage
                type: internal
        Port "enp0s3.102"
            tag: 102
            Interface "enp0s3.102"
                type: internal
    ovs_version: "2.6.1"

https://drive.google.com/open?id=0ByBwUBq5EbLEUjcxVlZ0R2JnQW8

description: updated
Changed in fuel:
milestone: none → 10.x-updates
milestone: 10.x-updates → 11.x-updates
tags: added: sla2
Anton Matveev (amatveev) on 2017-04-20
tags: added: sla1
removed: sla2

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/458880

Changed in fuel:
status: Confirmed → In Progress

Change abandoned by Sergey Vasilenko (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/458879

Change abandoned by Sergey Vasilenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/458881

Change abandoned by Sergey Vasilenko (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/458880

Reviewed: https://review.openstack.org/458881
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d096bfe121cba47ae27f5f803d3d65b239910c1e
Submitter: Jenkins
Branch: master

commit d096bfe121cba47ae27f5f803d3d65b239910c1e
Author: Sergey Vasilenko <email address hidden>
Date: Fri Apr 21 16:27:34 2017 +0300

    Fix OVS-related issues with underlay networking

    * more intelligent calculation of default port provider
    * additional check of patchcord existing in the bridge

    OVS provider should be used for fake (OVS) ports, inserted
    into OVS bridge. OVS provider should be used for inserting
    native linux interface into OVS bridge.

    But if we try to add native linux subinterface into
    OVS bridge we should use lnx provider for creating
    such port correctly.

    Change-Id: Ib76b3340eca1ea22095da1cfffe7c224a139fb71
    Closes-bug: #1682835

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/458880
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f7466d00c406e0fac03d0e012b21107933574bdd
Submitter: Jenkins
Branch: stable/ocata

commit f7466d00c406e0fac03d0e012b21107933574bdd
Author: Sergey Vasilenko <email address hidden>
Date: Fri Apr 21 16:27:34 2017 +0300

    Fix OVS-related issues with underlay networking

    * more intelligent calculation of default port provider
    * additional check of patchcord existing in the bridge

    OVS provider should be used for fake (OVS) ports, inserted
    into OVS bridge. OVS provider should be used for inserting
    native linux interface into OVS bridge.

    But if we try to add native linux subinterface into
    OVS bridge we should use lnx provider for creating
    such port correctly.

    Change-Id: Ib76b3340eca1ea22095da1cfffe7c224a139fb71
    Closes-bug: #1682835

tags: added: in-stable-ocata

Reviewed: https://review.openstack.org/458851
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e6907520ae67ad35105ecbf609161c4bf0901f10
Submitter: Jenkins
Branch: stable/mitaka

commit e6907520ae67ad35105ecbf609161c4bf0901f10
Author: Sergey Vasilenko <email address hidden>
Date: Fri Apr 21 16:27:34 2017 +0300

    Fix OVS-related issues with underlay networking

    * more intelligent calculation of default port provider
    * additional check of patchcord existing in the bridge

    OVS provider should be used for fake (OVS) ports, inserted
    into OVS bridge. OVS provider should be used for inserting
    native linux interface into OVS bridge.

    But if we try to add native linux subinterface into
    OVS bridge we should use lnx provider for creating
    such port correctly.

    Change-Id: Ib76b3340eca1ea22095da1cfffe7c224a139fb71
    Closes-bug: #1682835

Reviewed: https://review.openstack.org/465591
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=22e7dcb5396b503636c0a2f4ba14f100fd2850d1
Submitter: Jenkins
Branch: stable/mitaka

commit 22e7dcb5396b503636c0a2f4ba14f100fd2850d1
Author: Sergey Vasilenko <email address hidden>
Date: Wed May 17 17:02:03 2017 +0300

    Add more strict checking for veth pair into OVS bridge.

    Some time VETH pair may be self-disassempled or broken by 3d party action,
    but OVS doesn't know about it and does not remove port with broken interface.
    Such ports we can detect by 'ifindex' property. For really existing interfaces
    'ifindex' property should be equal to real ifindex. For broken interfaces
    'ifindex' mabe have zero or empty value.

    This patchset add checking for it.

    Also:
     * improove logging
     * monitor() function was refactored, because previous logic had no full
       coverage of failure cases.
     * set sysctl -w net.ipv4.ip_nonlocal_bind=1 into network namespace while creating

    Change-Id: I05062f84895e020b10a94ac09f8573598434b0e6
    Closes-Bug: #1682835

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers