Bug #1834556 “ovs cannot use unaccelerated interfaces” : Bugs : StarlingX

Litao Gao (gaolitao) on 2019-06-28

tags:	added: stx.2.0
tags:	added: stx.config

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-06-28:

#1

@Litao, In the future, please do not add tags. This is done during screening.

tags:

added: stx.networking
removed: stx.config

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-06-28:

#2

Please attach the collect logs from your system.

Changed in starlingx:
status:	New → Incomplete
assignee:	nobody → Litao Gao (gaolitao)

Revision history for this message

Litao Gao (gaolitao) wrote on 2019-07-01:

#3

/var/log/puppet/2019-07-01-00-37-51_worker Edit (823.5 KiB, text/plain)

/var/log/puppet/2019-07-01-00-37-51_worker

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-07-04:

#4

Logs provided. Assigning to Forrest's team to investigate.

Changed in starlingx:
assignee:	Litao Gao (gaolitao) → Forrest Zhao (forrest.zhao)
status:	Incomplete → Triaged
importance:	Undecided → Medium

Forrest Zhao (forrest.zhao) on 2019-07-04

Changed in starlingx:
assignee:	Forrest Zhao (forrest.zhao) → marvin Yu (marvin-yu)

Revision history for this message

marvin Yu (marvin-yu) wrote on 2019-07-09:

#5

Hi Litao,
I Tried it in the VM,and didn`t reproduce the issue.did I miss any steps?

Revision history for this message

marvin Yu (marvin-yu) wrote on 2019-07-10:

#6

Download full text (3.7 KiB)

Hi Litao,
I have created datanetwork and assign it to a unaccelerated interface,the interface`s named "bugdata" and uuid is 464dca96-c18b-4850-8999-ba77e0bd6c42;

after host unlock, it seems that enp0s10 works well.

[sysadmin@controller-0 ~(keystone_admin)]$ sudo ovs-vsctl show
Password:
7d554c33-3cc2-4c29-b0dd-f305c6ab48f8
    Manager "ptcp:6640:127.0.0.1"
    Bridge "br-phy0"
        Controller "tcp:127.0.0.1:6633"
        fail_mode: secure
        Port "eth1000"
            Interface "eth1000"
        Port "enp0s10"
            Interface "enp0s10"
        Port "p...

Hi Litao,
I have created datanetwork and assign it to a unaccelerated interface,the interface`s named "bugdata" and uuid is 464dca96-c18b-4850-8999-ba77e0bd6c42;

after host unlock, it seems that enp0s10 works well.

[sysadmin@controller-0 ~(keystone_admin)]$ sudo ovs-vsctl show
Password:
7d554c33-3cc2-4c29-b0dd-f305c6ab48f8
    Manager "ptcp:6640:127.0.0.1"
    Bridge "br-phy0"
        Controller "tcp:127.0.0.1:6633"
        fail_mode: secure
        Port "eth1000"
            Interface "eth1000"
        Port "enp0s10"
            Interface "enp0s10"
        Port "phy-br-phy0"
            Interface "phy-br-phy0"
                type: patch
                options: {peer="int-br-phy0"}
        Port "br-phy0"
            Interface "br-phy0"
                type: internal
......
in my environment, there only have one 'docker0' bridge. but in yours, the interface already has been assign to the linux bridge,so it doesn`t works.  can you provide more information about you configuration?(related about you interfaces).

[sysadmin@controller-0 ~(keystone_admin)]$ brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.02426ff9eea8       no

Revision history for this message

Litao Gao (gaolitao) wrote on 2019-07-16:

#7

Download full text (8.7 KiB)

Hi Marvin,

The interface related configuration pasted in this comment.

I am using STX R2 milestone3 ISO image for this testing.

Per the related code (sysinv/puppet/interface.py), it seems that the bridge will be created if
the interface is data ethernet and not dpdk compatible.

672 bridge = None
673 if (iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET and
674 is_data_interface(context, iface) and
675 not is_dpdk_compatible(context, iface)):
676 bridge = 'br-' + get_interface_os_ifname(context, iface)
677 iface['_bridge'] = bridge # cache the result

Hi Marvin,

The interface related configuration pasted in this comment.

I am using STX R2 milestone3 ISO image for this testing.

Per the related code (sysinv/puppet/interface.py), it seems that the bridge will be created if 
the interface is data ethernet and not dpdk compatible.

672         bridge = None
 673         if (iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET and
 674                 is_data_interface(context, iface) and
 675                 not is_dpdk_compatible(context, iface)):
 676             bridge = 'br-' + get_interface_os_ifname(context, iface)
 677         iface['_bridge'] = bridge  # cache the result

[root@controller-0 ~(keystone_admin)]# system host-port-list controller-0
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-----------------------------------------+
| uuid                                 | name   | type     | pci address  | device | processor | accelerated | device type                             |
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-----------------------------------------+
| 1716a76f-cf19-46eb-bdde-6564357ac385 | eno1   | ethernet | 0000:03:00.0 | 0      | 0         | False       | NetXtreme BCM5719 Gigabit Ethernet PCIe |
| 44b2628a-2280-47c3-b437-ccaf37257233 | eno2   | ethernet | 0000:03:00.1 | 0      | 0         | False       | NetXtreme BCM5719 Gigabit Ethernet PCIe |
| 907fb9c0-b930-4f91-8753-de3022ef1368 | eno3   | ethernet | 0000:03:00.2 | 0      | 0         | False       | NetXtreme BCM5719 Gigabit Ethernet PCIe |
| bc002f2f-f606-4f4e-acac-ef25432e20d6 | eno4   | ethernet | 0000:03:00.3 | 0      | 0         | False       | NetXtreme BCM5719 Gigabit Ethernet PCIe |
| e83801d1-96a5-4fdd-9a90-4933213c77ed | ens1f0 | ethernet | 0000:04:00.0 | 0      | 0         | True        | 82599ES 10-Gigabit SFI/SFP+ Network     |
|                                      |        |          |              |        |           |             | Connection                              |
|                                      |        |          |              |        |           |             |                                         |
| 29e6496f-dd93-4d77-8554-06154ac7ebe1 | ens1f1 | ethernet | 0000:04:00.1 | 0      | 0         | True        | 82599ES 10-Gigabit SFI/SFP+ Network     |
|                                      |        |          |              |        |           |             | Connection                              |
|                                      |        |          |              |        |           |             |                                         |
+--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-----------------------------------------+
[root@controller-0 ~(keystone_admin)]# system host-port-show controller-0 eno4
+-----------------------+-----------------------------------------+
| Property              | Value                                   |
+-----------------------+-----------------------------------------+
| name                  | eno4                                    |
| namedisplay           | None                                    |
| type                  | ethernet                                |
| pciaddr               | 0000:03:00.3                            |
| dev_id                | 0                                       |
| processor             | 0                                       |
| sriov_totalvfs        | None                                    |
| sriov_numvfs          | 0                                       |
| sriov_vfs_pci_address |                                         |
| sriov_vf_driver       | None                                    |
| driver                | tg3                                     |
| pclass                | Ethernet controller                     |
| pvendor               | Broadcom Inc. and subsidiaries          |
| pdevice               | NetXtreme BCM5719 Gigabit Ethernet PCIe |
| capabilities          | {}                                      |
| uuid                  | bc002f2f-f606-4f4e-acac-ef25432e20d6    |
| host_uuid             | 5b863404-eee7-468d-b0cc-56bdfc45f4ea    |
| interface_uuid        | 4507463c-3c71-44be-9d57-0bae63d38b5e    |
| accelerated           | False                                   |
| created_at            | 2019-06-25T07:29:29.956130+00:00        |
| updated_at            | 2019-06-25T07:29:30.800482+00:00        |
+-----------------------+-----------------------------------------+
[root@controller-0 ~(keystone_admin)]# system host-if-list controller-0
+--------------------------------------+-------+----------+----------+------+-------------+------+------+-------------------------+---------------+
| uuid                                 | name  | class    | type     | vlan | ports       | uses | used | attributes              | data networks |
|                                      |       |          |          | id   |             | i/f  | by   |                         |               |
|                                      |       |          |          |      |             |      | i/f  |                         |               |
+--------------------------------------+-------+----------+----------+------+-------------+------+------+-------------------------+---------------+
| 2783d422-10bb-4ccf-bd80-7fa2d84ed7cc | eno1  | platform | ethernet | None | [u'eno1']   | []   | []   | MTU=1500                | []            |
| 4507463c-3c71-44be-9d57-0bae63d38b5e | data0 | data     | ethernet | None | [u'eno4']   | []   | []   | MTU=1500,accelerated=   | [u'physnet0'] |
|                                      |       |          |          |      |             |      |      | False                   |               |
|                                      |       |          |          |      |             |      |      |                         |               |
| b07fcce7-3e9a-4cb5-bc58-8a285a54869f | lo    | platform | virtual  | None | []          | []   | []   | MTU=1500                | []            |
| d45e77dc-7727-4b17-9060-d6bcf1a30c4a | data1 | data     | ethernet | None | [u'ens1f0'] | []   | []   | MTU=1500,accelerated=   | [u'physnet1'] |
|                                      |       |          |          |      |             |      |      | True                    |               |
|                                      |       |          |          |      |             |      |      |                         |               |
+--------------------------------------+-------+----------+----------+------+-------------+------+------+-------------------------+---------------+
[root@controller-0 ~(keystone_admin)]# system host-if-show controller-0 data0
+-----------------+--------------------------------------+
| Property        | Value                                |
+-----------------+--------------------------------------+
| ifname          | data0                                |
| iftype          | ethernet                             |
| ports           | [u'eno4']                            |
| datanetworks    | [u'physnet0']                        |
| imac            | ac:16:2d:72:cd:3f                    |
| imtu            | 1500                                 |
| ifclass         | data                                 |
| aemode          | None                                 |
| schedpolicy     | None                                 |
| txhashpolicy    | None                                 |
| uuid            | 4507463c-3c71-44be-9d57-0bae63d38b5e |
| ihost_uuid      | 5b863404-eee7-468d-b0cc-56bdfc45f4ea |
| vlan_id         | None                                 |
| uses            | []                                   |
| used_by         | []                                   |
| created_at      |                                      |
| updated_at      |                                      |
| sriov_numvfs    | 0                                    |
| sriov_vf_driver | None                                 |
| ipv4_mode       | disabled                             |
| ipv6_mode       | disabled                             |
| accelerated     | [False]                              |
+-----------------+--------------------------------------+
[root@controller-0 ~(keystone_admin)]# system datanetwork-list
+--------------------------------------+----------+--------------+------+
| uuid                                 | name     | network_type | mtu  |
+--------------------------------------+----------+--------------+------+
| 1f42b27c-dbc9-42ea-8756-d07e449c3fda | physnet0 | flat         | 1500 |
| 0033c750-7f1d-403c-b9b4-08286fc8e086 | physnet1 | vlan         | 1500 |
+--------------------------------------+----------+--------------+------+

Revision history for this message

marvin Yu (marvin-yu) wrote on 2019-07-17:

#8

Hi Litao,

I tried it in the VM,and didn`t reproduce the issue. In my test environment, the bridge name is 'br-<ifname>'
never has been created.

did you create the bridge br-eno4 manually or not?

What configuration did you specify for the eno4 interface?

Can you provide the file that path is /opt/platform/puppet/XXXX/hieradata/<hostaddr>.yaml?

Revision history for this message

Litao Gao (gaolitao) wrote on 2019-07-19:

#9

192.168.204.3.yaml Edit (10.2 KiB, text/plain)

Hi Marvin,

Are you also using milestone3?

The br-eno4 is not created manually.

/opt/platform/puppet/19.01/hieradata/192.168.204.3.yaml has been attached.

Revision history for this message

marvin Yu (marvin-yu) wrote on 2019-07-19:

#10

I was using the ISO image with the same BUILD_ID, but I`m not sure if it`s exactly the same.

could you provide the ISO download link?

By the way, what is your test environment? (vm or bare metal)

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-07-25:

#11

Matt Peters believes this is a duplicate of https://bugs.launchpad.net/starlingx/+bug/1832697
Fixed by https://review.opendev.org/#/c/670286/
Merged on 2019-07-23

Marvin, do you agree?

Revision history for this message

marvin Yu (marvin-yu) wrote on 2019-07-26:

#12

192.168.204.3.yaml Edit (10.1 KiB, text/plain)

Download full text (3.3 KiB)

I think this Bug is different form that one.

After investigating the cause, the Bug is caused as below:

Create datanetwork and assign to a interface that is not is_dpdk_compatible, then the linux bridge configuration will be genernated for this interface. the detail as below:

----------------192.168.204.3.yaml-----------------------
!!python/unicode 'enp0s8':
    ensure: present
    family: inet
    hotplug: 'false'
    method: manual
    mtu: '1500'
    onboot: 'true'
    options:
      BRIDGE: !!python/unicode 'br-enp0s8'
      LINKDELAY: '20'
----------------------------------------------------------

after host unlock, the interface 'enp0s8' has been used by 'br-enp0s8'. But the ovs agent still auto add the interface to the ovs bridge 'br-phy1'.

------------openstack-neutron.yaml-------------------------
    Bridge "br-phy1"
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "phy-br-phy1"
            Interface "phy-br-phy1"
                type: patch
                options: {peer="int-br-phy1"}
        Port "enp0s8"
            Interface "enp0s8"
                error: "could not add network device enp0s8 to ofproto (Device or resource busy)"
----------------------------------------------------------

@Matt, I do not know the details it designed. so, what should I do?

the related code flow:
----------------------------------------------------------------------------------------------------------
def get_interface_network_config(...):
......
# Add type specific options
if iface['iftype'] == constants.INTERFACE_TYPE_VLAN:
config = get_vlan_network_config(context, iface, config)
elif iface['iftype'] == constants.INTERFACE_TYPE_AE:
config = get_bond_network_config(context, iface, config)
else:
config = get_ethernet_network_config(context, iface, config)
......
----------------------------------------------------------------------------------------------------------
def get_ethernet_network_config(...):
if is_bridged_interface(context, iface):
options['BRIDGE'] = get_bridge_interface_name(context, iface)
elif ......:
......
----------------------------------------------------------------------------------------------------------
def is_bridged_interface(context, iface):

    if '_bridged' in iface: # check the cached result
        return iface['_bridged']
    else:
        bridge = get_bridge_interface_name(context, iface)
        iface['_bridged'] = bool(bridge) # cache the result
        return iface['_bridged']
----------------------------------------------------------------------------------------------------------
def get_bridge_interface_name(context, iface):
    """
    If the given interface is a bridge member then retrieve the bridge
    interface name otherwise return None.
    """
    if '_bridge' in iface: # check the cached result
        return iface['_bridge']
    else:
        bridge = None
        if (iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET and
                is_data_interface(context, iface) and
                not is_dpdk_compatible(context, iface)):
            bridge = 'br-' + get_interface_os_ifname(context, iface)
        iface['...

I think this Bug is different form that one.

After investigating the cause, the Bug is caused as below:

Create datanetwork and assign to a interface that is not is_dpdk_compatible, then the linux bridge configuration will be genernated for this interface. the detail as below:

----------------192.168.204.3.yaml-----------------------
 !!python/unicode 'enp0s8':
    ensure: present
    family: inet
    hotplug: 'false'
    method: manual
    mtu: '1500'
    onboot: 'true'
    options:
      BRIDGE: !!python/unicode 'br-enp0s8'
      LINKDELAY: '20'
----------------------------------------------------------

after host unlock, the interface 'enp0s8' has been used by 'br-enp0s8'. But the ovs agent still auto add the interface to the ovs bridge 'br-phy1'.

------------openstack-neutron.yaml-------------------------
    Bridge "br-phy1"
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "phy-br-phy1"
            Interface "phy-br-phy1"
                type: patch
                options: {peer="int-br-phy1"}
        Port "enp0s8"
            Interface "enp0s8"
                error: "could not add network device enp0s8 to ofproto (Device or resource busy)"
----------------------------------------------------------

@Matt, I do not know the details it designed. so, what should I do?

the related code flow:
----------------------------------------------------------------------------------------------------------
def get_interface_network_config(...):
......
# Add type specific options
if iface['iftype'] == constants.INTERFACE_TYPE_VLAN:
config = get_vlan_network_config(context, iface, config)
elif iface['iftype'] == constants.INTERFACE_TYPE_AE:
config = get_bond_network_config(context, iface, config)
else:
config = get_ethernet_network_config(context, iface, config) 
......
----------------------------------------------------------------------------------------------------------
def get_ethernet_network_config(...):
if is_bridged_interface(context, iface):
options['BRIDGE'] = get_bridge_interface_name(context, iface)
    elif ......:
......
----------------------------------------------------------------------------------------------------------
def is_bridged_interface(context, iface):

if '_bridged' in iface:  # check the cached result
        return iface['_bridged']
    else:
        bridge = get_bridge_interface_name(context, iface)
        iface['_bridged'] = bool(bridge)  # cache the result
        return iface['_bridged']
----------------------------------------------------------------------------------------------------------
def get_bridge_interface_name(context, iface):
    """
    If the given interface is a bridge member then retrieve the bridge
    interface name otherwise return None.
    """
    if '_bridge' in iface:  # check the cached result
        return iface['_bridge']
    else:
        bridge = None
        if (iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET and
                is_data_interface(context, iface) and
                not is_dpdk_compatible(context, iface)):
            bridge = 'br-' + get_interface_os_ifname(context, iface)
        iface['_bridge'] = bridge  # cache the result
        return iface['_bridge']
----------------------------------------------------------------------------------------------------------

Revision history for this message

Matt Peters (mpeters-wrs) wrote on 2019-08-08:

#13

get_bridge_interface_name will need an additional check (not is_vswitch_type_unaccelerated) to determine if the vswitch type supports accelerated interfaces. If it doesn't support accelerated interfaces, a bridge should not be created.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-12: Fix proposed to config (master)

#14

Fix proposed to branch: master
Review: https://review.opendev.org/675811

Changed in starlingx:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-13: Fix merged to config (master)

#15

Reviewed: https://review.opendev.org/675811
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=8a4fd5dc79565a29d359bc2d99841e7ebea7e1ef
Submitter: Zuul
Branch: master

commit 8a4fd5dc79565a29d359bc2d99841e7ebea7e1ef
Author: marvin <email address hidden>
Date: Mon Aug 12 02:26:15 2019 +0000

Add an additional check for data interface configuration

    If an interface is data interface and the vswitch type is not
    accelerated, the data interface is a member of a linux bridge
    and then the linux bridge will occupy the interface after host
    unlock. if so, it will cause conflict with ovs bridge when ovs
    agent add the interface to ovs bridge.
    this patch add an additional check to determine if the vswitch
    type supports accelerated interfaces. If it doesn`t support
    accelerated interfaces, a bridge should not be created.

    Change-Id: I0fbae866c364fe7b787aa850db79bca1bf597389
    Closes-bug: #1834556
    Signed-off-by: marvin <email address hidden>

Changed in starlingx:
status:	In Progress → Fix Released

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-08-13:

#16

@Marvin, please cherrypick this to the stx.2.0 release branch.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-14: Fix proposed to config (r/stx.2.0)

#17

Fix proposed to branch: r/stx.2.0
Review: https://review.opendev.org/676299

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-14: Fix merged to config (r/stx.2.0)

#18

Reviewed: https://review.opendev.org/676299
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=5b65f38599d4c1286875770562ba9101a2f24c1f
Submitter: Zuul
Branch: r/stx.2.0

commit 5b65f38599d4c1286875770562ba9101a2f24c1f
Author: marvin <email address hidden>
Date: Mon Aug 12 02:26:15 2019 +0000

Add an additional check for data interface configuration

    If an interface is data interface and the vswitch type is not
    accelerated, the data interface is a member of a linux bridge
    and then the linux bridge will occupy the interface after host
    unlock. if so, it will cause conflict with ovs bridge when ovs
    agent add the interface to ovs bridge.
    this patch add an additional check to determine if the vswitch
    type supports accelerated interfaces. If it doesn`t support
    accelerated interfaces, a bridge should not be created.

    Change-Id: I0fbae866c364fe7b787aa850db79bca1bf597389
    Closes-bug: #1834556
    Signed-off-by: marvin <email address hidden>
    (cherry picked from commit 8a4fd5dc79565a29d359bc2d99841e7ebea7e1ef)

Ghada Khalil (gkhalil) on 2019-08-14

tags:

added: in-r-stx20

StarlingX

ovs cannot use unaccelerated interfaces

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches