_update_network_segmentation_id KeyError: 'provider:network_type'

[LIU Yulong]

ENV: master

Exception:
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager [None req-1c6689ba-8524-4f93-9387-d5bcce5101dd None None] Error during notification for neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSPluginApi._legacy_notifier-1722583 Network, after_update: KeyError: 'provider:network_type'
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager File "/usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs)
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/agent/rpc.py", line 259, in _legacy_notifier
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager getattr(self._legacy_interface, method)(context, **payload)
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager result = f(*args, **kwargs)
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 572, in network_update
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager self._update_network_segmentation_id(network)
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 418, in _update_network_segmentation_id
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager if network[provider_net.NETWORK_TYPE] != n_const.TYPE_VLAN:
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager KeyError: 'provider:network_type'
Jun 14 00:51:28 network2 neutron-openvswitch-agent[10081]: ERROR neutron_lib.callbacks.manager

For tenant HA network, the provider network type may in such style:

{
 "network": {
  "updated_at": "2019-02-19T13:25:15Z",
  "revision_number": 5,
  "id": "ba973781-065b-4c69-9f07-a5c4f9a3b754",
  "router:external": false,
  "availability_zone_hints": [],
  "availability_zones": ["nova"],
  "segments": [{
   "provider:network_type": "vxlan",
   "provider:physical_network": null,
   "provider:segmentation_id": 10041
  },
  {
   "provider:network_type": "vxlan",
   "provider:physical_network": null,
   "provider:segmentation_id": 10066
  }],
  "ipv4_address_scope": null,
  "shared": false,
  "project_id": "",
  "l2_adjacency": true,
  "status": "ACTIVE",
  "subnets": ["3e098a8e-9e1f-456b-a429-2ab46493e6b5"],
  "description": "",
  "tags": [],
  "ipv6_address_scope": null,
  "qos_policy_id": null,
  "name": "HA network tenant ae05deb7a16c485986c3666b65f71c71",
  "admin_state_up": true,
  "tenant_id": "",
  "created_at": "2018-12-19T14:53:20Z",
  "mtu": 1450
 }
}

[Brian Haley (brian-haley)]

There seem to be a lot of places in the code that assume a network has all the elements that _update_network_segmentation_id() is trying to access, I see it in trunk and L3 DB code as well, so a bigger audit of the code will be necessary to fix this.

[Miguel Lavalle (minsel)]

What are the steps to reproduce this?

[LIU Yulong (dragon889)]

Just a simple ovs-agent restart, this is a full start log:

Jun 16 00:04:58 network2 neutron-openvswitch-agent[14609]: INFO neutron.common.config [-] Logging enabled!
Jun 16 00:04:58 network2 neutron-openvswitch-agent[14609]: INFO neutron.common.config [-] /usr/bin/neutron-openvswitch-agent version 14.1.0.dev83
Jun 16 00:04:58 network2 neutron-openvswitch-agent[14609]: INFO os_ken.base.app_manager [-] loading app neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp
Jun 16 00:04:59 network2 neutron-openvswitch-agent[14609]: INFO os_ken.base.app_manager [-] loading app os_ken.app.ofctl.service
Jun 16 00:04:59 network2 neutron-openvswitch-agent[14609]: INFO os_ken.base.app_manager [-] loading app os_ken.controller.ofp_handler
Jun 16 00:04:59 network2 neutron-openvswitch-agent[14609]: INFO os_ken.base.app_manager [-] instantiating app os_ken.app.ofctl.service of OfctlService
Jun 16 00:04:59 network2 neutron-openvswitch-agent[14609]: INFO os_ken.base.app_manager [-] instantiating app os_ken.controller.ofp_handler of OFPHandler
Jun 16 00:04:59 network2 neutron-openvswitch-agent[14609]: INFO os_ken.base.app_manager [-] instantiating app neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_oskenapp of OVSNeutronAgentOSKenApp
Jun 16 00:04:59 network2 neutron-openvswitch-agent[14609]: INFO neutron.agent.agent_extensions_manager [-] Loaded agent extensions: []
Jun 16 00:04:59 network2 neutron-openvswitch-agent[14609]: INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmp6W2OGZ/privsep.sock']
Jun 16 00:04:59 network2 sudo[14622]: yulong : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --privsep_context neutron.privileged.default --privsep_sock_path /tmp/tmp6W2OGZ/privsep.sock
Jun 16 00:05:00 network2 neutron-openvswitch-agent[14609]: INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap
Jun 16 00:05:00 network2 neutron-openvswitch-agent[14609]: INFO oslo.privsep.daemon [-] privsep daemon starting
Jun 16 00:05:00 network2 neutron-openvswitch-agent[14609]: INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
Jun 16 00:05:00 network2 neutron-openvswitch-agent[14609]: INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/none
Jun 16 00:05:00 network2 neutron-openvswitch-agent[14609]: INFO oslo.privsep.daemon [-] privsep daemon running as pid 14639
Jun 16 00:05:01 network2 neutron-openvswitch-agent[14609]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int has datapath-ID 0000e2e21157754c
Jun 16 00:05:02 network2 neutron-openvswitch-agent[14609]: INFO neutron.plugins...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/666375

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/666375
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=dc80fc9fe7836840d25eac16c4f8013cda55acfa
Submitter: Zuul
Branch: master

commit dc80fc9fe7836840d25eac16c4f8013cda55acfa
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Jun 19 15:54:22 2019 +0000

    [OVS] Network segmentation ID change not allowed if multisegments

    If a network has several segments, the provider network segmentation ID
    cannot be changed. This is defined in the feature spec [1].

    In the case of having a multisegment network, the OVS agent RPC call
    "get_network_details" will return the following value:

      {
        "network": {
          "updated_at": "2019-02-19T13:25:15Z",
          "revision_number": 5,
          "id": "ba973781-065b-4c69-9f07-a5c4f9a3b754",
          ...
          "segments": [{
            "provider:network_type": "vxlan",
            "provider:physical_network": null,
            "provider:segmentation_id": 10041
          },
          {
            "provider:network_type": "vxlan",
            "provider:physical_network": null,
            "provider:segmentation_id": 10066
          }],
          ...
    }

    The provider network information will be contained inside the "segments"
    list, instead of being container in the parent "network" dictionary.

    Closes-Bug: #1832745

    [1]https://opendev.org/openstack/neutron-specs/src/branch/master/specs/stein/change-segmentation-id-vlan-provider-network.rst

    Change-Id: I4fa37519bbf91e93ebd2f0e46e4d14edd40728fd

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.