Contrail4.0- SRIOV port creation throws unknown exception

[Pratik]

Hi,

I have 2 node ( 1 controller + 1 Compute node) setup. I have 8 10G nic cards on compute node, I am trying to use one of the nic cards for SRIOV traffic.
But, I am seeing "Unknown Neutron Exception" while creating the port.

Here the detailed logs:

Controller node:

root@blr-mx86-01:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-heat 4.0.0.0-20 20
contrail-nova-networkapi 4.0.0.0-20 20
contrail-openstack 4.0.0.0-20 20
contrail-openstack-dashboard 4.0.0.0-20 20
contrail-setup 4.0.0.0-20 20
contrail-utils 4.0.0.0-20 20
contrail-web-core 4.0.0.0-20 20
neutron-plugin-contrail 4.0.0.0-20 20
nova-api 2:14.0.2-0ubuntu1~cloud0.1contrail 20
nova-common 2:14.0.2-0ubuntu1~cloud0.1contrail 20
nova-conductor 2:14.0.2-0ubuntu1~cloud0.1contrail 20
nova-console 2:14.0.2-0ubuntu1~cloud0.1contrail 20
nova-consoleauth 2:14.0.2-0ubuntu1~cloud0.1contrail 20
nova-novncproxy 2:14.0.2-0ubuntu1~cloud0.1contrail 20
nova-scheduler 2:14.0.2-0ubuntu1~cloud0.1contrail 20
python-contrail 4.0.0.0-20 20
python-neutronclient 1:6.0.0-0ubuntu1~cloud0.2contrail 20
python-nova 2:14.0.2-0ubuntu1~cloud0.1contrail 20
root@blr-mx86-01:~#

root@blr-mx86-01:~# neutron net-show c93b846c-26be-43fc-978f-b995167606df
+---------------------------+---------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------------+---------------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| description | |
| fq_name | default-domain |
| | admin |
| | vn1 |
| id | c93b846c-26be-43fc-978f-b995167606df |
| instance_count | 0 |
| name | vn1 |
| port_security_enabled | True |
| project_id | 07bc3b5fc94a45de9a57a9c8bae80ce2 |
| provider:physical_network | host2physnet1 |
| provider:segmentation_id | 100 |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnet_ipam | {"subnet_cidr": "10.1.1.0/24", "ipam_fq_name": ["default-domain", "default-project", "default-network-ipam"]} |
| subnets | 82486f5a-4b40-47e7-99dc-f29bcf712524 |
| tenant_id | 07bc3b5fc94a45de9a57a9c8bae80ce2 |
+---------------------------+---------------------------------------------------------------------------------------------------------------+
root@blr-mx86-01:~# neutron subnet-show 82486f5a-4b40-47e7-99dc-f29bcf712524
+--------------------+--------------------------------------------+
| Field | Value |
+--------------------+--------------------------------------------+
| allocation_pools | {"start": "10.1.1.2", "end": "10.1.1.254"} |
| cidr | 10.1.1.0/24 |
| dns_nameservers | |
| dns_server_address | 10.1.1.2 |
| enable_dhcp | True |
| gateway_ip | 10.1.1.1 |
| host_routes | |
| id | 82486f5a-4b40-47e7-99dc-f29bcf712524 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | |
| network_id | c93b846c-26be-43fc-978f-b995167606df |
| tenant_id | 07bc3b5fc94a45de9a57a9c8bae80ce2 |
+--------------------+--------------------------------------------+
root@blr-mx86-01:~#

root@blr-mx86-01:~# neutron --debug port-create --fixed-ip subnet_id=82486f5a-4b40-47e7-99dc-f29bcf712524 --name test1 c93b846c-26be-43fc-978f-b995167606df --binding:vnic_type direct
DEBUG: stevedore.extension found extension EntryPoint.parse('aodh-noauth = aodhclient.noauth:AodhNoAuthLoader')
DEBUG: stevedore.extension found extension EntryPoint.parse('token_endpoint = openstackclient.api.auth_plugin:TokenEndpoint')
DEBUG: stevedore.extension found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oauth1 = keystoneauth1.extras.oauth1._loading:V3OAuth1')
DEBUG: stevedore.extension found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG: stevedore.extension found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3samlpassword = keystoneauth1.extras._saml2._loading:Saml2Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcaccesstoken = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3kerberos = keystoneauth1.extras.kerberos._loading:Kerberos')
DEBUG: stevedore.extension found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcclientcredentials = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3tokenlessauth = keystoneauth1.loading._plugins.identity.v3:TokenlessAuth')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3totp = keystoneauth1.loading._plugins.identity.v3:TOTP')
DEBUG: stevedore.extension found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3fedkerb = keystoneauth1.extras.kerberos._loading:MappedKerberos')
DEBUG: stevedore.extension found extension EntryPoint.parse('table = cliff.formatters.table:TableFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('json = cliff.formatters.json_format:JSONFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('shell = cliff.formatters.shell:ShellFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('value = cliff.formatters.value:ValueFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('yaml = cliff.formatters.yaml_format:YAMLFormatter')
DEBUG: neutronclient.neutron.v2_0.port.CreatePort run(Namespace(admin_state=True, allowed_address_pairs=[], binding_profile=None, columns=[], description=None, device_id=None, device_owner=None, dns_name=None, extra_dhcp_opts=[], fixed_ip=[{u'subnet_id': u'82486f5a-4b40-47e7-99dc-f29bcf712524'}], formatter='table', mac_address=None, max_width=0, name=u'test1', network_id=u'c93b846c-26be-43fc-978f-b995167606df', no_allowed_address_pairs=False, no_security_groups=False, noindent=False, prefix='', qos_policy=None, request_format='json', security_groups=[], tenant_id=None, variables=[], vnic_type=None))
DEBUG: keystoneauth.session REQ: curl -g -i -X GET http://10.216.168.177:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: neutron keystoneauth1/2.12.1 python-requests/2.10.0 CPython/2.7.12"
DEBUG: keystoneauth.session RESP: [200] Date: Thu, 10 Aug 2017 08:38:30 GMT Server: Apache/2.4.18 (Ubuntu) Vary: X-Auth-Token X-Distribution: Ubuntu x-openstack-request-id: req-d4a297da-73e2-47c0-8dfa-017ddf8e8cf5 Content-Length: 344 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json
RESP BODY: {"version": {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.216.168.177:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG: keystoneauth.identity.v2 Making authentication request to http://10.216.168.177:5000/v2.0/tokens
DEBUG: keystoneauth.session REQ: curl -g -i -X GET http://102.4.2.1:9696/v2.0/networks.json?fields=id&id=c93b846c-26be-43fc-978f-b995167606df -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}74a83efd79c090f3a32acbb306c3ce3571b4faa5"
DEBUG: keystoneauth.session RESP: [200] Content-Type: application/json Content-Length: 62 X-Openstack-Request-Id: req-cff98a34-48c4-4df4-87c9-769a228f3342 Date: Thu, 10 Aug 2017 08:38:30 GMT Connection: keep-alive
RESP BODY: {"networks": [{"id": "c93b846c-26be-43fc-978f-b995167606df"}]}

DEBUG: neutronclient.v2_0.client GET call to neutron for http://102.4.2.1:9696/v2.0/networks.json?fields=id&id=c93b846c-26be-43fc-978f-b995167606df used request id req-cff98a34-48c4-4df4-87c9-769a228f3342
DEBUG: keystoneauth.session REQ: curl -g -i -X GET http://102.4.2.1:9696/v2.0/subnets.json?fields=id&id=82486f5a-4b40-47e7-99dc-f29bcf712524 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}74a83efd79c090f3a32acbb306c3ce3571b4faa5"
DEBUG: keystoneauth.session RESP: [200] Content-Type: application/json Content-Length: 61 X-Openstack-Request-Id: req-409c43c7-8a4d-4c44-9b87-fb3514037966 Date: Thu, 10 Aug 2017 08:38:30 GMT Connection: keep-alive
RESP BODY: {"subnets": [{"id": "82486f5a-4b40-47e7-99dc-f29bcf712524"}]}

DEBUG: neutronclient.v2_0.client GET call to neutron for http://102.4.2.1:9696/v2.0/subnets.json?fields=id&id=82486f5a-4b40-47e7-99dc-f29bcf712524 used request id req-409c43c7-8a4d-4c44-9b87-fb3514037966
DEBUG: keystoneauth.session REQ: curl -g -i -X POST http://102.4.2.1:9696/v2.0/ports.json -H "User-Agent: python-neutronclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}74a83efd79c090f3a32acbb306c3ce3571b4faa5" -d '{"port": {"network_id": "c93b846c-26be-43fc-978f-b995167606df", "binding:vnic_type": "direct", "fixed_ips": [{"subnet_id": "82486f5a-4b40-47e7-99dc-f29bcf712524"}], "name": "test1", "admin_state_up": true}}'
DEBUG: keystoneauth.session RESP: [500] Content-Type: application/json Content-Length: 105 X-Openstack-Request-Id: req-da57b32b-eac1-4e6f-8157-f328e5b5f91d Date: Thu, 10 Aug 2017 08:38:30 GMT Connection: keep-alive
RESP BODY: {"NeutronError": {"message": "An unknown exception occurred.", "type": "NeutronException", "detail": ""}}

DEBUG: neutronclient.v2_0.client Error message: {"NeutronError": {"message": "An unknown exception occurred.", "type": "NeutronException", "detail": ""}}
DEBUG: neutronclient.v2_0.client POST call to neutron for http://102.4.2.1:9696/v2.0/ports.json used request id req-da57b32b-eac1-4e6f-8157-f328e5b5f91d
ERROR: neutronclient.shell An unknown exception occurred.
Neutron server returns request_ids: ['req-da57b32b-eac1-4e6f-8157-f328e5b5f91d']
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/neutronclient/shell.py", line 898, in run_subcommand
    return run_command(cmd, cmd_parser, sub_argv)
  File "/usr/lib/python2.7/dist-packages/neutronclient/shell.py", line 117, in run_command
    return cmd.run(known_args)
  File "/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py", line 324, in run
    return super(NeutronCommand, self).run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 100, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py", line 407, in take_action
    data = obj_creator(body)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 758, in create_port
    return self.post(self.ports_path, body=body)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 365, in post
    headers=headers, params=params)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 300, in do_request
    self._handle_fault_response(status_code, replybody, resp)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 275, in _handle_fault_response
    exception_handler_v20(status_code, error_body)
  File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 91, in exception_handler_v20
    request_ids=request_ids)
InternalServerError: An unknown exception occurred.
Neutron server returns request_ids: ['req-da57b32b-eac1-4e6f-8157-f328e5b5f91d']
root@blr-mx86-01:~#

attaching Contrail API logs from inside the controller container

On compute node
root@blr-mx86-02:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-lib 4.0.0.0-20 20
contrail-nodemgr 4.0.0.0-20 20
contrail-nova-vif 4.0.0.0-20 20
contrail-setup 4.0.0.0-20 20
contrail-utils 4.0.0.0-20 20
contrail-vrouter-agent 4.0.0.0-20 20
contrail-vrouter-common 4.0.0.0-20 20
contrail-vrouter-dkms 4.0.0.0-20 20
contrail-vrouter-dpdk 4.0.0.0-20 20
contrail-vrouter-init 4.0.0.0-20 20
contrail-vrouter-utils 4.0.0.0-20 20
nova-common 2:14.0.2-0ubuntu1~cloud0.1contrail 20
nova-compute 2:14.0.2-0ubuntu1~cloud0.1contrail 20
nova-compute-kvm 2:14.0.2-0ubuntu1~cloud0.1contrail 20
nova-compute-libvirt 2:14.0.2-0ubuntu1~cloud0.1contrail 20
python-contrail 4.0.0.0-20 20
python-contrail-vrouter-api 4.0.0.0-20 20
python-neutronclient 1:6.0.0-0ubuntu1~cloud0.2contrail 20
python-nova 2:14.0.2-0ubuntu1~cloud0.1contrail 20
python-opencontrail-vrouter-netns 4.0.0.0-20 20

root@blr-mx86-02:~# ip link show enp129s0f0
6: enp129s0f0: <BROADCAST,MULTICAST,ALLMULTI,PROMISC,UP,LOWER_UP> mtu 9192 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 90:e2:ba:c6:16:6c brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

root@blr-mx86-02:~# ethtool -i enp129s0f0
driver: ixgbe
version: 4.2.1-k
firmware-version: 0x61bd0001
expansion-rom-version:
bus-info: 0000:81:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

root@blr-mx86-02:~# grep enp129s0f0 /etc/nova/nova.conf
pci_passthrough_whitelist = { "devname": "enp129s0f0", "physical_network": "host2physnet1"}
root@blr-mx86-02:~#

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34440
Submitter: Vedamurthy Joshi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34443
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34462
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34443
Committed: http://github.com/Juniper/contrail-controller/commit/e48307a1df1db2a5c21c0d5c361255219dc82105
Submitter: Zuul (<email address hidden>)
Branch: master

commit e48307a1df1db2a5c21c0d5c361255219dc82105
Author: Sachin Bansal <email address hidden>
Date: Thu Aug 10 07:21:24 2017 -0700

portbinding values must be strings, not dict

We store portbinding value as string (either plain or json encoded).
If this is stored as dict, the client will not be able to parse it correctly.
We made this change earlier, but one of the bindings that is created internally
is still being set as string. With this code, it will be converted to json
string before writing to database.

Change-Id: I0aa0040bb76c8f4eecb7530c0a744a5bc31b5daf
Closes-Bug: 1709822

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34462
Committed: http://github.com/Juniper/contrail-controller/commit/dcd8863e13d4b80c9ea767cf7a7e94d75578ed41
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit dcd8863e13d4b80c9ea767cf7a7e94d75578ed41
Author: Sachin Bansal <email address hidden>
Date: Thu Aug 10 07:21:24 2017 -0700

portbinding values must be strings, not dict

We store portbinding value as string (either plain or json encoded).
If this is stored as dict, the client will not be able to parse it correctly.
We made this change earlier, but one of the bindings that is created internally
is still being set as string. With this code, it will be converted to json
string before writing to database.

Change-Id: I0aa0040bb76c8f4eecb7530c0a744a5bc31b5daf
Closes-Bug: 1709822

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/34501
Submitter: Sachin Bansal (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/34501
Committed: http://github.com/Juniper/contrail-controller/commit/7188cd991d2ec6f661f852fb5ee36938843758d7
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 7188cd991d2ec6f661f852fb5ee36938843758d7
Author: Sachin Bansal <email address hidden>
Date: Thu Aug 10 07:21:24 2017 -0700

portbinding values must be strings, not dict

We store portbinding value as string (either plain or json encoded).
If this is stored as dict, the client will not be able to parse it correctly.
We made this change earlier, but one of the bindings that is created internally
is still being set as string. With this code, it will be converted to json
string before writing to database.

Change-Id: I0aa0040bb76c8f4eecb7530c0a744a5bc31b5daf
Closes-Bug: 1709822
(cherry picked from commit dcd8863e13d4b80c9ea767cf7a7e94d75578ed41)