Contrail Ocata :: R4.0.1.0 build 19 :: Neutron unable to list, create VN.

[Ritam Gangopadhyay]

Setup:-

nodei19 10.204.217.131 openstack
nodei17 10.204.217.129 controller, analytics, analyticsdb
nodec28 10.204.217.13 controller, analytics, analyticsdb
nodec10 10.204.217.176 controller, analytics, analyticsdb
nodeg37 10.204.217.77 lb
nodei20 10.204.217.132 compute
nodec33 10.204.217.168 compute

Neutron net-list fails with following error and traceback in neutron server log

(nova-api)[nova@nodei19 /]$ neutron net-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-247bc189-946c-4ead-ac12-35ce8d58e7d0']
(nova-api)[nova@nodei19 /]$

2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource [req-247bc189-946c-4ead-ac12-35ce8d58e7d0 a7d57a1f36fd4b98876a9b504a17a700 963338afd6b243daa10c5a1b37ba99fa - - -] index failed: No details.
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource Traceback (most recent call last):
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 79, in resource
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource result = method(request=request, **args)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 95, in wrapped
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 131, in wrapped
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource traceback.format_exc())
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 368, in index
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource return self._items(request, True, parent_id)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 308, in _items
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource obj_list = obj_getter(request.context, **kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin_base.py", line 247, in get_networks
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource fields)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 332, in _list_resource
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource res_type, 'READALL')
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 206, in _request_backend
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource response = self._relay_request(url_path, data=data)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 199, in _relay_request
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource url, data=data, headers={'Content-type': 'application/json'})
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 187, in _request_api_server_authn
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource response = self._request_api_server(url, data, headers=authn_headers)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 150, in _request_api_server
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource response = requests.post(url, data=data, headers=headers)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/requests/api.py", line 110, in post
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource return request('post', url, data=data, json=json, **kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/requests/api.py", line 56, in request
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource return session.request(method=method, url=url, **kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource resp = self.send(prep, **send_kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource r = adapter.send(request, **kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 487, in send
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource raise ConnectionError(e, request=request)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource ConnectionError: HTTPConnectionPool(host='192.168.100.12', port=8082): Max retries exceeded with url: /neutron/network (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f8e1d191490>: Failed to establish a new connection: [Errno 113] EHOSTUNREACH',))
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource
2017-08-28 15:32:38.741 20 INFO neutron.wsgi [req-247bc189-946c-4ead-ac12-35ce8d58e7d0 a7d57a1f36fd4b98876a9b504a17a700 963338afd6b243daa10c5a1b37ba99fa - - -] 10.204.217.131 - - [28/Aug/2017 15:32:38] "GET /v2.0/networks.json HTTP/1.1" 500 368 3.545291

[Ritam Gangopadhyay (ritam)]

Using openstack cli also hits the same neutron server traceback

(keystone)[root@nodei19 /]# openstack network list
HttpException: Internal Server Error, Request Failed: internal server error while processing your request.
(keystone)[root@nodei19 /]#

[Ramprakash R (ramprakash)]

open stack commands need to be executed from the base host. Based on Ritam's followup email, the commands worked from the host. Closing this based on this email snippet:

%<-----------------------------

Hi,

        The issues seen prior to R4.0.1.0 build 21 on a setup with control data interface separation were:-

1. Multiple patches to complete provisioning.
2. Neutron unable to list/create networks https://bugs.launchpad.net/juniperopenstack/+bug/1713453
3. Openstack compute provisioning failure https://bugs.launchpad.net/juniperopenstack/+bug/1713559
4. Horizon not reachable https://bugs.launchpad.net/juniperopenstack/+bug/1713444

       With R4.0.1.0 build 21 and a setup with “NO” control data interface separation (i.e. only management interface) and the following 2 patches
                https://review.opencontrail.org/#/c/35015/1/kolla-ansible/ansible/roles/nova/defaults/main.yml
                https://review.opencontrail.org/#/c/35013/1/src/client/preconfig.py

1. 0 touch provisioning works, just need to execute provision_containers.sh with proper json file and sku option
2. Neutron net-list, openstack net list, and create delete VN works fine.
3. Horizon works but has a upstream bug - https://bugs.launchpad.net/bugs/1672741 and needs to be patched with https://ask.openstack.org/en/question/107956/kolla-missing-js-and-css-in-horizon/ for it to work.
4. VN – VM creation from horizon and cli works and intra VN connectivity is fine. Inter VN connectivity, was planning to check through policy.

------------------------------>%

[Ritam Gangopadhyay (ritam)]

This doesn't work when neutron tries to connect over the control-data interface. The bug was filled in context of management and control-data plane separation.

2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource ConnectionError: HTTPConnectionPool(host='192.168.100.12', port=8082): Max retries exceeded with url: /neutron/network (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f8e1d191490>: Failed to establish a new connection: [Errno 113] EHOSTUNREACH',))

On single interface setup this works when neutron connects over the management interface, but the bug still holds good for setup with cntl-data interface.

[Ramprakash R (ramprakash)]

Please check if this works with something similar to the below config (cluster.json):

{
"cluster" : [
    {
        "id": "seven_node_kolla",
        "parameters" : {
            "domain": "sm-domain.com",
            "subnet_mask": "255.255.255.0",
            "gateway": "192.168.121.1",
            "provision": {
                "contrail": {
                    "ha": {
                        "contrail_internal_vip": "192.168.10.24",
                        "contrail_external_vip": "192.168.1.24"
                    },
                    "kernel_upgrade": false
                },
                "openstack":{
                    "openstack_manage_amqp": true,
                    "region": "RegionOne",
                    "multi_tenancy": true,
                    "openstack_manage_amqp": true,
                    "enable_ceilometer": false,
                    "keystone" : {
                        "mysql_service_password": "c0ntrail123",
                        "admin_user": "admin",
                        "admin_token": "c0ntrail123",
                        "ip": "192.168.10.100",
                        "auth_port": 35357,
                        "auth_protocol": "http",
                        "version": "v2.0",
                        "admin_password": "c0ntrail123",
                        "admin_tenant": "admin"
                    },
                    "ha": {
                        "internal_vip": "192.168.10.100",
                        "external_vip": "192.168.1.100"
                    }
                },
                "contrail_4": {
                    "rabbitmq_config": {
                        "user": "openstack",
                        "password": "c0ntrail123"
                    },
                    "global_config": {
                        "external_rabbitmq_servers": "192.168.10.25"
                    }
                },
                "kolla_globals": {
                    "enable_nova_compute": "yes",
                    "keystone_admin_user": "admin",
                    "keystone_admin_url": "{{ admin_protocol }}://{{ kolla_internal_fqdn }}:{{ keystone_admin_port }}",
                    "keystone_internal_url": "{{ internal_protocol }}://{{ kolla_internal_fqdn }}:{{ keystone_public_port }}",
                    "keystone_public_url": "{{ public_protocol }}://{{ kolla_external_fqdn }}:{{ keystone_public_port }}",
                    "kolla_base_distro": "ubuntu",
                    "openstack_release": "4.0.0",

                    "kolla_internal_vip_address": "192.168.10.100",
                    "network_interface": "eth2",
                    "kolla_external_vip_address": "192.168.1.100",
                    "kolla_external_vip_interface": "eth1",

                    "haproxy_password": "c0ntrail123",
                    "keepalived_password": "c0ntrail123",
                    "neutron_external_interface": "eth0",
                    "neutron_plugin_agent": "opencontrail",
                    "enable_neutron_opencontrail": "yes",
                    "contrail_api_interface_address": "192.168.10.24"
                }
            }
       }
    }
]
}

[Ramprakash R (ramprakash)]

Moving it to "incomplete" for Ritam to verify. Please reopen if there is still issue with neutron connectivity.