networking-sfc

port pair creation fails in create_port_pair_postcommit on kolla deployment

Bug #1724444 reported by vikrant on 2017-10-18

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	networking-sfc	New	Undecided	Unassigned

Bug Description

I have followed the blog [1] to configure the neutron-sfc using kolla deployment tool. After deployment I am able to successfully create an instance without any issue, but while creating neutron SFC port pair following [2] I am getting following error:

While creating sfc port pair group.

~~~
[root@controller-1 ~]# openstack sfc port pair create --ingress P1 --egress P2 PPAIR
create_port_pair_postcommit failed.
Neutron server returns request_ids: ['req-5e062f5c-2860-422c-964d-af191f6b4c4d']
~~~

Following call trace is reported in neutron-server container.

~~~
2017-10-18 04:13:36.505 23 ERROR networking_sfc.services.sfc.plugin [req-5e062f5c-2860-422c-964d-af191f6b4c4d 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] Cr
eate port pair failed, deleting port_pair 'e66b3968-7d8a-4fd7-848b-563fc5a2e248': SfcDriverError: create_port_pair_postcommit failed.
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource [req-5e062f5c-2860-422c-964d-af191f6b4c4d 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] create failed
: No details.: SfcDriverError: create_port_pair_postcommit failed.
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource Traceback (most recent call last):
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 98, in resource
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource result = method(request=request, **args)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 435, in create
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 92, in wrapped
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource self.force_reraise()
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 88, in wrapped
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource self.force_reraise()
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 127, in wrapped
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource self.force_reraise()
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 548, in _create
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource obj = do_create(body)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 530, in do_create
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource request.context, reservation.reservation_id)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource self.force_reraise()
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 523, in do_create
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource return obj_creator(request.context, **kwargs)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource return method(*args, **kwargs)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/plugin.py", line 126, in create_port_pair
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource self.delete_port_pair(context, portpair_db['id'])
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource self.force_reraise()
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/plugin.py", line 119, in create_port_pair
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource self.driver_manager.create_port_pair_postcommit(portpair_context)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/driver_manager.py", line 142, in create_port_pair_postcommit
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource self._call_drivers("create_port_pair_postcommit", context)
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/driver_manager.py", line 113, in _call_drivers
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource method=method_name
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource SfcDriverError: create_port_pair_postcommit failed.
2017-10-18 04:13:36.721 23 ERROR neutron.api.v2.resource
2017-10-18 04:13:36.722 23 INFO neutron.wsgi [req-5e062f5c-2860-422c-964d-af191f6b4c4d 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] 192.168.133.254,192.168.133.100 "POST /v2.0/sfc/port_pairs HTTP/1.1" status: 500 len: 302 time: 1.2051969
~~~

[1] http://networkop.co.uk/blog/2017/09/08/os-lab-docker/
[2] http://networkop.co.uk/blog/2017/09/15/os-sfc-skydive/

Regards,
Vikrant Aggarwal

Tags:

Revision history for this message

vikrant (ervikrant06) wrote on 2017-10-18:

From inside docker container.

~~~
(neutron-server)[neutron@controller-1 /]$ neutron --version
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
6.5.0
~~~

Boden R (boden) on 2017-10-18

tags:

added: sfc

Revision history for this message

Bernard Cafarelli (bcafarel) wrote on 2017-10-18:

All of this happens on the server (creating port pairs does not involve the agent), so I'd guess a configuration problem?

Let's check configuration first, what are the values for service_plugins, [sfc] section, [flowclassifier] section on the neutron-server? (should be in neutron.conf)
Also a more verbose log level may help.

Bernard Cafarelli (bcafarel) on 2017-10-18

summary:

- neutron sfc port group pair creation failed
+ networking-sfc port pair creation fails in create_port_pair_postcommit
+ on kolla deployment

Revision history for this message

vikrant (ervikrant06) wrote on 2017-10-18: Re: networking-sfc port pair creation fails in create_port_pair_postcommit on kolla deployment

Download full text (30.4 KiB)

Here is the configuration file from neutron-server container.

~~~
(neutron-server)[neutron@controller-1 /]$ egrep -v "^(#|$)" /etc/neutron/neutron.conf
[DEFAULT]
debug = False
log_dir = /var/log/kolla/neutron
use_stderr = False
bind_host = 192.168.133.100
bind_port = 9696
api_paste_config = /usr/share/neutron/api-paste.ini
endpoint_type = internalURL
api_workers = 1
metadata_workers = 1
rpc_workers = 1
rpc_state_report_workers = 1
metadata_proxy_socket = /var/lib/neutron/kolla/metadata_proxy
interface_driver = openvswitch
allow_overlapping_ips = true
core_plugin = ml2
service_plugins = flow_classifier,sfc,odl-router_v2
transport_url = rabbit://openstack:k6cxU2zB4aufroLCbcfwDGZk3hHZwENNSIgDMugT@192.168.133.100:5672
[nova]
auth_url = http://192.168.133.254:35357
auth_type = password
project_domain_id = default
user_domain_id = default
region_name = RegionOne
project_name = service
username = nova
password = S61ZNKMtDEjtwuE1lmKv5UeRzRktUtkJZv4OXco7
endpoint_type = internal
[oslo_concurrency]
lock_path = /var/lib/neutron/tmp
[agent]
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf
[database]
connection = mysql+pymysql://neutron:hCrMYKjHyIhbXtS5Gv9tEps5AYXxwWe4AOZUTZYI@192.168.133.254:3306/neutron
max_retries = -1
[keystone_authtoken]
auth_uri = http://192.168.133.254:5000
auth_url = http://192.168.133.254:35357
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = neutron
password = WplkI8SwtHkjWVpoZZ4Uum4svmB0FbHNYuzJEP4v
memcache_security_strategy = ENCRYPT
memcache_secret_key = Al2ocKLr194CSJL14fRZbmhCs3Ty54dbDBD9eMzh
memcached_servers = 192.168.133.100:11211
[oslo_messaging_notifications]
driver = noop
[sfc]
drivers = ovs
[flowclassifier]
drivers = ovs
[placement]
auth_type = password
auth_url = http://192.168.133.254:35357
username = placement
password = BOiwJ43AjggaciqsfaVZsXMlEFxswhDO71x0FLnZ
user_domain_name = Default
project_name = service
project_domain_name = Default
os_region_name = RegionOne
os_interface = internal
~~~

Added "--debug" while creating sfc port pair.

Here is the configuration file from neutron-server container.

Added "--debug" while creating sfc port pair.

~~~
[root@controller-1 ~]# openstack --debug sfc port pair create --ingress P1 --egress P2 PPAIR
START with options: [u'--debug', u'sfc', u'port', u'pair', u'create', u'--ingress', u'P1', u'--egress', u'P2', u'PPAIR']
options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', auth_type='', auth_url='http://192.168.133.254:35357/v3', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', insecure=None, interface='internal', key='', log_file=None, openid_scope='', os_beta_command=False, os_compute_api_version='', os_identity_api_version='3', os_image_api_version='', os_network_api_version='', os_object_api_version='', os_project_id=None, os_project_name=None, os_volume_api_version='', passcode='', password='***', project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='RegionOne', timing=False, token='***', trust_id='', url='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.133.254:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': u'1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'debug': True, u'interface': 'internal', u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'container_infra_api_version': u'1', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.133.254:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': u'1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'debug': True, u'interface': 'internal', u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 2, cmd group openstack.volume.v2
identity API version 3, cmd group openstack.identity.v3
object_store API version 1, cmd group openstack.object_store.v1
neutronclient API version 2, cmd group openstack.neutronclient.v2
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.133.254:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': u'1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'debug': True, u'interface': 'internal', u'disable_vendor_agent': {}}
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.133.254:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': u'1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'debug': True, u'interface': 'internal', u'disable_vendor_agent': {}}
command: sfc port pair create -> neutronclient.osc.v2.sfc.sfc_port_pair.CreateSfcPortPair (auth=True)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.133.254:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': u'2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': u'1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': '***', u'application_catalog_api_version': u'1', 'cacert': None, u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'debug': True, u'interface': 'internal', u'disable_vendor_agent': {}}
Using auth plugin: password
Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'http://192.168.133.254:35357/v3', 'password': '***', 'project_domain_name': 'Default'}
Get auth_ref
REQ: curl -g -i -X GET http://192.168.133.254:35357/v3 -H "Accept: application/json" -H "User-Agent: osc-lib/1.7.0 keystoneauth1/3.2.0 python-requests/2.18.4 CPython/2.7.5"
Starting new HTTP connection (1): 192.168.133.254
http://192.168.133.254:35357 "GET /v3 HTTP/1.1" 200 255
RESP: [200] Date: Wed, 18 Oct 2017 14:24:29 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 Vary: X-Auth-Token x-openstack-request-id: req-73235c01-159c-49d6-be7e-3a7e4c961299 Content-Length: 255 Content-Type: application/json
RESP BODY: {"version": {"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "http://192.168.133.254:35357/v3/", "rel": "self"}]}}

GET call to None for http://192.168.133.254:35357/v3 used request id req-73235c01-159c-49d6-be7e-3a7e4c961299
Making authentication request to http://192.168.133.254:35357/v3/auth/tokens
http://192.168.133.254:35357 "POST /v3/auth/tokens HTTP/1.1" 201 5451
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "0a7dac36c83f4097aee7ec10d0366dd6", "name": "admin"}, {"id": "4f9dc84898ab44338842b9e333948081", "name": "heat_stack_owner"}], "expires_at": "2017-10-18T15:24:30.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "766c772012ff4c4ca6e563accd51f1ea", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.133.254:8774/v2.1/766c772012ff4c4ca6e563accd51f1ea", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "62a3f516f266461bbdc71721a7c8ff00"}, {"url": "http://192.168.133.254:8774/v2.1/766c772012ff4c4ca6e563accd51f1ea", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "71ba9a6cf3bf47bbaf74fab4ab3c084c"}, {"url": "http://192.168.133.254:8774/v2.1/766c772012ff4c4ca6e563accd51f1ea", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f05462b571a4480a95bad545bc35615a"}], "type": "compute", "id": "26e789c0293e43bbb8d3cf0a25e60059", "name": "nova"}, {"endpoints": [{"url": "http://192.168.133.254:8004/v1/766c772012ff4c4ca6e563accd51f1ea", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2019eec8cb474956bc1a200c3e688d1e"}, {"url": "http://192.168.133.254:8004/v1/766c772012ff4c4ca6e563accd51f1ea", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "2449d0e2ce6040989003dd0e4c047d49"}, {"url": "http://192.168.133.254:8004/v1/766c772012ff4c4ca6e563accd51f1ea", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "d37622da48a74c3794fb8c65e06d99b8"}], "type": "orchestration", "id": "73474176451a4dc89f4b177ff9b4b627", "name": "heat"}, {"endpoints": [{"url": "http://192.168.133.254:5000", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "02198b870879482dab82a7dbb064158f"}, {"url": "http://192.168.133.254:5000", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "0575292f8a3a41f5bc1b93256fb5c870"}, {"url": "http://192.168.133.254:35357", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "1f84a62366ea4bd19db273d945c10c88"}], "type": "identity", "id": "865c31c15b8742b386afd3d219886849", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.133.254:9292", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2f62b8d71d9a464d9bbbb6a85b5245ea"}, {"url": "http://192.168.133.254:9292", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "f787498c35454a9b81a3355de811e48c"}, {"url": "http://192.168.133.254:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "f7a94732f3004ba4bd68aa4c69dba021"}], "type": "image", "id": "8ccef408cf584ca1b993742bdb0da1af", "name": "glance"}, {"endpoints": [{"url": "http://192.168.133.254:8780", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "295e46096a8142ebaba2725a8feb48a0"}, {"url": "http://192.168.133.254:8780", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "502f0e3ff7054f36b48d1f2b99e729e0"}, {"url": "http://192.168.133.254:8780", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "ded82de78d8c498fbe7e3094ac7c6094"}], "type": "placement", "id": "9fd058fe2ef343d883bba862a6916f3e", "name": "placement"}, {"endpoints": [{"url": "http://192.168.133.254:8774/v2/766c772012ff4c4ca6e563accd51f1ea", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "2a2cd8712881484cb61e8f02389f1e01"}, {"url": "http://192.168.133.254:8774/v2/766c772012ff4c4ca6e563accd51f1ea", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "74afcb1e7b494f67818630093d970f7a"}, {"url": "http://192.168.133.254:8774/v2/766c772012ff4c4ca6e563accd51f1ea", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "fbdaf30b350749cbb7ad8787e9c6bc57"}], "type": "compute_legacy", "id": "be37407ee1ba4cc3bf2759b4d0e18059", "name": "nova_legacy"}, {"endpoints": [{"url": "http://192.168.133.254:8000/v1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "b9029233a67c46da9090b78dc5c419e2"}, {"url": "http://192.168.133.254:8000/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "e7237d287f2f40ce85962ba769b65b49"}, {"url": "http://192.168.133.254:8000/v1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f407f537801f46c6940ce0cff61482b9"}], "type": "cloudformation", "id": "c70a9549c0684f98a63f5b1d707abb15", "name": "heat-cfn"}, {"endpoints": [{"url": "http://192.168.133.254:9696", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "12b31510f1374e118dac0284b8cf7606"}, {"url": "http://192.168.133.254:9696", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "bd2ddf062f9b4e558165e8078c16a0de"}, {"url": "http://192.168.133.254:9696", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "fdb3d451ecd741f68bd1faffd92d53ad"}], "type": "network", "id": "fa4f39bb92e345f7916acbc9699b0db1", "name": "neutron"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "0efeac25b8a845c799800fa87d850024"}, "audit_ids": ["fynRhpuFRh2kcUXwK9DjOA"], "issued_at": "2017-10-18T14:24:30.000000Z"}}
run(Namespace(columns=[], description=None, egress=u'P2', fit_width=False, formatter='table', ingress=u'P1', max_width=0, name=u'PPAIR', noindent=False, prefix='', print_empty=False, service_function_parameters=None, variables=[]))
Instantiating neutron client: <class 'neutronclient.v2_0.client.Client'>
Making authentication request to http://192.168.133.254:35357/v3/auth/tokens
http://192.168.133.254:35357 "POST /v3/auth/tokens HTTP/1.1" 201 5451
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "0a7dac36c83f4097aee7ec10d0366dd6", "name": "admin"}, {"id": "4f9dc84898ab44338842b9e333948081", "name": "heat_stack_owner"}], "expires_at": "2017-10-18T15:24:31.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "766c772012ff4c4ca6e563accd51f1ea", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.133.254:8774/v2.1/766c772012ff4c4ca6e563accd51f1ea", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "62a3f516f266461bbdc71721a7c8ff00"}, {"url": "http://192.168.133.254:8774/v2.1/766c772012ff4c4ca6e563accd51f1ea", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "71ba9a6cf3bf47bbaf74fab4ab3c084c"}, {"url": "http://192.168.133.254:8774/v2.1/766c772012ff4c4ca6e563accd51f1ea", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f05462b571a4480a95bad545bc35615a"}], "type": "compute", "id": "26e789c0293e43bbb8d3cf0a25e60059", "name": "nova"}, {"endpoints": [{"url": "http://192.168.133.254:8004/v1/766c772012ff4c4ca6e563accd51f1ea", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2019eec8cb474956bc1a200c3e688d1e"}, {"url": "http://192.168.133.254:8004/v1/766c772012ff4c4ca6e563accd51f1ea", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "2449d0e2ce6040989003dd0e4c047d49"}, {"url": "http://192.168.133.254:8004/v1/766c772012ff4c4ca6e563accd51f1ea", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "d37622da48a74c3794fb8c65e06d99b8"}], "type": "orchestration", "id": "73474176451a4dc89f4b177ff9b4b627", "name": "heat"}, {"endpoints": [{"url": "http://192.168.133.254:5000", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "02198b870879482dab82a7dbb064158f"}, {"url": "http://192.168.133.254:5000", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "0575292f8a3a41f5bc1b93256fb5c870"}, {"url": "http://192.168.133.254:35357", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "1f84a62366ea4bd19db273d945c10c88"}], "type": "identity", "id": "865c31c15b8742b386afd3d219886849", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.133.254:9292", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2f62b8d71d9a464d9bbbb6a85b5245ea"}, {"url": "http://192.168.133.254:9292", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "f787498c35454a9b81a3355de811e48c"}, {"url": "http://192.168.133.254:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "f7a94732f3004ba4bd68aa4c69dba021"}], "type": "image", "id": "8ccef408cf584ca1b993742bdb0da1af", "name": "glance"}, {"endpoints": [{"url": "http://192.168.133.254:8780", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "295e46096a8142ebaba2725a8feb48a0"}, {"url": "http://192.168.133.254:8780", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "502f0e3ff7054f36b48d1f2b99e729e0"}, {"url": "http://192.168.133.254:8780", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "ded82de78d8c498fbe7e3094ac7c6094"}], "type": "placement", "id": "9fd058fe2ef343d883bba862a6916f3e", "name": "placement"}, {"endpoints": [{"url": "http://192.168.133.254:8774/v2/766c772012ff4c4ca6e563accd51f1ea", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "2a2cd8712881484cb61e8f02389f1e01"}, {"url": "http://192.168.133.254:8774/v2/766c772012ff4c4ca6e563accd51f1ea", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "74afcb1e7b494f67818630093d970f7a"}, {"url": "http://192.168.133.254:8774/v2/766c772012ff4c4ca6e563accd51f1ea", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "fbdaf30b350749cbb7ad8787e9c6bc57"}], "type": "compute_legacy", "id": "be37407ee1ba4cc3bf2759b4d0e18059", "name": "nova_legacy"}, {"endpoints": [{"url": "http://192.168.133.254:8000/v1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "b9029233a67c46da9090b78dc5c419e2"}, {"url": "http://192.168.133.254:8000/v1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "e7237d287f2f40ce85962ba769b65b49"}, {"url": "http://192.168.133.254:8000/v1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f407f537801f46c6940ce0cff61482b9"}], "type": "cloudformation", "id": "c70a9549c0684f98a63f5b1d707abb15", "name": "heat-cfn"}, {"endpoints": [{"url": "http://192.168.133.254:9696", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "12b31510f1374e118dac0284b8cf7606"}, {"url": "http://192.168.133.254:9696", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "bd2ddf062f9b4e558165e8078c16a0de"}, {"url": "http://192.168.133.254:9696", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "fdb3d451ecd741f68bd1faffd92d53ad"}], "type": "network", "id": "fa4f39bb92e345f7916acbc9699b0db1", "name": "neutron"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "0efeac25b8a845c799800fa87d850024"}, "audit_ids": ["q31exfoFQr6doBEzcflDJQ"], "issued_at": "2017-10-18T14:24:31.000000Z"}}
REQ: curl -g -i -X GET http://192.168.133.254:9696/v2.0/ports?name=P1 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}78617636ee411ad63fab10b8f5188d18ff170b95"
Starting new HTTP connection (1): 192.168.133.254
http://192.168.133.254:9696 "GET /v2.0/ports?name=P1 HTTP/1.1" 200 848
RESP: [200] Content-Type: application/json Content-Length: 848 X-Openstack-Request-Id: req-54069fef-de0f-4bdf-ad53-fe3a102276fd Date: Wed, 18 Oct 2017 14:24:31 GMT
RESP BODY: {"ports":[{"allowed_address_pairs":[],"extra_dhcp_opts":[],"updated_at":"2017-10-13T10:57:54Z","device_owner":"compute:nova","revision_number":6,"port_security_enabled":true,"binding:profile":{},"fixed_ips":[{"subnet_id":"08bbc3a6-4fe0-4ca2-bc03-1425a69b53f6","ip_address":"10.0.0.14"}],"id":"49ba9fd8-5840-4ab3-b9f7-96fc59225b37","security_groups":["c1e0cdb7-c0b4-4023-9c3a-64508724793f"],"binding:vif_details":{},"binding:vif_type":"ovs","mac_address":"fa:16:3e:19:35:45","project_id":"766c772012ff4c4ca6e563accd51f1ea","status":"ACTIVE","binding:host_id":"compute-1","description":"","tags":[],"device_id":"1a59573b-b557-488f-917c-536a2fd21f35","name":"P1","admin_state_up":true,"network_id":"b30ef600-7c8e-4922-892f-571e9c22941a","tenant_id":"766c772012ff4c4ca6e563accd51f1ea","created_at":"2017-10-13T10:57:21Z","binding:vnic_type":"normal"}]}

GET call to network for http://192.168.133.254:9696/v2.0/ports?name=P1 used request id req-54069fef-de0f-4bdf-ad53-fe3a102276fd
REQ: curl -g -i -X GET http://192.168.133.254:9696/v2.0/ports?name=P2 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}78617636ee411ad63fab10b8f5188d18ff170b95"
http://192.168.133.254:9696 "GET /v2.0/ports?name=P2 HTTP/1.1" 200 848
RESP: [200] Content-Type: application/json Content-Length: 848 X-Openstack-Request-Id: req-2240e0cd-5dd6-491e-a310-5e1702dfbebc Date: Wed, 18 Oct 2017 14:24:31 GMT
RESP BODY: {"ports":[{"allowed_address_pairs":[],"extra_dhcp_opts":[],"updated_at":"2017-10-13T10:57:55Z","device_owner":"compute:nova","revision_number":6,"port_security_enabled":true,"binding:profile":{},"fixed_ips":[{"subnet_id":"08bbc3a6-4fe0-4ca2-bc03-1425a69b53f6","ip_address":"10.0.0.12"}],"id":"8f83b61d-3057-4d1c-a075-340271f845ee","security_groups":["c1e0cdb7-c0b4-4023-9c3a-64508724793f"],"binding:vif_details":{},"binding:vif_type":"ovs","mac_address":"fa:16:3e:44:20:29","project_id":"766c772012ff4c4ca6e563accd51f1ea","status":"ACTIVE","binding:host_id":"compute-1","description":"","tags":[],"device_id":"1a59573b-b557-488f-917c-536a2fd21f35","name":"P2","admin_state_up":true,"network_id":"b30ef600-7c8e-4922-892f-571e9c22941a","tenant_id":"766c772012ff4c4ca6e563accd51f1ea","created_at":"2017-10-13T10:57:28Z","binding:vnic_type":"normal"}]}

GET call to network for http://192.168.133.254:9696/v2.0/ports?name=P2 used request id req-2240e0cd-5dd6-491e-a310-5e1702dfbebc
REQ: curl -g -i -X POST http://192.168.133.254:9696/v2.0/sfc/port_pairs -H "User-Agent: python-neutronclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}78617636ee411ad63fab10b8f5188d18ff170b95" -d '{"port_pair": {"ingress": "49ba9fd8-5840-4ab3-b9f7-96fc59225b37", "egress": "8f83b61d-3057-4d1c-a075-340271f845ee", "service_function_parameters": null, "name": "PPAIR"}}'
http://192.168.133.254:9696 "POST /v2.0/sfc/port_pairs HTTP/1.1" 500 108
RESP: [500] Content-Type: application/json Content-Length: 108 X-Openstack-Request-Id: req-532290cf-3153-4170-8e6b-9beecee2ee93 Date: Wed, 18 Oct 2017 14:24:33 GMT
RESP BODY: {"NeutronError": {"message": "create_port_pair_postcommit failed.", "type": "SfcDriverError", "detail": ""}}

POST call to network for http://192.168.133.254:9696/v2.0/sfc/port_pairs used request id req-532290cf-3153-4170-8e6b-9beecee2ee93
Error message: {"NeutronError": {"message": "create_port_pair_postcommit failed.", "type": "SfcDriverError", "detail": ""}}
create_port_pair_postcommit failed.
Neutron server returns request_ids: ['req-532290cf-3153-4170-8e6b-9beecee2ee93']
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/display.py", line 115, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/osc/v2/sfc/sfc_port_pair.py", line 82, in take_action
    obj = client.create_port_pair(body)[resource]
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 2176, in create_port_pair
    return self.post(self.sfc_port_pairs_path, body=body)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 357, in post
    headers=headers, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 292, in do_request
    self._handle_fault_response(status_code, replybody, resp)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 268, in _handle_fault_response
    exception_handler_v20(status_code, error_body)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20
    request_ids=request_ids)
InternalServerError: create_port_pair_postcommit failed.
Neutron server returns request_ids: ['req-532290cf-3153-4170-8e6b-9beecee2ee93']
clean_up CreateSfcPortPair: create_port_pair_postcommit failed.
Neutron server returns request_ids: ['req-532290cf-3153-4170-8e6b-9beecee2ee93']
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/display.py", line 115, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/neutronclient/osc/v2/sfc/sfc_port_pair.py", line 82, in take_action
    obj = client.create_port_pair(body)[resource]
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 2176, in create_port_pair
    return self.post(self.sfc_port_pairs_path, body=body)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 357, in post
    headers=headers, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 292, in do_request
    self._handle_fault_response(status_code, replybody, resp)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 268, in _handle_fault_response
    exception_handler_v20(status_code, error_body)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20
    request_ids=request_ids)
InternalServerError: create_port_pair_postcommit failed.
Neutron server returns request_ids: ['req-532290cf-3153-4170-8e6b-9beecee2ee93']

END return value: 1
~~~

Revision history for this message

Bernard Cafarelli (bcafarel) wrote on 2017-10-18:

Thanks, the configuration looks OK (I was wondering if the ovs driver for SFC was correctly set).
In that case failure in port pair creation probably means some database problems, what is the result of:
neutron-db-manage --subproject networking-sfc current --verbose

Revision history for this message

vikrant (ervikrant06) wrote on 2017-10-18:

Thanks for looking into the issue.

Here is the output:

(neutron-server)[neutron@controller-1 /]$ neutron-db-manage --subproject networking-sfc current --verbose
Running current for networking-sfc ...
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
Current revision(s) for mysql+pymysql://neutron:XXXXX@192.168.133.254:3306/neutron:
Rev: 06382790fb2c (head)
Parent: 010308b06b49
Branch names: contract
Path: /var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/db/migration/alembic_migrations/versions/newton/contract/06382790fb2c_fix_foreign_constraints.py

fix foreign constraints

Revision ID: 06382790fb2c
Create Date: 2016-08-11 14:45:34.416120

Rev: 53ed5bec6cff (head)
Parent: 8329e9be2d8a
Branch names: expand
Path: /var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/db/migration/alembic_migrations/versions/queens/expand/53ed5bec6cff_add_service_graph_api_resource.py

Add Service Graph API resource

    Revision ID: 53ed5bec6cff
    Revises: 8329e9be2d8a
    Create Date: 2017-05-24 00:00:00.000000

Revision history for this message

Bernard Cafarelli (bcafarel) wrote on 2017-10-18:

Thanks, reassigning to networking-sfc for further investigation, as both conf and database look good

affects:

neutron → networking-sfc

Bernard Cafarelli (bcafarel) on 2017-10-18

summary:

- networking-sfc port pair creation fails in create_port_pair_postcommit
- on kolla deployment
+ port pair creation fails in create_port_pair_postcommit on kolla
+ deployment

Revision history for this message

Bernard Cafarelli (bcafarel) wrote on 2017-10-20:

I am not sure I will have time to test a kolla deployment soon, in the meantime can you paste the debug neutron-server.log?

Setting "debug=True" in neutron.conf default section and restart the service should be enough. Hopefully there will be more indications there

Revision history for this message

vikrant (ervikrant06) wrote on 2017-10-27:

Download full text (43.3 KiB)

Sorry for late response, here is debug log :

2017-10-27 10:20:56.839 25 DEBUG neutron_lib.callbacks.manager [req-b78f1283-5414-4ece-acaa-b9db6cb70770 - - - - -] Notify callbacks ['neutron.services.segments.db._update_segment_host_mapping_for_agent-7394263'] for agent, after_update _notify_loop /var/lib/kolla/venv/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2017-10-27 10:20:57.414 24 DEBUG neutron_lib.callbacks.manager [req-18c58833-6461-4092-9b81-4ac4206800b9 - - - - -] Notify callbacks ['neutron.services.segments.db._update_segment_host_mapping_for_agent-7394263'] for agent, after_update _notify_loop /var/lib/kolla/venv/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2017-10-27 10:21:01.359 23 DEBUG neutron.wsgi [-] (23) accepted ('192.168.133.100', 55964) server /var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py:883
2017-10-27 10:21:02.145 23 INFO neutron.wsgi [req-91a49273-34fa-4658-ae4b-b0de7ff8a25f 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] 192.168.133.254,192.168.133.100 "GET /v2.0/ports?name=P1 HTTP/1.1" status: 200  len: 1023 time: 0.7852690
2017-10-27 10:21:02.261 23 INFO neutron.wsgi [req-521faae1-aca6-4a10-aa16-8b79efa0196d 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] 192.168.133.254,192.168.133.100 "GET /v2.0/ports?name=P2 HTTP/1.1" status: 200  len: 1023 time: 0.1072490
2017-10-27 10:21:02.282 23 DEBUG neutron.api.v2.base [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] Request body: {u'port_pair': {u'ingress': u'49ba9fd8-5840-4ab3-b9f7-96fc59225b37', u'egress': u'8f83b61d-3057-4d1c-a075-340271f845ee', u'service_function_parameters': None, u'name': u'PPAIR'}} prepare_request_body /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py:695
2017-10-27 10:21:02.283 23 INFO neutron.quota [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] Loaded quota_driver: <neutron.db.quota.driver.DbQuotaDriver object at 0x9434d90>.
2017-10-27 10:21:02.298 23 DEBUG neutron.db.quota.driver [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] Resources subnetpool,service_graph have unlimited quota limit. It is not required to calculate headroom  make_reservation /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/quota/driver.py:223
2017-10-27 10:21:02.317 23 DEBUG neutron.db.quota.driver [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] Attempting to reserve 1 items for resource port_pair. Total usage: 0; quota limit: 100; headroom:100 make_reservation /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/quota/driver.py:255
2017-10-27 10:21:02.374 23 DEBUG networking_sfc.services.sfc.plugin [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] networking_sfc.services.sfc.plugin.SfcPlugin method create_port_pair called with arguments (<neutron_lib.context.Context object at 0x94340d0>,) {'port_pair': {u'port_pair': {u'ingress': u'49ba9fd8-5840-4ab3-b9f7-96fc59225b37', u'name': u'PPAIR', 'tenant_id': u'766c772012ff4c4ca6e563accd51f1ea', u'egress': u'8f83b61d-3057-4d1c-a075-340271f845ee', u'service_function_parameters': {'weight': 1, 'correlation': None}, 'project_id': u'766c772012ff4c4ca6e563accd51f1ea', 'description': ''}}} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66
2017-10-27 10:21:02.375 23 DEBUG networking_sfc.db.sfc_db [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] networking_sfc.services.sfc.plugin.SfcPlugin method create_port_pair called with arguments (<neutron_lib.context.Context object at 0x94340d0>, {u'port_pair': {u'ingress': u'49ba9fd8-5840-4ab3-b9f7-96fc59225b37', u'name': u'PPAIR', 'tenant_id': u'766c772012ff4c4ca6e563accd51f1ea', u'egress': u'8f83b61d-3057-4d1c-a075-340271f845ee', u'service_function_parameters': {'weight': 1, 'correlation': None}, 'project_id': u'766c772012ff4c4ca6e563accd51f1ea', 'description': ''}}) {} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66
2017-10-27 10:21:02.741 23 DEBUG networking_sfc.services.sfc.drivers.ovs.driver [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] networking_sfc.services.sfc.drivers.ovs.driver.OVSSfcDriver method create_port_pair called with arguments (<networking_sfc.services.sfc.common.context.PortPairContext object at 0x9454510>,) {} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66
2017-10-27 10:21:02.742 23 DEBUG networking_sfc.services.sfc.drivers.ovs.driver [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] networking_sfc.services.sfc.drivers.ovs.driver.OVSSfcDriver method _create_port_pair_detail called with arguments ({'ingress': u'49ba9fd8-5840-4ab3-b9f7-96fc59225b37', 'description': '', 'tenant_id': u'766c772012ff4c4ca6e563accd51f1ea', 'egress': u'8f83b61d-3057-4d1c-a075-340271f845ee', 'service_function_parameters': {'weight': 1, 'correlation': None}, 'project_id': u'766c772012ff4c4ca6e563accd51f1ea', 'id': '357b986b-e25f-43e5-8632-4ccf8d41a506', 'name': u'PPAIR'},) {} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66
2017-10-27 10:21:02.742 23 DEBUG networking_sfc.services.sfc.drivers.ovs.driver [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] networking_sfc.services.sfc.drivers.ovs.driver.OVSSfcDriver method _get_port_detail_info called with arguments (u'49ba9fd8-5840-4ab3-b9f7-96fc59225b37',) {} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66
2017-10-27 10:21:03.251 23 DEBUG neutron.plugins.ml2.drivers.type_tunnel [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] get_endpoint_by_host() called for host compute-1 get_endpoint_by_host /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_tunnel.py:350
2017-10-27 10:21:03.257 23 DEBUG networking_sfc.services.sfc.drivers.ovs.driver [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] networking_sfc.services.sfc.drivers.ovs.driver.OVSSfcDriver method _get_port_detail_info called with arguments (u'8f83b61d-3057-4d1c-a075-340271f845ee',) {} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66
2017-10-27 10:21:03.525 23 DEBUG neutron.plugins.ml2.drivers.type_tunnel [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] get_endpoint_by_host() called for host compute-1 get_endpoint_by_host /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_tunnel.py:350
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] DBAPIError exception wrapped from (pymysql.err.IntegrityError) (1048, u"Column 'local_endpoint' cannot be null") [SQL: u'INSERT INTO sfc_portpair_details (project_id, id, ingress, egress, host_id, in_mac_address, mac_address, network_type, segment_id, local_endpoint, correlation) VALUES (%(project_id)s, %(id)s, %(ingress)s, %(egress)s, %(host_id)s, %(in_mac_address)s, %(mac_address)s, %(network_type)s, %(segment_id)s, %(local_endpoint)s, %(correlation)s)'] [parameters: {'ingress': u'49ba9fd8-5840-4ab3-b9f7-96fc59225b37', 'segment_id': 48, 'correlation': None, 'network_type': u'vxlan', 'local_endpoint': None, 'in_mac_address': u'fa:16:3e:19:35:45', 'egress': u'8f83b61d-3057-4d1c-a075-340271f845ee', 'mac_address': u'fa:16:3e:44:20:29', 'host_id': u'compute-1', 'project_id': u'766c772012ff4c4ca6e563accd51f1ea', 'id': 'b54bec9f-23f9-45e6-9fac-0f6dfc75f25c'}]: IntegrityError: (1048, u"Column 'local_endpoint' cannot be null")
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters     context)
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters     packet.check_error()
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters     err.raise_mysql_exception(self._data)
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters     raise errorclass(errno, errval)
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters IntegrityError: (1048, u"Column 'local_endpoint' cannot be null")
2017-10-27 10:21:03.538 23 ERROR oslo_db.sqlalchemy.exc_filters
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] (pymysql.err.IntegrityError) (1048, u"Column 'local_endpoint' cannot be null") [SQL: u'INSERT INTO sfc_portpair_details (project_id, id, ingress, egress, host_id, in_mac_address, mac_address, network_type, segment_id, local_endpoint, correlation) VALUES (%(project_id)s, %(id)s, %(ingress)s, %(egress)s, %(host_id)s, %(in_mac_address)s, %(mac_address)s, %(network_type)s, %(segment_id)s, %(local_endpoint)s, %(correlation)s)'] [parameters: {'ingress': u'49ba9fd8-5840-4ab3-b9f7-96fc59225b37', 'segment_id': 48, 'correlation': None, 'network_type': u'vxlan', 'local_endpoint': None, 'in_mac_address': u'fa:16:3e:19:35:45', 'egress': u'8f83b61d-3057-4d1c-a075-340271f845ee', 'mac_address': u'fa:16:3e:44:20:29', 'host_id': u'compute-1', 'project_id': u'766c772012ff4c4ca6e563accd51f1ea', 'id': 'b54bec9f-23f9-45e6-9fac-0f6dfc75f25c'}]: DBError: (pymysql.err.IntegrityError) (1048, u"Column 'local_endpoint' cannot be null") [SQL: u'INSERT INTO sfc_portpair_details (project_id, id, ingress, egress, host_id, in_mac_address, mac_address, network_type, segment_id, local_endpoint, correlation) VALUES (%(project_id)s, %(id)s, %(ingress)s, %(egress)s, %(host_id)s, %(in_mac_address)s, %(mac_address)s, %(network_type)s, %(segment_id)s, %(local_endpoint)s, %(correlation)s)'] [parameters: {'ingress': u'49ba9fd8-5840-4ab3-b9f7-96fc59225b37', 'segment_id': 48, 'correlation': None, 'network_type': u'vxlan', 'local_endpoint': None, 'in_mac_address': u'fa:16:3e:19:35:45', 'egress': u'8f83b61d-3057-4d1c-a075-340271f845ee', 'mac_address': u'fa:16:3e:44:20:29', 'host_id': u'compute-1', 'project_id': u'766c772012ff4c4ca6e563accd51f1ea', 'id': 'b54bec9f-23f9-45e6-9fac-0f6dfc75f25c'}]
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager Traceback (most recent call last):
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/driver_manager.py", line 101, in _call_drivers
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     getattr(driver.obj, method_name)(context)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/drivers/base.py", line 80, in create_port_pair_postcommit
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     self.create_port_pair(context)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     return method(*args, **kwargs)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/drivers/ovs/driver.py", line 1169, in create_port_pair
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     self._create_port_pair_detail(port_pair)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     return method(*args, **kwargs)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/drivers/ovs/driver.py", line 1162, in _create_port_pair_detail
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     r = self.create_port_pair_detail(portpair_detail)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/drivers/ovs/db.py", line 244, in create_port_pair_detail
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     return self._make_port_detail_dict(port_obj)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     self.gen.next()
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1034, in _transaction_scope
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     yield resource
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     self.gen.next()
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 646, in _session
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     self.session.rollback()
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     self.force_reraise()
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     six.reraise(self.type_, self.value, self.tb)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 643, in _session
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     self._end_session_transaction(self.session)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 671, in _end_session_transaction
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     session.commit()
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 906, in commit
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     self.transaction.commit()
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 461, in commit
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     self._prepare_impl()
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 430, in _prepare_impl
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     self.session.dispatch.before_commit(self.session)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/event/attr.py", line 218, in __call__
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     fn(*args, **kw)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 266, in load_one_to_manys
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     session.flush()
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2177, in flush
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     self._flush(objects)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2297, in _flush
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     transaction.rollback(_capture_exception=True)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     compat.reraise(exc_type, exc_value, exc_tb)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2261, in _flush
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     flush_context.execute()
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     rec.execute(self)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     uow
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     mapper, table, insert)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 799, in _emit_insert_statements
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     execute(statement, multiparams)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     return meth(self, multiparams, params)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     return connection._execute_clauseelement(self, multiparams, params)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     compiled_sql, distilled_params
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     context)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1398, in _handle_dbapi_exception
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     util.raise_from_cause(newraise, exc_info)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     context)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     cursor.execute(statement, parameters)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     result = self._query(query)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     conn.query(q)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     result.read()
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     first_packet = self.connection._read_packet()
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     packet.check_error()
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     err.raise_mysql_exception(self._data)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager     raise errorclass(errno, errval)
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager DBError: (pymysql.err.IntegrityError) (1048, u"Column 'local_endpoint' cannot be null") [SQL: u'INSERT INTO sfc_portpair_details (project_id, id, ingress, egress, host_id, in_mac_address, mac_address, network_type, segment_id, local_endpoint, correlation) VALUES (%(project_id)s, %(id)s, %(ingress)s, %(egress)s, %(host_id)s, %(in_mac_address)s, %(mac_address)s, %(network_type)s, %(segment_id)s, %(local_endpoint)s, %(correlation)s)'] [parameters: {'ingress': u'49ba9fd8-5840-4ab3-b9f7-96fc59225b37', 'segment_id': 48, 'correlation': None, 'network_type': u'vxlan', 'local_endpoint': None, 'in_mac_address': u'fa:16:3e:19:35:45', 'egress': u'8f83b61d-3057-4d1c-a075-340271f845ee', 'mac_address': u'fa:16:3e:44:20:29', 'host_id': u'compute-1', 'project_id': u'766c772012ff4c4ca6e563accd51f1ea', 'id': 'b54bec9f-23f9-45e6-9fac-0f6dfc75f25c'}]
2017-10-27 10:21:03.571 23 ERROR networking_sfc.services.sfc.driver_manager
2017-10-27 10:21:03.585 23 ERROR networking_sfc.services.sfc.driver_manager [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] SFC driver 'ovs' failed in create_port_pair_postcommit: DBError: (pymysql.err.IntegrityError) (1048, u"Column 'local_endpoint' cannot be null") [SQL: u'INSERT INTO sfc_portpair_details (project_id, id, ingress, egress, host_id, in_mac_address, mac_address, network_type, segment_id, local_endpoint, correlation) VALUES (%(project_id)s, %(id)s, %(ingress)s, %(egress)s, %(host_id)s, %(in_mac_address)s, %(mac_address)s, %(network_type)s, %(segment_id)s, %(local_endpoint)s, %(correlation)s)'] [parameters: {'ingress': u'49ba9fd8-5840-4ab3-b9f7-96fc59225b37', 'segment_id': 48, 'correlation': None, 'network_type': u'vxlan', 'local_endpoint': None, 'in_mac_address': u'fa:16:3e:19:35:45', 'egress': u'8f83b61d-3057-4d1c-a075-340271f845ee', 'mac_address': u'fa:16:3e:44:20:29', 'host_id': u'compute-1', 'project_id': u'766c772012ff4c4ca6e563accd51f1ea', 'id': 'b54bec9f-23f9-45e6-9fac-0f6dfc75f25c'}]
2017-10-27 10:21:03.585 23 ERROR networking_sfc.services.sfc.plugin [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] create_port_pair_postcommit failed.: SfcDriverError: create_port_pair_postcommit failed.
2017-10-27 10:21:03.585 23 ERROR networking_sfc.services.sfc.plugin Traceback (most recent call last):
2017-10-27 10:21:03.585 23 ERROR networking_sfc.services.sfc.plugin   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/plugin.py", line 119, in create_port_pair
2017-10-27 10:21:03.585 23 ERROR networking_sfc.services.sfc.plugin     self.driver_manager.create_port_pair_postcommit(portpair_context)
2017-10-27 10:21:03.585 23 ERROR networking_sfc.services.sfc.plugin   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/driver_manager.py", line 142, in create_port_pair_postcommit
2017-10-27 10:21:03.585 23 ERROR networking_sfc.services.sfc.plugin     self._call_drivers("create_port_pair_postcommit", context)
2017-10-27 10:21:03.585 23 ERROR networking_sfc.services.sfc.plugin   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/driver_manager.py", line 113, in _call_drivers
2017-10-27 10:21:03.585 23 ERROR networking_sfc.services.sfc.plugin     method=method_name
2017-10-27 10:21:03.585 23 ERROR networking_sfc.services.sfc.plugin SfcDriverError: create_port_pair_postcommit failed.
2017-10-27 10:21:03.585 23 ERROR networking_sfc.services.sfc.plugin
2017-10-27 10:21:03.591 23 ERROR networking_sfc.services.sfc.plugin [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] Create port pair failed, deleting port_pair '357b986b-e25f-43e5-8632-4ccf8d41a506': SfcDriverError: create_port_pair_postcommit failed.
2017-10-27 10:21:03.591 23 DEBUG networking_sfc.services.sfc.plugin [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] networking_sfc.services.sfc.plugin.SfcPlugin method delete_port_pair called with arguments (<neutron_lib.context.Context object at 0x94340d0>, '357b986b-e25f-43e5-8632-4ccf8d41a506') {} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66
2017-10-27 10:21:03.592 23 DEBUG networking_sfc.db.sfc_db [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] networking_sfc.services.sfc.plugin.SfcPlugin method get_port_pair called with arguments (<neutron_lib.context.Context object at 0x94340d0>, '357b986b-e25f-43e5-8632-4ccf8d41a506') {} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66
2017-10-27 10:21:03.607 23 DEBUG networking_sfc.services.sfc.drivers.ovs.driver [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] networking_sfc.services.sfc.drivers.ovs.driver.OVSSfcDriver method delete_port_pair called with arguments (<networking_sfc.services.sfc.common.context.PortPairContext object at 0x9434290>,) {} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66
2017-10-27 10:21:03.626 23 DEBUG networking_sfc.db.sfc_db [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] networking_sfc.services.sfc.plugin.SfcPlugin method get_port_pair called with arguments (<neutron_lib.context.Context object at 0x94340d0>, '357b986b-e25f-43e5-8632-4ccf8d41a506') {} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66
2017-10-27 10:21:03.636 23 DEBUG networking_sfc.db.sfc_db [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] networking_sfc.services.sfc.plugin.SfcPlugin method delete_port_pair called with arguments (<neutron_lib.context.Context object at 0x94340d0>, '357b986b-e25f-43e5-8632-4ccf8d41a506') {} wrapper /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py:66
2017-10-27 10:21:03.676 27 DEBUG networking_odl.journal.journal [req-762d9010-3579-4fe6-b572-5345bb45a3e9 - - - - -] Start processing journal entries sync_pending_entries /var/lib/kolla/venv/lib/python2.7/site-packages/networking_odl/journal/journal.py:221
2017-10-27 10:21:03.702 27 DEBUG networking_odl.journal.journal [req-762d9010-3579-4fe6-b572-5345bb45a3e9 - - - - -] No journal entries to process sync_pending_entries /var/lib/kolla/venv/lib/python2.7/site-packages/networking_odl/journal/journal.py:225
2017-10-27 10:21:03.728 28 INFO networking_odl.journal.periodic_task [req-104b8320-b47c-4864-9d1c-175ba76fbe4f - - - - -] Starting hostconfig periodic task.
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] create failed: No details.: SfcDriverError: create_port_pair_postcommit failed.
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource Traceback (most recent call last):
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 98, in resource
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 435, in create
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     return self._create(request, body, **kwargs)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 92, in wrapped
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     self.force_reraise()
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 88, in wrapped
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     self.force_reraise()
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 127, in wrapped
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     self.force_reraise()
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/api.py", line 123, in wrapped
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 548, in _create
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     obj = do_create(body)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 530, in do_create
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     request.context, reservation.reservation_id)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     self.force_reraise()
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/api/v2/base.py", line 523, in do_create
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     return obj_creator(request.context, **kwargs)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_log/helpers.py", line 67, in wrapper
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     return method(*args, **kwargs)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/plugin.py", line 126, in create_port_pair
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     self.delete_port_pair(context, portpair_db['id'])
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     self.force_reraise()
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/plugin.py", line 119, in create_port_pair
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     self.driver_manager.create_port_pair_postcommit(portpair_context)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/driver_manager.py", line 142, in create_port_pair_postcommit
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     self._call_drivers("create_port_pair_postcommit", context)
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource   File "/var/lib/kolla/venv/lib/python2.7/site-packages/networking_sfc/services/sfc/driver_manager.py", line 113, in _call_drivers
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource     method=method_name
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource SfcDriverError: create_port_pair_postcommit failed.
2017-10-27 10:21:03.854 23 ERROR neutron.api.v2.resource
2017-10-27 10:21:03.901 23 INFO neutron.wsgi [req-a21451b3-92fb-4b66-9d25-cf7b30e3f675 0efeac25b8a845c799800fa87d850024 766c772012ff4c4ca6e563accd51f1ea - default default] 192.168.133.254,192.168.133.100 "POST /v2.0/sfc/port_pairs HTTP/1.1" status: 500  len: 302 time: 1.6327991
2017-10-27 10:21:03.950 28 INFO networking_odl.journal.periodic_task [req-104b8320-b47c-4864-9d1c-175ba76fbe4f - - - - -] Starting _get_and_update_hostconfigs phase of periodic task hostconfig.
2017-10-27 10:21:04.003 28 INFO networking_odl.ml2.pseudo_agentdb_binding [req-104b8320-b47c-4864-9d1c-175ba76fbe4f - - - - -] REST/GET hostconfigs from ODL
2017-10-27 10:21:04.003 28 DEBUG networking_odl.common.client [req-104b8320-b47c-4864-9d1c-175ba76fbe4f - - - - -] Sending METHOD (get) URL (http://192.168.133.254:8088/restconf/operational/neutron:neutron/hostconfigs/) JSON (None) request /var/lib/kolla/venv/lib/python2.7/site-packages/networking_odl/common/client.py:89
2017-10-27 10:21:04.016 28 DEBUG networking_odl.ml2.pseudo_agentdb_binding [req-104b8320-b47c-4864-9d1c-175ba76fbe4f - - - - -] ODLPORTBINDING hostconfigs:
[
    "{\"hostconfigs\":{\"hostconfig\":[{\"host-id\":\"controller-1\",\"host-type\":\"ODL L2\",\"config\":\"{\\\"supported_vnic_types\\\": [{\\\"vnic_type\\",
    "\": \\\"normal\\\", \\\"vif_type\\\": \\\"ovs\\\", \\\"vif_details\\\": {} }], \\\"allowed_network_types\\\": [\\\"flat\\\", \\\"vlan\\\", \\\"vxlan\\\"], \\\"data",
    "path_types\\\": [\\\"netdev\\\", \\\"system\\\"], \\\"bridge_mappings\\\": {\\\"physnet1\\\":\\\"br-ex\\\"} }\"},{\"host-id\":\"compute-1\",\"host-type\":\"OD",
    "L L2\",\"config\":\"{\\\"supported_vnic_types\\\": [{\\\"vnic_type\\\": \\\"normal\\\", \\\"vif_type\\\": \\\"ovs\\\", \\\"vif_details\\\": {} }], \\\"allowed",
    "_network_types\\\": [\\\"flat\\\", \\\"vlan\\\", \\\"vxlan\\\"], \\\"datapath_types\\\": [\\\"netdev\\\", \\\"system\\\"], \\\"bridge_mappings\\\": {\\\"physnet",
    "1\\\":\\\"br-ex\\\"} }\"}]}}"
] _rest_get_hostconfigs /var/lib/kolla/venv/lib/python2.7/site-packages/networking_odl/ml2/pseudo_agentdb_binding.py:89
2017-10-27 10:21:04.016 28 DEBUG networking_odl.ml2.pseudo_agentdb_binding [req-104b8320-b47c-4864-9d1c-175ba76fbe4f - - - - -] ODLPORTBINDING Updating agents DB with ODL hostconfigs update_agents_db /var/lib/kolla/venv/lib/python2.7/site-packages/networking_odl/ml2/pseudo_agentdb_binding.py:273
2017-10-27 10:21:04.080 28 DEBUG neutron_lib.callbacks.manager [req-104b8320-b47c-4864-9d1c-175ba76fbe4f - - - - -] Notify callbacks ['neutron.services.segments.db._update_segment_host_mapping_for_agent-7394263'] for agent, after_update _notify_loop /var/lib/kolla/venv/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2017-10-27 10:21:04.138 28 DEBUG neutron_lib.callbacks.manager [req-104b8320-b47c-4864-9d1c-175ba76fbe4f - - - - -] Notify callbacks ['neutron.services.segments.db._update_segment_host_mapping_for_agent-7394263'] for agent, after_update _notify_loop /var/lib/kolla/venv/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2017-10-27 10:21:04.139 28 INFO networking_odl.journal.periodic_task [req-104b8320-b47c-4864-9d1c-175ba76fbe4f - - - - -] Finished _get_and_update_hostconfigs phase of hostconfig task.
2017-10-27 10:21:04.250 28 INFO networking_odl.journal.periodic_task [req-104b8320-b47c-4864-9d1c-175ba76fbe4f - - - - -] hostconfig task has been finished
2017-10-27 10:21:07.701 29 DEBUG neutron.db.agents_db [req-4a02bca4-88e9-43d7-9475-e665e4564baf - - - - -] Agent healthcheck: found 4 active agents agent_health_check /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/db/agents_db.py:270

Revision history for this message

Bernard Cafarelli (bcafarel) wrote on 2017-10-27:

No problem, so the error is local_endpoint cannot be NULL

I think I already saw some errors like that when the ports are not attached to a VM. But from your command outputs, it does not looks like this is the case here (P1 and P2 are marked as active, and are the FW vm ports right?)

Revision history for this message

vikrant (ervikrant06) wrote on 2017-10-31:

#10

Yes, in my case that doesn't seem to be an issue.

~~~
[root@controller-1 ~]# nova list
+--------------------------------------+-------+--------+------------+-------------+-------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+-------------------------------+
| 1a59573b-b557-488f-917c-536a2fd21f35 | FW | ACTIVE | - | Running | demo-net=10.0.0.14, 10.0.0.12 |
| c540bcc3-ce27-46e9-85e3-cc4d124194ae | demo1 | ACTIVE | - | Running | demo-net=10.0.0.6 |
+--------------------------------------+-------+--------+------------+-------------+-------------------------------+
[root@controller-1 ~]# neutron port-list | egrep 'P1|P2'
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
| 49ba9fd8-5840-4ab3-b9f7-96fc59225b37 | P1 | 766c772012ff4c4ca6e563accd51f1ea | fa:16:3e:19:35:45 | {"subnet_id": "08bbc3a6-4fe0-4ca2-bc03-1425a69b53f6", "ip_address": "10.0.0.14"} |
| 8f83b61d-3057-4d1c-a075-340271f845ee | P2 | 766c772012ff4c4ca6e563accd51f1ea | fa:16:3e:44:20:29 | {"subnet_id": "08bbc3a6-4fe0-4ca2-bc03-1425a69b53f6", "ip_address": "10.0.0.12"} |
~~~

Revision history for this message

vikrant (ervikrant06) wrote on 2017-11-07:

#11

Any luck on this issue?

Revision history for this message

Bernard Cafarelli (bcafarel) wrote on 2017-11-20:

#12

Sorry, no progress on my side. I suspect this is related to the containerized setup, but I'd need to setup a similar kolla setup to test.

Not the code parts I am most familiar with, but this local_endpoint info comes from the port (somewhere in the "extended" parts), so that's probably the direction to look into

Revision history for this message

lokendra (lokendrarathour-b) wrote on 2023-01-11:

#13

Hi,
I also had the same error, but I was able to fix the same by creating the network of vxlan type.
By default, I was creating the network of flat type, because of which I was getting the same error.

So what I did is:
recreated the network from flat type to vxlan type. and the re-attempted port-pair/port-pair-group creation it worked.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.