Juniper Openstack

[Build "Ubuntu 12.04.3 LTS" 2.0-2442~icehouse] service instance launch failing due to PortInUse

Bug #1390396 reported by Sandip Dey
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Fix Committed
Undecided
Rudra Rugge

Bug Description

vm interface
=== ==============
73506588-1488-4891-8465-7c2299fd18a1 2bb4f797-f7f5-4ca9-aa87-d84ceddc0bf9
b3cf6611-25b7-4cd6-a373-2327fdb6e64e 2bb4f797-f7f5-4ca9-aa87-d84ceddc0bf9

2 service instance vms (as shown above),trying to use the same port
port 2bb4f797-f7f5-4ca9-aa87-d84ceddc0bf9 deleted at 2014-11-07 10:13:16
Another service vm tried to use the same port at 2014-11-07 10:15:36.846

All logs saved at http://10.204.216.50/Docs/bugs/<bug_id>

Logs
=====

1:cassandra_log_nodeb2_2014_11_07_12_26_05.log:2014 Nov 07 10:11:54.110335 nodeb2 [Config:contrail-api:0:__default__][SYS_INFO] : VncApiConfigLog:220 [VncApiCommon: identifier_uuid = 2bb4f797-f7f5-4ca9-aa87-d84ceddc0bf9, object_type = virtual_machine_interface, identifier_name = default-domain:TestECMPSanity-74583632:default-domain__TestECMPSanity-74583632__2b9fb15e-13ad-4df9-a401-c652e99b529a__2-management, url = http://127.0.0.1:9100/virtual-machine-interface/2bb4f797-f7f5-4ca9-aa87-d84ceddc0bf9, operation = put, useragent = /usr/bin/contrail-api, remote_ip = 127.0.0.1:9100, body = {u'virtual-machine-interface': {'id_perms': {u'enable': True, u'description': None, u'created': u'2014-11-07T04:41:46.557184', u'uuid': {u'uuid_mslong': 3149414271510989993, u'uuid_lslong': 12288027933223422969L}, u'user_visible': True, u'last_modified': u'2014-11-07T04:41:47.038040', u'permissions': {u'owner': u'admin', u'owner_access': 7, u'other_access': 7, u'group': u'KeystoneServiceAdmin', u'group_access': 7}}, u'virtual_machine_refs': [{u'to': [u'73506588-1488-4891-8465-7c2299fd18a1'], u'uuid': u'73506588-1488-4891-8465-7c2299fd18a1'}], u'uuid': u'2bb4f797-f7f5-4ca9-aa87-d84ceddc0bf9', u'virtual_machine_interface_device_owner': u'compute:None'}}, user = admin, project = admin, domain = None]

contrail/contrail-api-0-stdout.log:127.0.0.1 - - [2014-11-07 10:13:16] "DELETE /virtual-machine-interface/2bb4f797-f7f5-4ca9-aa87-d84ceddc0bf9 HTTP/1.1" 200 115 0.074727

contrail/nova-scheduler-supervisor-stdout.log:2014-11-07 10:15:36.846 3173 ERROR nova.scheduler.filter_scheduler [req-cb73c3e1-39c0-4e22-bd72-7a5da42b1322 9b1225f666d343fabd135fb39aefb8c3 9423c009888441099a42bca904b9f9c5] [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] Error from last host: nodeb2 (node nodeb2.englab.juniper.net): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1308, in _build_instance\n set_access_ip=set_access_ip)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 396, in decorated_function\n return function(self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1720, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', u' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1717, in _spawn\n block_device_info)\n', u' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2258, in spawn\n admin_pass=admin_password)\n', u' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2728, in _create_image\n instance, network_info, admin_pass, files, suffix)\n', u' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2527, in _inject_data\n net = netutils.get_injected_network_template(network_info)\n', u' File "/usr/lib/python2.7/dist-packages/nova/virt/netutils.py", line 71, in get_injected_network_template\n if not (network_info and template):\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 420, in __len__\n return self._sync_wrapper(fn, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 407, in _sync_wrapper\n self.wait()\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 445, in wait\n self[:] = self._gt.wait()\n', u' File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait\n return self._exit_event.wait()\n', u' File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 120, in wait\n current.throw(*self._exc)\n', u' File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main\n result = function(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1507, in _allocate_network_async\n dhcp_options=dhcp_options)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 240, in allocate_for_instance\n raise exception.PortInUse(port_id=port_id)\n', u'PortInUse: Port 2bb4f797-f7f5-4ca9-aa87-d84ceddc0bf9 is still in use.\n']
haproxy.log:Nov 7 10:11:48 nodeb2 haproxy[2202]: 10.204.216.33:48333 [07/Nov/2014:10:11:48.014] contrail-api contrail-api-backend/10.204.216.33 2/0/0/6/8 200 2341 - - ---- 17/14/3/4/0 0/0 "GET /virtual-machine-interface/2bb4f797-f7f5-4ca9-aa87-d84ceddc0bf9?exclude_back_refs=True&exclude_children=True HTTP/1.1"

Revision history for this message
Sandip Dey (sandipd) wrote :
Download full text (5.1 KiB)

Nova compute error
=================

2014-11-07 10:15:28.454 2223 ERROR nova.compute.manager [req-cb73c3e1-39c0-4e22-bd72-7a5da42b1322 9b1225f666d343fabd135fb39aefb8c3 9423c009888441099a42bca904b9f9c5] [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] Instance failed to spawn
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] Traceback (most recent call last):
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1717, in _spawn
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] block_device_info)
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2258, in spawn
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] admin_pass=admin_password)
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2728, in _create_image
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] instance, network_info, admin_pass, files, suffix)
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2527, in _inject_data
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] net = netutils.get_injected_network_template(network_info)
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] File "/usr/lib/python2.7/dist-packages/nova/virt/netutils.py", line 71, in get_injected_network_template
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] if not (network_info and template):
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 420, in __len__
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] return self._sync_wrapper(fn, *args, **kwargs)
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 407, in _sync_wrapper
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] self.wait()
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] File "/usr/lib/python2.7/dist-packages/nova/network/model.py", line 445, in wait
2014-11-07 10:15:28.454 2223 TRACE nova.compute.manager [instance: b3cf6611-25b7-4cd6-a373-2327fdb6e64e] self[:] = self._gt.wait()
2014-11-07 10:15:28.454 2223 TRACE nova.compute.ma...

Read more...

OpenContrail Admin (ci-admin-f)
Changed in juniperopenstack:
assignee: nobody → Rudra Rugge (rudrarugge)
information type: Proprietary → Public
Sandip Dey (sandipd)
tags: added: blocker
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/4793
Committed: http://github.org/Juniper/contrail-controller/commit/7cd97d2a70e43fa1e3c2300f90bbf051a9099063
Submitter: Zuul
Branch: R2.0

commit 7cd97d2a70e43fa1e3c2300f90bbf051a9099063
Author: Rudra Rugge <email address hidden>
Date: Tue Nov 18 15:00:02 2014 -0800

Service monitor changes for VM services

- Store VMI information since delete by Nova and
API server are synced. Hence a new VM needs to
be launched with a different fully qualified name.

- Add virtual machine state information to DB so
that status check is not started until SI is active.

- Add exception handling for Nova calls and return None
for cases of exception to the clients.

Change-Id: I4037547ce81c1712697115b49a3d4eaa1002c8af
Closes-Bug: #1390396

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/4725
Committed: http://github.org/Juniper/contrail-controller/commit/66cb5d56c927279631a5aa873ce35818e0a82fc4
Submitter: Zuul
Branch: master

commit 66cb5d56c927279631a5aa873ce35818e0a82fc4
Author: Rudra Rugge <email address hidden>
Date: Tue Nov 18 15:00:02 2014 -0800

Service monitor changes for VM services

- Store VMI information since delete by Nova and
API server are synced. Hence a new VM needs to
be launched with a different fully qualified name.

- Add virtual machine state information to DB so
that status check is not started until SI is active.

- Add exception handling for Nova calls and return None
for cases of exception to the clients.

Change-Id: I4037547ce81c1712697115b49a3d4eaa1002c8af
Closes-Bug: #1390396

Changed in juniperopenstack:
status: New → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.