kolla

Fail to create instance on Ubuntu host

Bug #1604343 reported by beanking77
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Invalid
Critical
Unassigned
kolla newton-rc3 "rc3"

Bug Description

Host: ubuntu 14.04.4
AIO with image from source and ubuntu
neutron_plugin_agent: "openvswitch" (linuxbridge work fine)

When create instance I got error "No valid host was found. There are not enough hosts available. 500".
Require resources are satisfy and I try to trace log.

And Open vSwitch agent are not in agent list
# neutron agent-list
+--------------------------------------+----------------+-------+-------------------+-------+----------------+------------------------+
| id | agent_type | host | availability_zone | alive | admin_state_up | binary |
+--------------------------------------+----------------+-------+-------------------+-------+----------------+------------------------+
| 43d9cb18-bc3e-4777-b026-6218a9793e4e | L3 agent | kolla | nova | :-) | True | neutron-l3-agent |
| 9ba3e422-43df-424e-9656-75f1a1bef81e | DHCP agent | kolla | nova | :-) | True | neutron-dhcp-agent |
| c4abac8b-21f1-481c-9d86-489d6b6b8770 | Metadata agent | kolla | | :-) | True | neutron-metadata-agent |
+--------------------------------------+----------------+-------+-------------------+-------+----------------+------------------------+

neutron port-show 526d83d5-d7f2-4afe-b49b-8bf5dbdfbc05
+-----------------------+-----------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | kolla |
| binding:profile | {} |
| binding:vif_details | {} |
| binding:vif_type | binding_failed |
| binding:vnic_type | normal |
| created_at | 2016-07-20T07:48:55 |
| description | |
| device_id | ec129423-561f-4713-be73-2997316d9614 |
| device_owner | network:router_gateway |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "d3b40350-aa6f-4dec-b431-a74e83a5470c", "ip_address": "10.0.2.159"} |
| id | 526d83d5-d7f2-4afe-b49b-8bf5dbdfbc05 |
| mac_address | fa:16:3e:30:eb:26 |
| name | |
| network_id | 9f3e0bf3-0bdd-430a-8ad4-83969443c4fd |
| security_groups | |
| status | DOWN |
| tenant_id | |
| updated_at | 2016-07-20T07:48:57 |
+-----------------------+-----------------------------------------------------------------------------------+

nova_compute:
===========================================
2016-07-19 18:06:13.651 1 INFO nova.virt.libvirt.driver [req-5a573e50-df47-4da7-be29-c7a106df6c7a 44704d64984542e5bdd7b0d9a71b73f1 67f28363deb74e0881bba1e41688b79f - - -] [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] Creating image
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [req-5a573e50-df47-4da7-be29-c7a106df6c7a 44704d64984542e5bdd7b0d9a71b73f1 67f28363deb74e0881bba1e41688b79f - - -] [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] Instance failed to spawn
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] Traceback (most recent call last):
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2066, in _build_resources
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] yield resources
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1910, in _build_and_run_instance
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] block_device_info=block_device_info)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2664, in spawn
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] write_to_disk=True)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4659, in _get_guest_xml
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] network_info_str = str(network_info)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/network/model.py", line 524, in __str__
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] return self._sync_wrapper(fn, *args, **kwargs)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/network/model.py", line 507, in _sync_wrapper
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] self.wait()
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/network/model.py", line 539, in wait
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] self[:] = self._gt.wait()
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] return self._exit_event.wait()
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] current.throw(*self._exc)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] result = function(*args, **kwargs)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/utils.py", line 1053, in context_wrapper
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] return func(*args, **kwargs)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1405, in _allocate_network_async
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] six.reraise(*exc_info)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1388, in _allocate_network_async
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] bind_host_id=bind_host_id)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 737, in allocate_for_instance
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] bind_host_id, dhcp_opts, available_macs)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 852, in _update_ports_for_instance
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] self._delete_ports(neutron, instance, created_port_ids)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] self.force_reraise()
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] six.reraise(self.type_, self.value, self.tb)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 825, in _update_ports_for_instance
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] port_client, instance, port_id, port_req_body)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 303, in _update_port
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] _ensure_no_port_binding_failure(port)
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in _ensure_no_port_binding_failure
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] raise exception.PortBindingFailed(port_id=port['id'])
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af] PortBindingFailed: Binding failed for port 5dff4895-0c5d-46da-8636-f611ae2061a7, please check neutron logs for more information.
2016-07-19 18:06:13.653 1 ERROR nova.compute.manager [instance: 917f7f49-4d66-4f97-a9ed-539ebb5672af]

neutron-server.log:
===========================================
2016-07-19 18:19:54.629 19 ERROR neutron.plugins.ml2.managers [req-0c05db13-41df-45d9-a68a-18bc1823eef4 - - - - -] Failed to bind port 96218493-4fdf-457a-a91a-fc0c860cf223 on host kolla for vnic_type normal using segments [{'segmentation_id': 44, 'physical_network': None, 'id': u'ec360281-3a7a-462e-82e3-02ac4a16aac2', 'network_type': u'vxlan'}]

neutron/neutron-openvswitch-agent.log:
===========================================
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/ovsdb/native/connection.py", line 88, in do_get_schema_helper
    self.schema_name)
  File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/ovsdb/native/idlutils.py", line 109, in get_schema_helper
    'err': os.strerror(err)})
Exception: Could not retrieve schema from tcp:127.0.0.1:6640: Connection refused

neutron/neutron-l3-agent.log:
===========================================
2016-07-19 18:21:39.963 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 377, in _internal_network_added
2016-07-19 18:21:39.963 1 ERROR neutron.agent.l3.agent prefix=prefix, mtu=mtu)
2016-07-19 18:21:39.963 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 251, in plug
2016-07-19 18:21:39.963 1 ERROR neutron.agent.l3.agent bridge, namespace, prefix, mtu)
2016-07-19 18:21:39.963 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 326, in plug_new
2016-07-19 18:21:39.963 1 ERROR neutron.agent.l3.agent self.check_bridge_exists(bridge)
2016-07-19 18:21:39.963 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 228, in check_bridge_exists
2016-07-19 18:21:39.963 1 ERROR neutron.agent.l3.agent raise exceptions.BridgeDoesNotExist(bridge=bridge)
2016-07-19 18:21:39.963 1 ERROR neutron.agent.l3.agent BridgeDoesNotExist: Bridge br-int does not exist.
2016-07-19 18:21:39.963 1 ERROR neutron.agent.l3.agent

See original description
beanking77 (beanking77)
description: updated
beanking77 (beanking77)
description: updated
beanking77 (beanking77)
description: updated
beanking77 (beanking77)
description: updated
Revision history for this message
Rahul Sharma (rahulsharmaait) wrote :

Can you please provide the kolla release you are using. I faced a similar issue with stable/mitaka few weeks back. After upgrading the codebase (to release 2.0.2 or 2.0.3), my error of no openvswitch-agent got resolved.

You can check if br-int or br-tun are getting created or not by logging into openvswitch-vswitchd container and run "ovs-vsctl show".

Revision history for this message
beanking77 (beanking77) wrote :

I'm following master branch.Thanks for your reply and hints.
Let me try if it work or not.

Revision history for this message
beanking77 (beanking77) wrote :

I'm checkout to 2.0.2 and re-deploy.The openvswitch-agent still not exist and I
login to openvswitch-vswitchd container and run "ovs-vsctl show":

aef9db35-a212-4f46-b5b5-47c7752cd70d
    Manager "ptcp:6640:127.0.0.1"
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "eth1"
            Interface "eth1"

looks like br-ex are been created but neutron still not working.

zhubingbing (zhubingbing)
Changed in kolla:
status: New → Triaged
milestone: none → newton-rc2
zhubingbing (zhubingbing)
Changed in kolla:
importance: Undecided → Critical
importance: Critical → Medium
Duong Ha-Quang (duonghq)
Changed in kolla:
importance: Medium → High
Revision history for this message
Steven Dake (sdake) wrote :

beanking77,

The last report in the bug log was from July. Could you reproduce this issue with 3.0.0-rc2 which will be released tomorrow (when release team releases it). I am tagging at 23:45 UTC - release team should get to it in next 24 hours.

Until then marking incomplete.

I am unable to reproduce this issue.

Changed in kolla:
importance: High → Critical
status: Triaged → Incomplete
Steven Dake (sdake)
Changed in kolla:
milestone: newton-rc2 → newton-rc3
shake.chen (shake-chen)
Changed in kolla:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.