[root@fuel ~]# fuel --fuel-version
api: '1.0'
astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
auth_required: true
build_id: 2015-02-07_20-50-01
build_number: '76'
feature_groups:
- mirantis
fuellib_sha: 64f3ebe9fcbd18bf6c80a948e06061783a090347
fuelmain_sha: c799e3a6d88289e58db764a6be7910aab7da3149
nailgun_sha: 2ef819732a3ee7acf7b610e7d1c1a6da0434c1a0
ostf_sha: 3b57985d4d2155510894a1f6d03b478b201f7780
production: docker
release: 6.0.1
release_versions:
2014.2-6.0.1:
VERSION:
api: '1.0'
astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
build_id: 2015-02-07_20-50-01
build_number: '76'
feature_groups:
- mirantis
fuellib_sha: 64f3ebe9fcbd18bf6c80a948e06061783a090347
fuelmain_sha: c799e3a6d88289e58db764a6be7910aab7da3149
nailgun_sha: 2ef819732a3ee7acf7b610e7d1c1a6da0434c1a0
ostf_sha: 3b57985d4d2155510894a1f6d03b478b201f7780
production: docker
release: 6.0.1
Baremetal,Ubuntu, HA, Neutron-vlan,Сeph-all, Debug, 6.0.1_76
Controllers:3 Computes:22
Deployment was successful, but during rally tests we got 14 exceptions like this:
Iteration Exception type Exception message
▼ 0 GetResourceErrorStatus Resource <Server: rally_novaserver_hltalxntncpojhwo-4f4411dd-c375-483a-bbbb-343955ab23b9> has ERROR status: No valid host was found. Exceeded max scheduling attempts 3 for instance 4f4411dd-c375-483a-bbbb-343955ab23b9. Last exception: [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2014, in do_
Traceback (most recent call last):
File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 77, in _run_scenario_once
method_name)(**kwargs) or scenario_output
File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/nova/servers.py", line 64, in boot_and_list_server
self._generate_random_name(), image, flavor, **kwargs)
File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 254, in func_atomic_actions
f = func(self, *args, **kwargs)
File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/nova/utils.py", line 126, in _boot_server
check_interval=CONF.benchmark.nova_server_boot_poll_interval
File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/utils.py", line 104, in wait_for
resource = update_resource(resource)
File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/utils.py", line 67, in _get_from_manager
status=status, fault=msg)
GetResourceErrorStatus: Resource <Server: rally_novaserver_hltalxntncpojhwo-4f4411dd-c375-483a-bbbb-343955ab23b9> has ERROR status: No valid host was found. Exceeded max scheduling attempts 3 for instance 4f4411dd-c375-483a-bbbb-343955ab23b9. Last exception: [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2014, in do_
from controller node:
/var/log/nova/nova-conductor.log:10959:2015-02-18 09:00:44.956 11477 ERROR nova.scheduler.utils [req-f9d9308f-5369-4017-bfa4-9f5d0c75f7ad None] [instance: 4f4411dd-c375-483a-bbbb-343955ab23b9] Error from last host: node-33 (node node-33.domain.tld): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2014, in do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2149, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 4f4411dd-c375-483a-bbbb-343955ab23b9 was re-scheduled: Unexpected vif_type=binding_failed\n']
instance can't be started because "Unexpected vif_type=binding_failed"
from compute node-33:
/var/log/nova/nova-compute.log:2015-02-18 09:08:18.229 25356 DEBUG nova.virt.libvirt.vif [req-04998402-a8cb-4efc-beec-b8b590ccd500 None] vif_type=binding_failed instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=False,availability_zone=None,cell_name=None,cleaned=True,config_drive='',created_at=2015-02-18T09:02:40Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,disable_terminate=False,display_description='rally_novaserver_lznhitgdmrzegjbv',display_name='rally_novaserver_lznhitgdmrzegjbv-75169ccc-69de-45b7-b324-1550787a0ee2',ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,host='node-33',hostname='rally-novaserver-lznhitgdmrzegjbv-75169ccc-69de-45b7-b324-1550787a0ee2',id=7812,image_ref='f004b125-2631-4d7a-bae0-cf0bfd2ab2d0',info_cache=InstanceInfoCache,instance_type_id=9,kernel_id='',key_data=None,key_name=None,launch_index=1,launched_at=None,launched_on='node-33',locked=False,locked_by=None,memory_mb=64,metadata={},node='node-33.domain.tld',numa_topology=<?>,os_type=None,pci_devices=<?>,power_state=0,progress=0,project_id='92d27e7ac3024d0b97f9950c243fda9a',ramdisk_id='',reservation_id='r-255ubgq9',root_device_name='/dev/vda',root_gb=0,scheduled_at=None,security_groups=SecurityGroupList,shutdown_terminate=False,system_metadata={clean_attempts='2',image_base_image_ref='f004b125-2631-4d7a-bae0-cf0bfd2ab2d0',image_container_format='bare',image_disk_format='qcow2',image_min_disk='0',image_min_ram='64',image_murano_image_info='{"title": "Murano Demo", "type": "cirros.demo"}',instance_type_ephemeral_gb='0',instance_type_flavorid='0',instance_type_id='9',instance_type_memory_mb='64',instance_type_name='m1.nano',instance_type_root_gb='0',instance_type_rxtx_factor='1.0',instance_type_swap='0',instance_type_vcpu_weight=None,instance_type_vcpus='1',network_allocated='True'},task_state='deleting',terminated_at=None,updated_at=2015-02-18T09:08:18Z,user_data=None,user_id='664a1df632e4497e86d9dddc56d433b6',uuid=75169ccc-69de-45b7-b324-1550787a0ee2,vcpus=1,vm_mode=None,vm_state='error') vif=VIF({'profile': {}, 'ovs_interfaceid': None, 'network': Network({'bridge': None, 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': u'fixed', 'floating_ips': [], 'address': u'100.1.42.13'})], 'version': 4, 'meta': {u'dhcp_server': u'100.1.42.4'}, 'dns': [], 'routes': [], 'cidr': u'100.1.42.0/24', 'gateway': IP({'meta': {}, 'version': 4, 'type': u'gateway', 'address': u'100.1.42.1'})})], 'meta': {u'injected': False, u'tenant_id': u'92d27e7ac3024d0b97f9950c243fda9a'}, 'id': u'ec8f9e31-8fc0-4ef5-9bed-4613f9e947ed', 'label': u'rally_net_qenswrmupakmbuir'}), 'devname': u'tap76b60493-46', 'vnic_type': u'normal', 'qbh_params': None, 'meta': {}, 'details': {}, 'address': u'fa:16:3e:59:c8:ab', 'active': False, 'type': u'binding_failed', 'id': u'76b60493-4687-4a08-9221-295b147623d4', 'qbg_params': None}) unplug /usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py:747
The process has been failed on the OpenvswitchMechanismDriver:check_segment_for_agent step
We used 30 VLANs for internal networks. It means that we could create only 30 tenants with individual networks. During the rally test we have created 44 tenants and got 14 errors. When we used all 30 VLANs for segmentation network, we got the exception.
It occurred because fuel configure neutron with vxlan and gre even in VLAN segmentation case.
I think we need to fill /etc/neutron/plugin.ini in depend on segmentation type
Snapshot is here: https://drive.google.com/a/mirantis.com/file/d/0Bx4ptZV1Jt7hb243RXNNNktNeHc/view?usp=sharing
The root cause is that neurton is configured to use all network types (vlan, gre, vxlan), but tunnel types are not configured on agents.
So when neutron is out of vlans, it creates tunnel network (vxlan), but port creation fails because agents are not configured to support tunneling.
Need to properly configure neutron network types according to deployment type (VLAN/GRE)