[vmware_dvs] Fuel Health check tests have failed after deploy cluster with ceilometer

Bug #1484029 reported by Olesia Tsvigun
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel VMware DVS plugin
Fix Released
High
Igor Gajsin
vmware-dvs
Fix Released
Undecided
Andrzej Skupień

Bug Description

Description
Following OSTF test cases are failed 'Ceilometer test to check alarm state and get Nova metrics' and 'Ceilometer test to check notifications from Neutron'

Steps to reproduce:
1.Install DVS plugin on master node.
2.Create a new environment using the Fuel UI Wizard.
add name of env and select release version with OS
as hypervisor type: select vcenter check box and QEMU/KVM radio button
network setup : Neutron with Vlan segmentation.
storage backends: default
additional services: install ceilometer
3.In Settings tab:
enable DVS plugin
set dvSwitch name 'dvSwitch'
4.Add nodes:
3 controller + mongo
1 compute
5.Interfaces on slaves should be setup this way in Fuel interface:
eth0 - admin(PXE)
eth1 - public
eth2 - management
eth3 - VM(Fixed) ID:103
eth4 – storage
6. In Networks tab:
Public network: start'10.109.1.2' end '10.109.1.127'
CIDR '10.109.1.0/24'
Gateway '10.109.1.1'
Floating ip range start'10.109.1.128' end '10.109.1.254'
Storage: CIDR '10.109.4.0/24'
Vlan tag is not set-Management: CIDR '10.109.2.0/24'
Vlan tag is not set
Neutron L2 configuration by default
Neutron L3 configuration by default
7.Verify networks.
8. Fill vcenter credentials:
Availability zone: vcenter
vCenter host: '172.16.0.254'
vCenter username: <email address hidden>
vCenter password: Qwer!1234
9.Add 1 vSphera Clusters:
vSphera Cluster: Cluster1
Service name: vmcluster1
Datastore regex:.*
10.Deploy cluster
11.Run OSTF

Actual Result
Following test cases have failed;
 'Ceilometer test to check alarm state and get Nova metrics' with error 'Failed to get to expected status. In error state. Please refer to OpenStack logs for more details.'
'Ceilometer test to check notifications from Neutron with error'Failed to create some Neutron resources. Please refer to OpenStack logs for more details.'

Expected result:
All OSTF should be passed.

Additional info:
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'

id | name | version | package_version
---|------------------------|---------|----------------
1 | fuel-plugin-vmware-dvs | 1.0.0 | 2.0.0

Tags: dvs
Revision history for this message
Olesia Tsvigun (otsvigun) wrote :
Changed in fuel-plugins:
importance: Undecided → High
assignee: nobody → Igor Gajsin (igajsin)
milestone: none → 6.1
tags: added: dvs
summary: [vmware_dvs] Fuel Health check tests have failed after deploy cluster
- with ceilometr..
+ with ceilometr.
Revision history for this message
Olesia Tsvigun (otsvigun) wrote : Re: [vmware_dvs] Fuel Health check tests have failed after deploy cluster with ceilometr.

Also following OSTF test cases have faild too:
{u'Check stack autoscaling (failure)': u'Failed to get to expected status. In CREATE_FAILED state. Please refer to OpenStack logs for more details.'},
{u'Update stack actions: inplace, replace and update whole template (failure)': u'Failed to get to expected status. In CREATE_FAILED state. Please refer to OpenStack logs for more details.'}]

Revision history for this message
Olesia Tsvigun (otsvigun) wrote :
Revision history for this message
Igor Gajsin (igajsin) wrote :
Download full text (3.7 KiB)

paste from /var/log/neutron/sever.log when ostf-test "Ceilometer test to check notifications from Neutron" failed.

2015-08-28 15:54:43.077 31075 ERROR neutron.plugins.ml2.managers [req-847850be-44f6-441f-8bcb-2c0af7393695 None] Mechanism driver 'vmware_dvs' failed in update_network_precommit
2015-08-28 15:54:43.077 31075 TRACE neutron.plugins.ml2.managers Traceback (most recent call last):
2015-08-28 15:54:43.077 31075 TRACE neutron.plugins.ml2.managers File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 291, in _call_on_drivers
2015-08-28 15:54:43.077 31075 TRACE neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
2015-08-28 15:54:43.077 31075 TRACE neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/mech_vmware_dvs/driver.py", line 86, in update_network_precommit
2015-08-28 15:54:43.077 31075 TRACE neutron.plugins.ml2.managers dvs.update_network(context.current)
2015-08-28 15:54:43.077 31075 TRACE neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/mech_vmware_dvs/util.py", line 80, in update_network
2015-08-28 15:54:43.077 31075 TRACE neutron.plugins.ml2.managers pg_ref = self._get_pg_by_name(name)
2015-08-28 15:54:43.077 31075 TRACE neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/mech_vmware_dvs/util.py", line 277, in _get_pg_by_name
2015-08-28 15:54:43.077 31075 TRACE neutron.plugins.ml2.managers raise exceptions.PortGroupNotFound(pg_name=pg_name)
2015-08-28 15:54:43.077 31075 TRACE neutron.plugins.ml2.managers PortGroupNotFound: Port Group ceilo-net-update542478196-a625ee31-116a-4fd3-9ead-e9d16d471479 not found
2015-08-28 15:54:43.077 31075 TRACE neutron.plugins.ml2.managers
2015-08-28 15:54:43.082 31075 ERROR neutron.api.v2.resource [req-847850be-44f6-441f-8bcb-2c0af7393695 None] update failed
2015-08-28 15:54:43.082 31075 TRACE neutron.api.v2.resource Traceback (most recent call last):
2015-08-28 15:54:43.082 31075 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 87, in resource
2015-08-28 15:54:43.082 31075 TRACE neutron.api.v2.resource result = method(request=request, **args)
2015-08-28 15:54:43.082 31075 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 538, in update
2015-08-28 15:54:43.082 31075 TRACE neutron.api.v2.resource obj = obj_updater(request.context, id, **kwargs)
2015-08-28 15:54:43.082 31075 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 544, in update_network
2015-08-28 15:54:43.082 31075 TRACE neutron.api.v2.resource self.mechanism_manager.update_network_precommit(mech_context)
2015-08-28 15:54:43.082 31075 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", line 343, in update_network_precommit
2015-08-28 15:54:43.082 31075 TRACE neutron.api.v2.resource self._call_on_drivers("update_network_precommit", context)
2015-08-28 15:54:43.082 31075 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/managers.py", l...

Read more...

Changed in fuel-plugins:
status: New → Confirmed
status: Confirmed → Triaged
Revision history for this message
Igor Gajsin (igajsin) wrote :

It is a docker-ostf.log

urllib3.connectionpool: DEBUG: "PUT http://172.16.0.2:9696//v2.0/networks/3cebfe8b-201d-460b-b917-4dee452324cf.json HTTP/1.1" 500 111
neutronclient.client: DEBUG: RESP:500 CaseInsensitiveDict({'content-length': '111', 'via': '1.1 apache_api_proxy:9696', 'server': 'Apache/2.4.7 (Ubuntu)', 'connection': 'close', 'date': 'Mon, 31 Aug 2015 11:32:14 GMT', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-e79400b2-2c1e-46ec-9f37-5e652430a360'}) {"NeutronError": {"message": "update_network_precommit failed.", "type": "MechanismDriverError", "detail": ""}}

neutronclient.v2_0.client: DEBUG: Error message: {"NeutronError": {"message": "update_network_precommit failed.", "type": "MechanismDriverError", "detail": ""}}
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/common/test_mixins.py", line 176, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/fuel_health/ceilometermanager.py", line 249, in neutron_helper
    net["id"], {"network": {"name": rand_name("ceilo-net-update")}})
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 354, in update_network
    return self.put(self.network_path % (network), body=body)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1329, in put
    headers=headers, params=params)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1297, in retry_request
    headers=headers, params=params)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1251, in do_request
    self._handle_fault_response(status_code, replybody)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1216, in _handle_fault_response
    exception_handler_v20(status_code, des_error_body)
  File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 66, in exception_handler_v20
    status_code=status_code)
InternalServerError: update_network_precommit failed.

--------------------- >> end captured logging << ---------------------

Revision history for this message
Igor Gajsin (igajsin) wrote :

same error in the /var/log/neutron/server.log after restart ceilometer's corosync resources

http://paste.openstack.org/show/437121/

Changed in vmware-dvs:
assignee: nobody → Andrzej Skupień (kendriu)
Changed in vmware-dvs:
status: New → In Progress
Revision history for this message
Andrzej Skupień (kendriu) wrote :

I have manage to have all OSTF green. There have to be made some changes in fuel plugin.
In neutron.conf is right now:
notification_topics=vmware_dvs

but should be:
notification_topics=notifications,vmware_dvs

Changed in vmware-dvs:
status: In Progress → Fix Released
Revision history for this message
Olesia Tsvigun (otsvigun) wrote :

Continue failing :
{u'Advanced stack actions: suspend, resume and check (failure)': u'Failed to get to expected status. In CREATE_FAILED state. Please refer to OpenStack logs for more details.'}, {u'Update stack actions: inplace, replace and update whole template (failure)': u'Failed to get to expected status. In UPDATE_FAILED state. Please refer to OpenStack logs for more details.'}]

Revision history for this message
Olesia Tsvigun (otsvigun) wrote :
summary: [vmware_dvs] Fuel Health check tests have failed after deploy cluster
- with ceilometr.
+ with ceilometer
Igor Gajsin (igajsin)
Changed in fuel-plugins:
status: Triaged → Fix Committed
Revision history for this message
Igor Gajsin (igajsin) wrote :
Revision history for this message
Olesia Tsvigun (otsvigun) wrote :
Changed in fuel-plugins:
status: Fix Committed → Fix Released
affects: fuel-plugins → fuel-plugin-vmware-dvs
Changed in fuel-plugin-vmware-dvs:
milestone: 6.1 → none
milestone: none → 1.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.