CI: OVB tripleo-quickstart jobs: pingtest or tempest servers tests fail

Bug #1660627 reported by Sagi (Sergey) Shnaidman on 2017-01-31
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Unassigned

Bug Description

In OVB quickstart jobs pingtest fails.
Sometimes it succeeds, but any other calls to nova fail (like 'nova list'). Also tempest tests with servers booting fail.
Usually it fails with HTTP 504 Error.

http://logs.openstack.org/51/426851/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/6d82087/logs/oooq/collected_logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz

+ nova list
2017-01-30 23:13:23.000 | ERROR (ClientException): Unknown Error (HTTP 504)

504 Errors are in heat logs when deleting the server and trying to connect to nova:
http://logs.openstack.org/51/426851/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/6d82087/logs/oooq/collected_logs/controller-0-tripleo-ci-a-foo/var/log/heat/heat-engine.log.txt.gz

ERROR heat.engine.resource Traceback (most recent call last):
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 763, in _action_recorder
ERROR heat.engine.resource yield
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1697, in delete
ERROR heat.engine.resource *action_args)
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 335, in wrapper
ERROR heat.engine.resource step = next(subtask)
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 810, in action_handler_task
ERROR heat.engine.resource handler_data = handler(*args)
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server.py", line 1465, in handle_delete
ERROR heat.engine.resource return self._delete()
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server.py", line 1450, in _delete
ERROR heat.engine.resource self.client_plugin().ignore_not_found(e)
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 342, in __call__
ERROR heat.engine.resource six.reraise(exc_type, exc_val, traceback)
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/nova/server.py", line 1448, in _delete
ERROR heat.engine.resource self.client().servers.delete(self.resource_id)
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1414, in delete
ERROR heat.engine.resource return self._delete("/servers/%s" % base.getid(server))
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 365, in _delete
ERROR heat.engine.resource resp, body = self.api.client.delete(url)
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 232, in delete
ERROR heat.engine.resource return self.request(url, 'DELETE', **kwargs)
ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 117, in request
ERROR heat.engine.resource raise exceptions.from_response(resp, body, url, method)
ERROR heat.engine.resource ClientException: Unknown Error (HTTP 504)

And in heat-engine:
http://logs.openstack.org/51/426851/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/6d82087/logs/oooq/collected_logs/controller-0-tripleo-ci-a-foo/var/log/heat/heat-engine.log.txt.gz

Not sure if it's related, but a few nova errors are in logs:

On compute:
ERROR nova.compute.manager [req-89692f3f-94f5-4f95-9992-5a7d55776af0 - - - - -] No compute node record for host compute-0-tripleo-ci-a-test.localdomain
http://logs.openstack.org/51/426851/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/6d82087/logs/oooq/collected_logs/compute-0-tripleo-ci-a-test/var/log/nova/nova-compute.log.txt.gz

On controllers:
oslo_db.sqlalchemy.engines [req-007e8f8c-66f4-4382-a642-a784e119bf22 b1122b272a19481894a0365a1e8a1bca 5046ffa25a7d4863840f80fc9b2daf2f - default default] SQL connection failed.
http://logs.openstack.org/51/426851/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/6d82087/logs/oooq/collected_logs/controller-1-tripleo-ci-b-bar/var/log/nova/nova-api.log.txt.gz

Changed in tripleo:
status: New → Triaged
tags: added: quickstart
Changed in tripleo:
importance: Undecided → Medium
milestone: none → ocata-rc1
importance: Medium → High

Should be possibly fixed by https://review.openstack.org/#/c/426837/

Changed in tripleo:
milestone: ocata-rc1 → ocata-rc2
Changed in tripleo:
milestone: ocata-rc2 → pike-1
Changed in tripleo:
milestone: pike-1 → pike-2
Changed in tripleo:
milestone: pike-2 → pike-3
Changed in tripleo:
milestone: pike-3 → pike-rc1
Ronelle Landy (rlandy) wrote :

CI shows ping test working since January 2017.

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers