functional tests fail due to ping timeout on servers start

Bug #1497508 reported by Venkatesh Sampath
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Low
Venkatesh Sampath

Bug Description

Whenever I try running ‘tox -epy27’, the functional tests continuously fails due to ping timeout while trying to start the servers (glance-api, glance-registry etc.,) for running the tests.

I am running the tests from a VM with 8GB of RAM and 2.7GHz Intel Core i7 Processor.

And I could never make the functional tests to pass until I bump up the timeout value from the current value of 10 secs.

Below is the snippet of exception stack trace captured from the console output.

CONSOLE OUTPUT WITH ERROR STACKTRACE:

venkatesh@vsbox:~/workspace/sf_oswork/repos/glance$ tox -epy27
py27 develop-inst-noop: /media/sf_oswork/repos/glance
py27 installed: aioeventlet==0.4,alembic==0.8.2,amqp==1.4.6,anyjson==0.3.3,appdirs==1.4.0,automaton==0.7.0,Babel==2.0,cachetools==1.1.1,castellan==0.2.1,cffi==1.2.1,contextlib2==0.4.0,coverage==3.7.1,cryptography==1.0.1,debtcollector==0.8.0,decorator==4.0.2,docutils==0.12,enum34==1.0.4,eventlet==0.17.4,extras==0.0.3,fasteners==0.13.0,fixtures==1.3.1,flake8==2.2.4,funcsigs==0.4,functools32==3.2.3.post2,futures==3.0.3,futurist==0.5.0,-e <email address hidden>:openstack/glance.git@cef71f71ded895817eb245cd6aa5519293443d71#egg=glance-gerrit_master,glance-store==0.9.1,greenlet==0.4.9,hacking==0.10.2,httplib2==0.9.1,idna==2.0,ipaddress==1.0.14,iso8601==0.1.10,Jinja2==2.8,jsonschema==2.5.1,keystonemiddleware==2.2.0,kombu==3.0.26,linecache2==1.0.0,Mako==1.0.2,MarkupSafe==0.23,mccabe==0.2.1,mock==1.3.0,monotonic==0.3,mox3==0.10.0,msgpack-python==0.4.6,netaddr==0.7.18,netifaces==0.10.4,networkx==1.10,os-client-config==1.6.3,oslo.concurrency==2.6.0,oslo.config==2.4.0,oslo.context==0.6.0,oslo.db==2.5.0,oslo.i18n==2.6.0,oslo.log==1.11.0,oslo.messaging==2.5.0,oslo.middleware==2.8.0,oslo.policy==0.11.0,oslo.serialization==1.9.0,oslo.service==0.9.0,oslo.utils==2.5.0,oslosphinx==3.2.0,oslotest==1.11.0,osprofiler==0.3.0,Paste==2.0.2,PasteDeploy==1.5.2,pbr==1.7.0,pep8==1.5.7,prettytable==0.7.2,psutil==1.2.1,psycopg2==2.6.1,pyasn1==0.1.8,pycadf==1.1.0,pycparser==2.14,pycrypto==2.6.1,pyflakes==0.8.1,Pygments==2.0.2,PyMySQL==0.6.6,pyOpenSSL==0.15.1,pysendfile==2.0.1,python-editor==0.4,python-keystoneclient==1.7.0,python-mimeparse==0.1.4,python-subunit==1.1.0,pytz==2015.4,PyYAML==3.11,qpid-python==0.26,repoze.lru==0.6,requests==2.7.0,retrying==1.3.3,Routes==2.2,semantic-version==2.4.2,simplegeneric==0.8.1,six==1.9.0,Sphinx==1.2.3,SQLAlchemy==1.0.8,sqlalchemy-migrate==0.10.0,sqlparse==0.1.16,stevedore==1.8.0,taskflow==1.20.0,Tempita==0.5.2,testrepository==0.0.20,testresources==0.2.7,testscenarios==0.5.0,testtools==1.8.0,traceback2==1.4.0,trollius==2.0,unittest2==1.1.0,WebOb==1.4.1,wheel==0.24.0,wrapt==1.10.5,WSME==0.8.0,xattr==0.7.8
py27 runtests: PYTHONHASHSEED='3954332983'
py27 runtests: commands[0] | lockutils-wrapper python setup.py testr --slowest --testr-args=
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ./glance/tests
======================================================================
FAIL: glance.tests.functional.artifacts.test_artifacts.TestArtifacts.test_bad_update_property
tags: worker-0
----------------------------------------------------------------------
registry.log: {{{
2015-09-19 11:00:31.913 2303 DEBUG glance.common.config [-] Loading glance-registry from /tmp/tmpWrfnCY/etc/registry-paste.ini load_paste_app glance/common/config.py:266
2015-09-19 11:00:42.558 2317 DEBUG glance.common.config [-] Loading glance-registry from /tmp/tmpWrfnCY/etc/registry-paste.ini load_paste_app glance/common/config.py:266
2015-09-19 11:00:53.144 2331 DEBUG glance.common.config [-] Loading glance-registry from /tmp/tmpWrfnCY/etc/registry-paste.ini load_paste_app glance/common/config.py:266
}}}

Traceback (most recent call last):
  File "glance/tests/functional/artifacts/test_artifacts.py", line 92, in setUp
    self.start_servers(**self.__dict__.copy())
  File "glance/tests/functional/artifacts/test_artifacts.py", line 181, in start_servers
    super(TestArtifacts, self).start_servers(**kwargs)
  File "glance/tests/functional/__init__.py", line 789, in start_servers
    **kwargs)
  File "glance/tests/functional/__init__.py", line 770, in start_with_retry
    self.assertTrue(launch_msg is None, launch_msg)
  File "/media/sf_oswork/repos/glance/.tox/py27/local/lib/python2.7/site-packages/unittest2/case.py", line 702, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : Unexpected server launch status for: registry,
strace:
======================================================================
FAIL: glance.tests.functional.artifacts.test_artifacts.TestArtifacts.test_create_artifact_bad_dependency_format
tags: worker-0
----------------------------------------------------------------------
registry.log: {{{
2015-09-19 11:01:04.257 2345 DEBUG glance.common.config [-] Loading glance-registry from /tmp/tmpock43V/etc/registry-paste.ini load_paste_app glance/common/config.py:266
2015-09-19 11:01:15.213 2359 DEBUG glance.common.config [-] Loading glance-registry from /tmp/tmpock43V/etc/registry-paste.ini load_paste_app glance/common/config.py:266
2015-09-19 11:01:26.170 2373 DEBUG glance.common.config [-] Loading glance-registry from /tmp/tmpock43V/etc/registry-paste.ini load_paste_app glance/common/config.py:266
}}}

Traceback (most recent call last):
  File "glance/tests/functional/artifacts/test_artifacts.py", line 92, in setUp
    self.start_servers(**self.__dict__.copy())
  File "glance/tests/functional/artifacts/test_artifacts.py", line 181, in start_servers
    super(TestArtifacts, self).start_servers(**kwargs)
  File "glance/tests/functional/__init__.py", line 789, in start_servers
    **kwargs)
  File "glance/tests/functional/__init__.py", line 770, in start_with_retry
    self.assertTrue(launch_msg is None, launch_msg)
  File "/media/sf_oswork/repos/glance/.tox/py27/local/lib/python2.7/site-packages/unittest2/case.py", line 702, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : Unexpected server launch status for: registry,
strace:

Changed in glance:
assignee: nobody → Venkatesh Sampath (venkateshsampath)
status: New → In Progress
summary: - fix functional tests failure due to ping timeout on servers start
+ functional tests fail due to ping timeout on servers start
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/225391

Changed in glance:
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/225391
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=9cb3fa7408b197032da07b88e025e0bad6096a0f
Submitter: Jenkins
Branch: master

commit 9cb3fa7408b197032da07b88e025e0bad6096a0f
Author: Venkatesh Sampath <email address hidden>
Date: Tue Sep 15 08:18:44 2015 +0530

    Fix server start ping timeout for functional tests

    When we run the functional tests using 'tox -epy27',
    the functional tests keeps failing due to ping timeout
    that happens after server starts (api, registry etc.,)
    in order to run the functional tests. Could make the tests
    pass only after increasing the ping timeout.

    This minor patch fixes ping timeout issue by increasing the timeout
    value from 10 to 30 seconds for ensuring consistent success of
    functional tests. Found any value lesser than 30 secs still continue
    to fail the tests.

    Closes-bug: #1497508
    Change-Id: If1c5fc69ae34c976de2a0c5e9e70810b5b994491

Changed in glance:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in glance:
milestone: none → liberty-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in glance:
milestone: liberty-rc1 → 11.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.