deployment and snapshot dump failure

[Vladimir Kuklin]

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.thread_1/fuelweb_test/helpers/decorators.py", line 56, in wrapper
    create_diagnostic_snapshot(args[0].env, "fail", func.__name__)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.thread_1/fuelweb_test/helpers/decorators.py", line 156, in create_diagnostic_snapshot
    save_logs(url, os.path.join(settings.LOGS_DIR, log_file_name))
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.thread_1/fuelweb_test/helpers/decorators.py", line 34, in save_logs
    urllib2.urlopen(url).read()
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1146, in do_open
    h = http_class(host, timeout=req.timeout) # will parse host:port
  File "/usr/lib/python2.7/httplib.py", line 693, in __init__
    self._set_hostport(host, port)
  File "/usr/lib/python2.7/httplib.py", line 721, in _set_hostport
    raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])

[Dmitry Pyzhov (dpyzhov)]

Looks like get_admin_node_ip in fuelweb_test returns empty string.

[Vladimir Kuklin (vkuklin)]

also:

2014-05-06 12:33:52,013 - INFO fuel_web_client.py:422 -- current timeouts is 5 count of attempts is 10
2014-05-06 12:33:52,216 - INFO fuel_web_client.py:422 -- current timeouts is 5 count of attempts is 10
2014-05-06 12:33:52,403 - INFO fuel_web_client.py:422 -- current timeouts is 5 count of attempts is 10
2014-05-06 12:33:52,586 - INFO fuel_web_client.py:422 -- current timeouts is 5 count of attempts is 10
2014-05-06 12:33:52,769 - INFO fuel_web_client.py:422 -- current timeouts is 5 count of attempts is 10
2014-05-06 12:33:52,965 - INFO fuel_web_client.py:422 -- current timeouts is 5 count of attempts is 10
2014-05-06 12:33:53,175 - INFO fuel_web_client.py:422 -- current timeouts is 5 count of attempts is 10
2014-05-06 12:33:53,372 - INFO fuel_web_client.py:422 -- current timeouts is 5 count of attempts is 10
2014-05-06 12:33:53,566 - INFO fuel_web_client.py:422 -- current timeouts is 5 count of attempts is 10
2014-05-06 12:33:53,839 - INFO fuel_web_client.py:422 -- current timeouts is 5 count of attempts is 10
2014-05-06 12:33:55,654 - INFO fuel_web_client.py:338 -- Deploy cluster 1
2014-05-06 12:33:55,654 - INFO fuel_web_client.py:356 -- Launch deployment of a cluster #1
2014-05-06 12:34:01,493 - INFO fuel_web_client.py:524 -- Wait for task {u'status': u'error', u'name': u'dump', u'cluster': None, u'result': {}, u'progress': 100, u'message': u"Queue.declare: (406) PRECONDITION_FAILED - parameters for queue 'naily' in vhost '/' not equivalent", u'id': 7, u'uuid': u'49aa6a04-4412-4fcc-9a71-7dc7b4a06b09'} 300 seconds

[Openstack Gerrit (openstack-gerrit)]

Related fix proposed to branch: master
Review: https://review.openstack.org/92602

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/92602
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=61d42a35f25aee3d2351c735fa0619aa04450def
Submitter: Jenkins
Branch: master

commit 61d42a35f25aee3d2351c735fa0619aa04450def
Author: Vladimir Kuklin <email address hidden>
Date: Wed May 7 18:36:05 2014 +0400

    Add checkers for all containers

    check containers services status
    1) for postgres do select 1 in nailgun
    2) for rabbitmq use AMQP management plugin
    and fetch nodes list
    3) for other containers: check listening on ports
    or check processes in process list
    4) add check function to dockerctl for convinience
    5) add check function call to container start
    Closes-Bug: #1315940
    Related-Bug: #1316044

    Change-Id: Id21a75718b4ad789c18aa248efca8bf16c90f6b8

[Vladimir Kuklin (vkuklin)]

was not reproduced further. marking as incomplete

[Dmitry Borodaenko (angdraug)]

I've reproduced it with 5.0.1 ISO #110.

2014-07-09 01:54:10.220 DEBUG [7f042556d700] (logger) Request PUT /api/clusters/1/changes from 172.17.42.1:59202 {}
2014-07-09 01:54:12.001 ERROR [7f042556d700] (logger) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 420, in _delegate
    return handle_class(cls)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 396, in handle_class
    return tocall(*args)
  File "<string>", line 2, in PUT
  File "/usr/lib/python2.6/site-packages/nailgun/api/handlers/base.py", line 95, in content_json
    data = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nailgun/api/handlers/base.py", line 367, in PUT
    task = task_manager.execute()
  File "/usr/lib/python2.6/site-packages/nailgun/task/manager.py", line 230, in execute
    rpc.cast('naily', task_messages)
  File "/usr/lib/python2.6/site-packages/nailgun/rpc/__init__.py", line 86, in cast
    declare=[use_queue])
  File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 168, in publish
    routing_key, mandatory, immediate, exchange, declare)
  File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 180, in _publish
    [maybe_declare(entity) for entity in declare]
  File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 111, in maybe_declare
    return maybe_declare(entity, self.channel, retry, **retry_policy)
  File "/usr/lib/python2.6/site-packages/kombu/common.py", line 99, in maybe_declare
    return _maybe_declare(entity)
  File "/usr/lib/python2.6/site-packages/kombu/common.py", line 110, in _maybe_declare
    entity.declare()
  File "/usr/lib/python2.6/site-packages/kombu/entity.py", line 505, in declare
    self.queue_declare(nowait, passive=False)
  File "/usr/lib/python2.6/site-packages/kombu/entity.py", line 531, in queue_declare
    nowait=nowait)
  File "/usr/lib/python2.6/site-packages/amqp/channel.py", line 1258, in queue_declare
    (50, 11), # Channel.queue_declare_ok
  File "/usr/lib/python2.6/site-packages/amqp/abstract_channel.py", line 69, in wait
    return self.dispatch_method(method_sig, args, content)
  File "/usr/lib/python2.6/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
    return amqp_method(self, args)
  File "/usr/lib/python2.6/site-packages/amqp/channel.py", line 241, in _close
    reply_code, reply_text, (class_id, method_id), ChannelError,
PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - parameters for queue 'naily' in vhost '/' not equivalent

2014-07-09 01:54:12.005 ERROR [7f042556d700] (logger) Response code '500 Internal Server Error' for PUT /api/clusters/1/changes from 172.17.42.1:59202

[Dmitry Borodaenko (angdraug)]

Could be related to a similar problem with Astute:
https://bugs.launchpad.net/fuel/+bug/1319451

[Dmitry Pyzhov (dpyzhov)]

Your case is not releated to #1316044. Closing this bug and reopening bug with amqp