[murano] Unable to connect RabbitMQ socket.error: [Errno 111] ECONNREFUSED

[Alexey Deryugin]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "648"
  build_id: "2015-11-17_10-32-30"
  fuel-nailgun_sha: "25f7d417777e0ca12c80d9e8b56455906018380f"
  python-fuelclient_sha: "e685d68c1c0d0fa0491a250f07d9c3a8d0f9608c"
  fuel-agent_sha: "6faa1e0ba836ef114b3c1a6c4d12469fc66ae402"
  fuel-nailgun-agent_sha: "3e9d17211d65c80bf97c8d83979979f6c7feb687"
  astute_sha: "959b06c5ef8143125efd1727d350c050a922eb12"
  fuel-library_sha: "f443e173aaa83ff2791635b8450429d3b847df84"
  fuel-ostf_sha: "9690a2de829d3b063ed1e64b0b10dde39f711dc0"
  fuel-createmirror_sha: "22a7aacd95bbdca69f9e0f08b70facabdec8fb28"
  fuelmenu_sha: "06bbcebed6c8d0b0f9279e2997d2f958c800e98c"
  shotgun_sha: "c377d163519f6d10b69a654019d6086ba5f14edc"
  network-checker_sha: "a57e1d69acb5e765eb22cab0251c589cd76f51da"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "266f9b374934c64629a84b5146632cc0de70ac91"

Murano fails deployment with the following error:

  socket.error: [Errno 111] ECONNREFUSED
  Traceback (most recent call last):
    File "/tmp/murano-packages-cache/f2a7b90485084ab6afe212017b7e8c42/612e7b0c85864beaae017c41f35f98d2/Classes/Environment.yaml", line 70:9 in method deploy of class io.mura
no.Environment
        $.applications.pselect($.deploy())
    File "/tmp/murano-packages-cache/f2a7b90485084ab6afe212017b7e8c42/b9e650bb9f7e42258e51586c9213cb66/Classes/Simple.yaml", line 28:13 in method deploy of class io.murano.a
pps.Simple
        $.instance.deploy()
    File "/tmp/murano-packages-cache/f2a7b90485084ab6afe212017b7e8c42/612e7b0c85864beaae017c41f35f98d2/Classes/resources/Instance.yaml", line 105:9 in method deploy of class
 io.murano.resources.Instance
        $.agent.prepare()
    File "/usr/lib/python2.7/dist-packages/murano/dsl/helpers.py", line 45 in method evaluate
        return value(context)
    File "/usr/lib/python2.7/dist-packages/murano/dsl/yaql_expression.py", line 82 in method __call__
        return self._parsed_expression.evaluate(context=context)
    File "/usr/lib/python2.7/dist-packages/yaql/language/expressions.py", line 165 in method evaluate
        return self(utils.NO_VALUE, context, self.engine)
    File "/usr/lib/python2.7/dist-packages/yaql/language/expressions.py", line 156 in method __call__
        return super(Statement, self).__call__(receiver, context, engine)
    File "/usr/lib/python2.7/dist-packages/yaql/language/expressions.py", line 37 in method __call__
        return context(self.name, engine, receiver, context)(*self.args)
    File "/usr/lib/python2.7/dist-packages/yaql/language/contexts.py", line 65 in method <lambda>
        data_context, use_convention, function_filter)
    File "/usr/lib/python2.7/dist-packages/yaql/language/runner.py", line 49 in method call
        name, all_overloads, engine, receiver, data_context, args, kwargs)
    File "/usr/lib/python2.7/dist-packages/yaql/language/runner.py", line 117 in method choose_overload
        args = tuple(arg_evaluator(i, arg) for i, arg in enumerate(args))
    File "/usr/lib/python2.7/dist-packages/yaql/language/runner.py", line 117 in method <genexpr>
        args = tuple(arg_evaluator(i, arg) for i, arg in enumerate(args))
    File "/usr/lib/python2.7/dist-packages/yaql/language/runner.py", line 113 in method <lambda>
        and not isinstance(arg, expressions.Constant))
    File "/usr/lib/python2.7/dist-packages/yaql/language/expressions.py", line 37 in method __call__
        return context(self.name, engine, receiver, context)(*self.args)
    File "/usr/lib/python2.7/dist-packages/yaql/language/contexts.py", line 65 in method <lambda>
        data_context, use_convention, function_filter)
    File "/usr/lib/python2.7/dist-packages/yaql/language/runner.py", line 51 in method call
        result = delegate()
    File "/usr/lib/python2.7/dist-packages/yaql/language/runner.py", line 142 in method <lambda>
        return lambda: delegate()
    File "/usr/lib/python2.7/dist-packages/yaql/language/specs.py", line 341 in method func
        six.iteritems(keyword_args)))
    File "/usr/lib/python2.7/dist-packages/murano/dsl/yaql_functions.py", line 140 in method op_dot
        return operator(ctx2, sender, expr)
    File "/usr/lib/python2.7/dist-packages/yaql/language/yaqltypes.py", line 290 in method func
        new_name, engine, new_receiver, new_context)(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/yaql/language/contexts.py", line 65 in method <lambda>
        data_context, use_convention, function_filter)
    File "/usr/lib/python2.7/dist-packages/yaql/language/runner.py", line 51 in method call
        result = delegate()
    File "/usr/lib/python2.7/dist-packages/yaql/language/runner.py", line 142 in method <lambda>
        return lambda: delegate()
    File "/usr/lib/python2.7/dist-packages/yaql/language/specs.py", line 341 in method func
        six.iteritems(keyword_args)))
    File "/usr/lib/python2.7/dist-packages/yaql/standard_library/system.py", line 34 in method op_dot
        return expr(receiver)
    File "/usr/lib/python2.7/dist-packages/yaql/language/yaqltypes.py", line 236 in method func
        engine, args, kwargs)
    File "/usr/lib/python2.7/dist-packages/yaql/language/yaqltypes.py", line 205 in method _call
        result = value(receiver, context, engine)
    File "/usr/lib/python2.7/dist-packages/yaql/language/expressions.py", line 37 in method __call__
        return context(self.name, engine, receiver, context)(*self.args)
    File "/usr/lib/python2.7/dist-packages/yaql/language/contexts.py", line 65 in method <lambda>
        data_context, use_convention, function_filter)
    File "/usr/lib/python2.7/dist-packages/yaql/language/runner.py", line 51 in method call
        result = delegate()
    File "/usr/lib/python2.7/dist-packages/yaql/language/runner.py", line 142 in method <lambda>
        return lambda: delegate()
    File "/usr/lib/python2.7/dist-packages/yaql/language/specs.py", line 341 in method func
        six.iteritems(keyword_args)))
    File "/usr/lib/python2.7/dist-packages/murano/dsl/yaql_integration.py", line 189 in method payload
        executor, __sender, args, kwargs, __context, True), __context)
    File "/usr/lib/python2.7/dist-packages/murano/dsl/murano_method.py", line 116 in method invoke
        context, args, kwargs, skip_stub)
    File "/usr/lib/python2.7/dist-packages/murano/dsl/executor.py", line 113 in method invoke_method
        result = call()
    File "/usr/lib/python2.7/dist-packages/murano/dsl/executor.py", line 105 in method call
        yaql_engine, context, native_this)(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/yaql/language/specs.py", line 60 in method func
        return self.get_delegate(receiver, engine, context, args, kwargs)()
    File "/usr/lib/python2.7/dist-packages/yaql/language/specs.py", line 341 in method func
        six.iteritems(keyword_args)))
    File "/usr/lib/python2.7/dist-packages/murano/dsl/yaql_integration.py", line 165 in method payload
        return body(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/murano/engine/system/agent.py", line 70 in method prepare
        with common.create_rmq_client() as client:
    File "/usr/lib/python2.7/dist-packages/murano/common/messaging/mqclient.py", line 48 in method __enter__
        self.connect()
    File "/usr/lib/python2.7/dist-packages/murano/common/messaging/mqclient.py", line 56 in method connect
        self._connection.connect()
    File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 237 in method connect
        return self.connection
    File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 741 in method connection
        self._connection = self._establish_connection()
    File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 696 in method _establish_connection
        conn = self.transport.establish_connection()
    File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 116 in method establish_connection
        conn = self.Connection(**opts)
    File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 165 in method __init__
        self.transport = self.Transport(host, connect_timeout, ssl)
    File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 186 in method Transport
        return create_transport(host, connect_timeout, ssl)
    File "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 299 in method create_transport
        return TCPTransport(host, connect_timeout)
    File "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 95 in method __init__
        raise socket.error(last_err)

[Anastasia Kuznetsova (akuznetsova)]

Also need to fix IP address of rabbitmq host that is used by murano-agent, need to use some public address, otherwise I get connection error in murano-agent log.

[Timur Nurlygayanov (tnurlygayanov)]

Please, add link to review if it is "in progress". Status changed to Confirmed because we have not fixes on review.

[Alexey Deryugin (velovec)]

Fix on review: https://review.openstack.org/#/c/247494

[ruhe (ruhe)]

This bug is super critical. Murano is not operational without the fix. Deployment of separate RabbitMQ instance was removed from upstream Puppet manifests and substituted by vhost in standard RabbitMQ. But that RabbitMQ cannot be accessed from VMs. We need to deploy separate RabbitMQ for Murano as it was before.

- Moving to Critical

[Alexey Deryugin (velovec)]

Fix on review: https://review.openstack.org/#/c/255980/

[Timur Nurlygayanov (tnurlygayanov)]

Reproduced on my lab,

if we will run on controller node:

root@node-4:~# haproxy-status.sh | grep DOWN
murano_rabbitmq node-5 Status: DOWN/L4TOUT Sessions: 0 Rate: 0
murano_rabbitmq node-4 Status: DOWN/L4CON Sessions: 0 Rate: 0
murano_rabbitmq node-3 Status: DOWN/L4TOUT Sessions: 0 Rate: 0
murano_rabbitmq BACKEND Status: DOWN Sessions: 0 Rate: 0

[Ivan Berezovskiy (iberezovskiy)]

https://review.openstack.org/#/c/255980/ is merged

[Anastasia Kuznetsova (akuznetsova)]

Verified on
ERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "328"
  build_id: "328"
  fuel-nailgun_sha: "ec25ae8df28e1f1d87da653c5aab2711dff729f6"
  python-fuelclient_sha: "7c7a756fb6a3d091851c060003a2965c67aa353d"
  fuel-agent_sha: "d354cbe61b055db848a15ce66fb7ae92178d2c0a"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "c56dfde2da034151a7e707b381c4cf9d213b4ba2"
  fuel-library_sha: "bcc3d042a47547f6ad826360a85ef21dcaf25882"
  fuel-ostf_sha: "b2ebf15a3530b5c7b57707acf8642c1c3ac71bd8"
  fuel-mirror_sha: "8100acb3a566358d0d4ecc66de32d39626263028"
  fuelmenu_sha: "2942a85796d37f09ba8c8c6d762d8813292cf0d4"
  shotgun_sha: "cacb93cbc28910ff0dc38f30a855efa9af50d8ce"
  network-checker_sha: "d443ef47abeda58d319bc8d33d5005dd09440a02"
  fuel-upgrade_sha: "718aa3d7021fee2970f0fa6791cf5188578cc516"
  fuelmain_sha: "3faa824728ce60734abe602ff3778976f8a16eed"

[Roman Podoliaka (rpodolyaka)]

8.0 specific, 7.0 is not affected.