Murano environment fails to deploy when using haproxy

[Mick Thompson]

Failing to deploy murano environment with haproxy: [socket.error]: [Errno 111] ECONNREFUSED

2016-10-19 09:45:55.219 20 ERROR murano.common.engine [-]
  socket.error: [Errno 111] ECONNREFUSED
  Traceback (most recent call last):
    File "/tmp/murano-packages-cache/io.murano/0.0.0/03a33ff0d5a3481fbe210382591c68bb/Classes/Environment.yaml", line 102:9 in
        $.applications.pselect($.deploy())
    File "/tmp/murano-packages-cache/com.example.apache.ApacheHttpServer/0.0.0/dce35a43533740e6b45068d5ad599fa3/Classes/Apache.ApacheHttpServer
        $.instance.deploy()
    File "/tmp/murano-packages-cache/io.murano/0.0.0/03a33ff0d5a3481fbe210382591c68bb/Classes/resources/Instance.yaml", line 1
        $.agent.prepare()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/murano/dsl/helpers.py", line 51 in method evaluate
        return value(context)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/murano/dsl/yaql_expression.py", line 85 in method __call__
        return self._parsed_expression.evaluate(context=context)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/expressions.py", line 165 in method evaluate
        return self(utils.NO_VALUE, context, self.engine)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/expressions.py", line 156 in method __call__
        return super(Statement, self).__call__(receiver, context, engine)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/expressions.py", line 37 in method __call__
        return context(self.name, engine, receiver, context)(*self.args)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/contexts.py", line 65 in method <lambda>
        data_context, use_convention, function_filter)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/runner.py", line 49 in method call
        name, all_overloads, engine, receiver, data_context, args, kwargs)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/runner.py", line 117 in method choose_overload
        args = tuple(arg_evaluator(i, arg) for i, arg in enumerate(args))
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/runner.py", line 117 in method <genexpr>
        args = tuple(arg_evaluator(i, arg) for i, arg in enumerate(args))
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/runner.py", line 113 in method <lambda>
        and not isinstance(arg, expressions.Constant))
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/expressions.py", line 37 in method __call__
        return context(self.name, engine, receiver, context)(*self.args)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/contexts.py", line 65 in method <lambda>
        data_context, use_convention, function_filter)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/runner.py", line 51 in method call
        result = delegate()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/runner.py", line 142 in method <lambda>
        return lambda: delegate()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/specs.py", line 341 in method func
        six.iteritems(keyword_args)))
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/murano/dsl/yaql_functions.py", line 167 in method op_dot
        return operator(ctx2, receiver, expr)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/yaqltypes.py", line 323 in method func
        new_name, engine, new_receiver, new_context)(*args, **kwargs)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/contexts.py", line 65 in method <lambda>
        data_context, use_convention, function_filter)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/runner.py", line 51 in method call
        result = delegate()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/runner.py", line 142 in method <lambda>
        return lambda: delegate()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/specs.py", line 341 in method func
        six.iteritems(keyword_args)))
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/standard_library/system.py", line 34 in method op_dot
        return expr(receiver)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/yaqltypes.py", line 269 in method func
        engine, args, kwargs)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/yaqltypes.py", line 236 in method _call
        result = value(receiver, context, engine)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/expressions.py", line 37 in method __call__
        return context(self.name, engine, receiver, context)(*self.args)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/contexts.py", line 65 in method <lambda>
        data_context, use_convention, function_filter)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/runner.py", line 51 in method call
        result = delegate()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/runner.py", line 142 in method <lambda>
        return lambda: delegate()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/specs.py", line 341 in method func
        six.iteritems(keyword_args)))
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/murano/dsl/yaql_integration.py", line 247 in method payload
        executor, __receiver, args, kwargs, __context, True), __context)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/murano/dsl/murano_method.py", line 172 in method invoke
        self, this, context, args, kwargs, skip_stub)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/murano/dsl/executor.py", line 70 in method invoke_method
        method, this, context, args, kwargs, skip_stub=skip_stub)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/murano/dsl/executor.py", line 138 in method _invoke_method
        result = call()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/murano/dsl/executor.py", line 128 in method call
        yaql_engine, context, native_this)(*args, **kwargs)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/specs.py", line 60 in method func
        return self.get_delegate(receiver, engine, context, args, kwargs)()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/yaql/language/specs.py", line 341 in method func
        six.iteritems(keyword_args)))
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/murano/dsl/yaql_integration.py", line 192 in method payload
        return body(__self.extension, *args, **kwargs)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/murano/engine/system/agent.py", line 65 in method prepare
        with common.create_rmq_client() as client:
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/murano/common/messaging/mqclient.py", line 56 in method __enter__
        self.connect()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/murano/common/messaging/mqclient.py", line 64 in method connect
        self._connection.connect()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/kombu/connection.py", line 237 in method connect
        return self.connection
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/kombu/connection.py", line 742 in method connection
        self._connection = self._establish_connection()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/kombu/connection.py", line 697 in method _establish_connection
        conn = self.transport.establish_connection()
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116 in method establish_connection
        conn = self.Connection(**opts)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/amqp/connection.py", line 165 in method __init__
        self.transport = self.Transport(host, connect_timeout, ssl)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/amqp/connection.py", line 186 in method Transport
        return create_transport(host, connect_timeout, ssl)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/amqp/transport.py", line 299 in method create_transport
        return TCPTransport(host, connect_timeout)
    File "/var/lib/kolla/venv/lib/python2.7/site-packages/amqp/transport.py", line 95 in method __init__
        raise socket.error(last_err)

All good when not using haproxy.

[Eduardo Gonzalez (egonzalez90)]

What Operating System are you using? What kolla version?
Can you share globals file and any related configuration option as custom config?
Also murano logs in heka and docker logs.

[Mick Thompson (michael-a-thompson)]

Looks like murano and haproxy are using different ports. I'll patch and push for review.

[Mick Thompson (michael-a-thompson)]

tried reconfiguring port but it didn't work; different error: IOError: Socket closed.

[Christian Berendt (berendt)]

I think this bug is not related to HAProxy. Murano does not use HAProxy to access the RabbitMQ service (https://github.com/openstack/kolla/blob/master/ansible/roles/murano/templates/murano.conf.j2#L11). Like other services it uses direct RabbitMQ connections.

[Mick Thompson (michael-a-thompson)]

I tried that but it didn't work for me. I can only make it work as I outlined.

[Mick Thompson (michael-a-thompson)]

This may be required for us at mo.

[Javier Castillo (javcasalc)]

In my case, I had to:
 - define a new listener in haproxy.cfg for rabbitmq service (mode tcp/bind internal_vip port 5672)
 - define this in murano.conf:

     [rabbitmq]
     host = vip_internal
     port = 5672
     login = openstack
     password = <rabbitmq_password>

[OpenStack Infra (hudson-openstack)]

Change abandoned by Paul Bourke (pbourke) (<email address hidden>) on branch: master
Review: https://review.openstack.org/389571