Bug #1445422 “Received error response from Keystone server at ht...” : Bugs : Fuel for OpenStack

Anastasia Palkina (apalkina) on 2015-04-17

description:

updated

Anastasia Palkina (apalkina) on 2015-04-17

description:

updated

Vladimir Kuklin (vkuklin) on 2015-04-17

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Igor Yozhikov (iyozhikov)
status:	New → Confirmed

ruhe (ruhe) on 2015-04-17

tags:

added: heat

Vasyl Saienko (vsaienko) on 2015-04-17

Changed in fuel:
assignee:	Igor Yozhikov (iyozhikov) → Vasyl Saienko (vsaienko)

Revision history for this message

Dmitry Ilyin (idv1985) wrote on 2015-04-17:

#1

It looks like Keystone have broken for some reason? Why would it return Gateway Timeout (HTTP 504)?
Can anyone reproduce this?

Revision history for this message

Vasyl Saienko (vsaienko) wrote on 2015-04-17:

#2

Download full text (29.4 KiB)

According to provided logs servers were overloaded, 2GB is not enough for normal MOS work, especially on controller nodes.
Rabbit/mysql were down during ostf tests.

root@node-9:~# free -m
total used free shared buffers cached
Mem: 2001 1917 84 58 3 124
-/+ buffers/cache: 1789 212
Swap: 4015 1310 2705

There is logs from cluster_6_smoke.log:

urllib3.connectionpool: DEBUG: "POST http://172.16.0.14:5000/v2.0/tokens HTTP/1.1" 504 None
keystoneclient.session: DEBUG: Request returned failure status: 504
fuel_health.nmanager: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 81, in __init__
    self.identity_client = self._get_identity_client()
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 216, in _get_identity_client
    insecure=dscv)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 147, in __init__
    self.authenticate()
  File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 425, in authenticate
    resp = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 190, in get_raw_token_from_identity_service
    "%s" % e)
AuthorizationFailure: Authorization Failed: Gateway Timeout (HTTP 504)

fuel_health.nmanager: DEBUG: Unable to initialize Keystone client: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 81, in __init__
    self.identity_client = self._get_identity_client()
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 216, in _get_identity_client
    insecure=dscv)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 147, in __init__
    self.authenticate()
  File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 425, in authenticate
    resp = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 190, in get_raw_token_from_identity_service
    "%s" % e)
AuthorizationFailure: Authorization Failed: Gateway Timeout (HTTP 504)

[root@fuel ostf]# ls -lah cluster_6_smoke.log
-rw-r--r-- 1 root root 129K Apr 17 09:37 cluster_6_smoke.log

Rabbit was down in that period of time:

<45>Apr 17 09:35:51 node-11 rabbitmq-sasl pid: <0.9446.6>
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl registered_name: []
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl exception exit: channel_termination_timeout
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl in function rabbit_reader:wait_for_channel_termination/3 (src/rabbit_reader.erl, line 568)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl in call from rabbit_reader:handle_exception/3 (src/rabbit_reader.erl, line 596)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl ...

According to provided logs servers were overloaded, 2GB is not enough for normal MOS work, especially on controller nodes.
Rabbit/mysql were down during ostf tests.

root@node-9:~# free -m
             total       used       free     shared    buffers     cached
Mem:          2001       1917         84         58          3        124
-/+ buffers/cache:       1789        212
Swap:         4015       1310       2705

There is logs from cluster_6_smoke.log:

urllib3.connectionpool: DEBUG: "POST http://172.16.0.14:5000/v2.0/tokens HTTP/1.1" 504 None
keystoneclient.session: DEBUG: Request returned failure status: 504
fuel_health.nmanager: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 81, in __init__
    self.identity_client = self._get_identity_client()
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 216, in _get_identity_client
    insecure=dscv)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 147, in __init__
    self.authenticate()
  File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 425, in authenticate
    resp = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 190, in get_raw_token_from_identity_service
    "%s" % e)
AuthorizationFailure: Authorization Failed: Gateway Timeout (HTTP 504)

fuel_health.nmanager: DEBUG: Unable to initialize Keystone client: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 81, in __init__
    self.identity_client = self._get_identity_client()
  File "/usr/lib/python2.6/site-packages/fuel_health/nmanager.py", line 216, in _get_identity_client
    insecure=dscv)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 147, in __init__
    self.authenticate()
  File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 425, in authenticate
    resp = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 190, in get_raw_token_from_identity_service
    "%s" % e)
AuthorizationFailure: Authorization Failed: Gateway Timeout (HTTP 504)

[root@fuel ostf]# ls -lah cluster_6_smoke.log 
-rw-r--r-- 1 root root 129K Apr 17 09:37 cluster_6_smoke.log

Rabbit was down in that period of time:

<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    pid: <0.9446.6>
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    registered_name: []
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    exception exit: channel_termination_timeout
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in function  rabbit_reader:wait_for_channel_termination/3 (src/rabbit_reader.erl, line 568)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:handle_exception/3 (src/rabbit_reader.erl, line 596)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:terminate/2 (src/rabbit_reader.erl, line 433)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:handle_other/2 (src/rabbit_reader.erl, line 360)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:mainloop/4 (src/rabbit_reader.erl, line 330)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:run/1 (src/rabbit_reader.erl, line 281)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:start_connection/5 (src/rabbit_reader.erl, line 255)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    ancestors: [<0.9441.6>,rabbit_tcp_client_sup,rabbit_sup,<0.763.0>]
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    messages: [{'EXIT',#Port<0.23999>,normal}]
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    links: []
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    dictionary: [{{channel,1},
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl                   {<0.9477.6>,{method,rabbit_framing_amqp_0_9_1}}},
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl                  {process_name,
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl                      {rabbit_reader,<<"127.0.0.1:36383 -> 127.0.0.1:5673">>}},
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl                  {{ch_pid,<0.9477.6>},{1,#Ref<0.0.5.89264>}}]
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    trap_exit: true
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    status: running
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    heap_size: 1598
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    stack_size: 27
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    reductions: 7119
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl  neighbours:
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl =CRASH REPORT==== 17-Apr-2015::09:35:42 ===
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl  crasher:
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    initial call: rabbit_reader:init/2
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    pid: <0.9363.6>
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    registered_name: []
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    exception exit: channel_termination_timeout
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in function  rabbit_reader:wait_for_channel_termination/3 (src/rabbit_reader.erl, line 568)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:handle_exception/3 (src/rabbit_reader.erl, line 596)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:terminate/2 (src/rabbit_reader.erl, line 433)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:handle_other/2 (src/rabbit_reader.erl, line 360)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:mainloop/4 (src/rabbit_reader.erl, line 330)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:run/1 (src/rabbit_reader.erl, line 281)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:start_connection/5 (src/rabbit_reader.erl, line 255)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    ancestors: [<0.9361.6>,rabbit_tcp_client_sup,rabbit_sup,<0.763.0>]
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    messages: [{'EXIT',#Port<0.23989>,normal}]
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    links: []
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    dictionary: [{{channel,1},
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl                   {<0.9369.6>,{method,rabbit_framing_amqp_0_9_1}}},
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl                  {process_name,
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl                      {rabbit_reader,
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl                          <<"192.168.0.19:60394 -> 192.168.0.21:5673">>}},
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl                  {{ch_pid,<0.9369.6>},{1,#Ref<0.0.5.88483>}}]
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    trap_exit: true
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    status: running
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    heap_size: 1598
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    stack_size: 27
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    reductions: 5986
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl  neighbours:
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl =CRASH REPORT==== 17-Apr-2015::09:35:42 ===
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl  crasher:
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    initial call: rabbit_reader:init/2
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    pid: <0.9833.6>
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    registered_name: []
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    exception exit: channel_termination_timeout
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in function  rabbit_reader:wait_for_channel_termination/3 (src/rabbit_reader.erl, line 568)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:handle_exception/3 (src/rabbit_reader.erl, line 596)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:terminate/2 (src/rabbit_reader.erl, line 433)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:handle_other/2 (src/rabbit_reader.erl, line 360)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:mainloop/4 (src/rabbit_reader.erl, line 330)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:run/1 (src/rabbit_reader.erl, line 281)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl      in call from rabbit_reader:start_connection/5 (src/rabbit_reader.erl, line 255)
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    ancestors: [<0.9824.6>,rabbit_tcp_client_sup,rabbit_sup,<0.763.0>]
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    messages: [handshake_timeout,{'EXIT',#Port<0.24063>,normal}]
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    links: []
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl    dictionary: [{{channel,1},
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl                   {<0.9862.6>,{method,rabbit_framing_amqp_0_9_1}}},
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl                  {{ch_pid,<0.9862.6>},{1,#Ref<0.0.5.93542>}},
<45>Apr 17 09:35:51 node-11 rabbitmq-sasl                  {process_name,
<45>Apr 17 09:35:41 node-11 rabbitmq AMQP connection <0.9757.6> (running), channel 0 - error:
<45>Apr 17 09:35:41 node-11 rabbitmq {amqp_error,connection_forced,
<45>Apr 17 09:35:41 node-11 rabbitmq            "broker forced connection closure with reason 'shutdown'",none}
<45>Apr 17 09:35:41 node-11 rabbitmq =ERROR REPORT==== 17-Apr-2015::09:35:40 ===
<45>Apr 17 09:35:41 node-11 rabbitmq AMQP connection <0.9833.6> (running), channel 0 - error:
<45>Apr 17 09:35:41 node-11 rabbitmq {amqp_error,connection_forced,
<45>Apr 17 09:35:41 node-11 rabbitmq            "broker forced connection closure with reason 'shutdown'",none}
<45>Apr 17 09:35:41 node-11 rabbitmq =ERROR REPORT==== 17-Apr-2015::09:35:40 ===
<45>Apr 17 09:35:41 node-11 rabbitmq AMQP connection <0.10015.6> (running), channel 0 - error:
<45>Apr 17 09:35:41 node-11 rabbitmq {amqp_error,connection_forced,
<45>Apr 17 09:35:41 node-11 rabbitmq            "broker forced connection closure with reason 'shutdown'",none}
<45>Apr 17 09:35:41 node-11 rabbitmq =ERROR REPORT==== 17-Apr-2015::09:35:40 ===
<45>Apr 17 09:35:41 node-11 rabbitmq AMQP connection <0.2884.6> (running), channel 0 - error:
<45>Apr 17 09:35:41 node-11 rabbitmq {amqp_error,connection_forced,
<45>Apr 17 09:35:41 node-11 rabbitmq            "broker forced connection closure with reason 'shutdown'",none}
<45>Apr 17 09:35:41 node-11 rabbitmq =ERROR REPORT==== 17-Apr-2015::09:35:40 ===
<45>Apr 17 09:35:41 node-11 rabbitmq AMQP connection <0.8606.6> (running), channel 0 - error:
<45>Apr 17 09:35:41 node-11 rabbitmq {amqp_error,connection_forced,
<45>Apr 17 09:35:41 node-11 rabbitmq            "broker forced connection closure with reason 'shutdown'",none}
<45>Apr 17 09:35:41 node-11 rabbitmq =ERROR REPORT==== 17-Apr-2015::09:35:40 ===
<45>Apr 17 09:35:41 node-11 rabbitmq AMQP connection <0.9363.6> (running), channel 0 - error:
<45>Apr 17 09:35:41 node-11 rabbitmq {amqp_error,connection_forced,
<45>Apr 17 09:35:41 node-11 rabbitmq            "broker forced connection closure with reason 'shutdown'",none}
<45>Apr 17 09:35:41 node-11 rabbitmq =ERROR REPORT==== 17-Apr-2015::09:35:40 ===
<45>Apr 17 09:35:41 node-11 rabbitmq AMQP connection <0.9446.6> (running), channel 0 - error:
<45>Apr 17 09:35:41 node-11 rabbitmq {amqp_error,connection_forced,
<45>Apr 17 09:35:41 node-11 rabbitmq            "broker forced connection closure with reason 'shutdown'",none}

root@node-9:~# egrep "ECONNREFUSED|TRACE"  /var/log/keystone-all.log

2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 686, in ensure
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit     return method()
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 789, in _publish
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic=topic, **kwargs)
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 424, in __init__
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit     topic, **kwargs)
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 399, in __init__
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit     **options)
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 342, in __init__
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 440, in reconnect
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit     queue.declare()
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 505, in declare
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit     self.queue_declare(nowait, passive=False)
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 531, in queue_declare
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait)
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 1258, in queue_declare
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit     (50, 11),  # Channel.queue_declare_ok
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit     self.channel_id, allowed_methods)
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 237, in _wait_method
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit     self.method_reader.read_method()
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit     raise m
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit IOError: Socket closed
2015-04-17 09:38:06.961 10982 TRACE oslo.messaging._drivers.impl_rabbit 
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 686, in ensure
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit     return method()
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 789, in _publish
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic=topic, **kwargs)
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 424, in __init__
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit     topic, **kwargs)
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 399, in __init__
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit     **options)
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 342, in __init__
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 440, in reconnect
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit     queue.declare()
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 505, in declare
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit     self.queue_declare(nowait, passive=False)
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 531, in queue_declare
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait)
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 1258, in queue_declare
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit     (50, 11),  # Channel.queue_declare_ok
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit     self.channel_id, allowed_methods)
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 237, in _wait_method
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit     self.method_reader.read_method()
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit     raise m
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit IOError: Socket closed
2015-04-17 09:38:07.044 10981 TRACE oslo.messaging._drivers.impl_rabbit 
<11>Apr 17 09:38:08 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
<11>Apr 17 09:38:08 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
<11>Apr 17 09:38:08 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
<11>Apr 17 09:38:10 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
<11>Apr 17 09:38:10 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
<11>Apr 17 09:38:10 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
<11>Apr 17 09:38:14 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
<11>Apr 17 09:38:15 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
<11>Apr 17 09:38:15 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
<11>Apr 17 09:38:21 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.
<11>Apr 17 09:38:22 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.
<11>Apr 17 09:38:22 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.
<11>Apr 17 09:38:29 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.
<11>Apr 17 09:38:30 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.
<11>Apr 17 09:38:30 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.
<11>Apr 17 09:38:39 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.
<11>Apr 17 09:38:41 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.
<11>Apr 17 09:38:41 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.
<11>Apr 17 09:38:52 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
<11>Apr 17 09:38:53 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 13 seconds.
<11>Apr 17 09:38:53 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 13 seconds.
<11>Apr 17 09:38:54 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 13 seconds.
<11>Apr 17 09:38:54 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
<11>Apr 17 09:38:58 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
<11>Apr 17 09:39:01 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
<11>Apr 17 09:39:03 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
<11>Apr 17 09:39:04 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.
<11>Apr 17 09:39:07 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
<11>Apr 17 09:39:08 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.
<11>Apr 17 09:39:08 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.
<11>Apr 17 09:39:09 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.
<11>Apr 17 09:39:13 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.
<11>Apr 17 09:39:13 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.
<11>Apr 17 09:39:22 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.
<11>Apr 17 09:39:23 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.
<11>Apr 17 09:39:25 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 17 seconds.
<11>Apr 17 09:39:25 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 17 seconds.
<11>Apr 17 09:39:25 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 17 seconds.
<11>Apr 17 09:39:30 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
<11>Apr 17 09:39:32 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.
<11>Apr 17 09:39:32 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
<11>Apr 17 09:39:36 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
<11>Apr 17 09:39:36 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 13 seconds.
<11>Apr 17 09:39:37 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
<11>Apr 17 09:39:38 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
<11>Apr 17 09:39:42 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
<11>Apr 17 09:39:43 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 19 seconds.
<11>Apr 17 09:39:43 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 19 seconds.
<11>Apr 17 09:39:43 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.
<11>Apr 17 09:39:43 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 19 seconds.
<11>Apr 17 09:39:44 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 13 seconds.
<11>Apr 17 09:39:49 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 7 seconds.
<11>Apr 17 09:39:50 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.
<11>Apr 17 09:39:51 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.
<11>Apr 17 09:39:57 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.
<11>Apr 17 09:39:58 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.
<11>Apr 17 09:40:02 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.
<11>Apr 17 09:40:04 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 21 seconds.
<11>Apr 17 09:40:04 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 21 seconds.
<11>Apr 17 09:40:04 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 21 seconds.
<11>Apr 17 09:40:07 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 17 seconds.
<11>Apr 17 09:40:08 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.
<11>Apr 17 09:40:14 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 13 seconds.
<11>Apr 17 09:40:15 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 17 seconds.
<11>Apr 17 09:40:20 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 13 seconds.
<11>Apr 17 09:40:25 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 19 seconds.
<11>Apr 17 09:40:26 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 23 seconds.
<11>Apr 17 09:40:26 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 23 seconds.
<11>Apr 17 09:40:27 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 23 seconds.
<11>Apr 17 09:40:28 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.
<11>Apr 17 09:40:33 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 19 seconds.
<11>Apr 17 09:40:34 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 15 seconds.
<11>Apr 17 09:40:44 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 17 seconds.
<11>Apr 17 09:40:46 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 21 seconds.
<11>Apr 17 09:40:50 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 25 seconds.
<11>Apr 17 09:40:50 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 25 seconds.
<11>Apr 17 09:40:50 node-9 keystone-all AMQP server on 192.168.0.19:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 17 seconds.
<11>Apr 17 09:40:51 node-9 keystone-all AMQP server on 192.168.0.21:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 25 seconds.
<11>Apr 17 09:40:53 node-9 keystone-all AMQP server on 192.168.0.20:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 21 seconds

I'm marking this bug as invalid, since it is rather lack of resources.

Changed in fuel:
status:	Confirmed → Invalid

Revision history for this message

Dmitry Pyzhov (dpyzhov) wrote on 2015-04-17:

#3

Then we have incorrect settings in virtualbox scripts and bug needs to be fixed with increasing of it. What is the minimal requirement for controllers?

Changed in fuel:
status:	Invalid → New
importance:	Critical → High

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2015-04-17:

#4

I want to add that earlier such deployments were successful in the same amount of resources on controllers (2 GB). I can try to use 3 GB on controllers for Ubuntu HA deployment.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-17: Related fix proposed to fuel-web (master)

#5

Related fix proposed to branch: master
Review: https://review.openstack.org/174912

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-17: Related fix proposed to fuel-library (master)

#6

Related fix proposed to branch: master
Review: https://review.openstack.org/174913

Revision history for this message

Vasyl Saienko (vsaienko) wrote on 2015-04-17:

#7

for example in fuel-devops default RAM allocation for Ubuntu is 2.5G

https://github.com/stackforge/fuel-devops/blob/master/devops/settings.py#L185-L188

Revision history for this message

Vasyl Saienko (vsaienko) wrote on 2015-04-17:

#8

what exactly virtualbox scripts you are using?

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2015-04-17:

#9

Scripts:
https://github.com/stackforge/fuel-main/tree/master/virtualbox

Revision history for this message

Vasyl Saienko (vsaienko) wrote on 2015-04-17:

#10

Anastasia,

I see 12 running slaves + 1 fuel = 13 total, 13*2Gb = 26Gb what is RAM capacity of your server? How many VMs were running at the moment when you performed tests?

Even if we change default RAM allocation to slaves there is no guerantee that hardware server is not run out of memory or overloaded by CPU/HDD.
As improvement I think that it make sense to add some check of load on Compute/Controller before running test, and produce warning if load avarage is to high.

[root@fuel ~]# fuel nodes
DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|-----------|------------------|---------|------------|-------------------|-------------------|---------------|--------|---------
10 | ready | Untitled (03:03) | 6 | 10.20.0.10 | 02:4a:01:6a:3d:52 | controller, mongo | | True | 6
11 | ready | Untitled (f6:a1) | 6 | 10.20.0.14 | 08:00:27:8f:f6:a1 | controller, mongo | | True | 6
12 | ready | Untitled (71:30) | 6 | 10.20.0.13 | 08:00:27:b0:71:30 | cinder, compute | | True | 6
9 | ready | Untitled (7d:02) | 6 | 10.20.0.9 | 08:00:27:bd:7d:02 | controller, mongo | | True | 6
13 | deploying | Untitled (19:78) | 7 | 10.20.0.16 | 08:00:27:ab:19:78 | controller | | True | 7
14 | deploying | Untitled (d8:c0) | 8 | 10.20.0.15 | 08:00:27:31:d8:c0 | controller | | True | 8
20 | discover | Untitled (d3:e9) | None | 10.20.0.18 | 08:00:27:cd:d3:e9 | | | True | None
17 | discover | Untitled (b8:0a) | None | 10.20.0.20 | 08:00:27:37:b8:0a | | | True | None
18 | discover | Untitled (43:2b) | None | 10.20.0.19 | 08:00:27:93:43:2b | | | True | None
15 | discover | Untitled (71:3f) | None | 10.20.0.17 | 08:00:27:c8:71:3f | | | True | None
19 | discover | Untitled (4a:87) | None | 10.20.0.21 | 08:00:27:15:4a:87 | | | True | None
16 | discover | Untitled (5a:89) | None | 10.20.0.4 | 08:00:27:4f:5a:89 | | | True | None
[root@fuel ~]#

Anastasia,

I see 12 running slaves + 1 fuel = 13 total, 13*2Gb = 26Gb what is RAM capacity of your server? How many VMs were running at the moment when you performed tests?

Even if we change default RAM allocation to slaves there is no guerantee that hardware server is not run out of memory or overloaded by CPU/HDD.
As improvement I think that it make sense to add some check of load on Compute/Controller before running test, and produce warning if load avarage is to high.

[root@fuel ~]# fuel nodes
DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.
id | status    | name             | cluster | ip         | mac               | roles             | pending_roles | online | group_id
---|-----------|------------------|---------|------------|-------------------|-------------------|---------------|--------|---------
10 | ready     | Untitled (03:03) | 6       | 10.20.0.10 | 02:4a:01:6a:3d:52 | controller, mongo |               | True   | 6       
11 | ready     | Untitled (f6:a1) | 6       | 10.20.0.14 | 08:00:27:8f:f6:a1 | controller, mongo |               | True   | 6       
12 | ready     | Untitled (71:30) | 6       | 10.20.0.13 | 08:00:27:b0:71:30 | cinder, compute   |               | True   | 6       
9  | ready     | Untitled (7d:02) | 6       | 10.20.0.9  | 08:00:27:bd:7d:02 | controller, mongo |               | True   | 6       
13 | deploying | Untitled (19:78) | 7       | 10.20.0.16 | 08:00:27:ab:19:78 | controller        |               | True   | 7       
14 | deploying | Untitled (d8:c0) | 8       | 10.20.0.15 | 08:00:27:31:d8:c0 | controller        |               | True   | 8       
20 | discover  | Untitled (d3:e9) | None    | 10.20.0.18 | 08:00:27:cd:d3:e9 |                   |               | True   | None    
17 | discover  | Untitled (b8:0a) | None    | 10.20.0.20 | 08:00:27:37:b8:0a |                   |               | True   | None    
18 | discover  | Untitled (43:2b) | None    | 10.20.0.19 | 08:00:27:93:43:2b |                   |               | True   | None    
15 | discover  | Untitled (71:3f) | None    | 10.20.0.17 | 08:00:27:c8:71:3f |                   |               | True   | None    
19 | discover  | Untitled (4a:87) | None    | 10.20.0.21 | 08:00:27:15:4a:87 |                   |               | True   | None    
16 | discover  | Untitled (5a:89) | None    | 10.20.0.4  | 08:00:27:4f:5a:89 |                   |               | True   | None    
[root@fuel ~]#

Revision history for this message

Vasyl Saienko (vsaienko) wrote on 2015-04-17:

#11

VirtualBox is used on Desktops. In fuel-main largest cluster size is 5 slaves for 16Gb of RAM (https://github.com/stackforge/fuel-main/blob/master/virtualbox/config.sh#L103). If you have more RAM it makes sense to use KVM with hardware virtualization intead of VB. All necessary scripts you can find at fuel-qa[0], fuel-devops[1].

[0] https://www.google.com.ua/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0CB0QFjAA&url=https%3A%2F%2Fgithub.com%2Fstackforge%2Ffuel-qa&ei=yjwxVbatEIPiywPi6YCwBA&usg=AFQjCNGKr1O03r8yovLCvC6L6dGMM_0hcg&bvm=bv.91071109,d.bGQ
[1] https://www.google.com.ua/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&uact=8&ved=0CCEQFjAB&url=https%3A%2F%2Fgithub.com%2Fstackforge%2Ffuel-devops&ei=2TwxVZfZF6vjywONg4DADg&usg=AFQjCNHNH1CD7kfY4e1h470N6-WNWtaguw&bvm=bv.91071109,d.bGQ

Dmitry Ilyin (idv1985) on 2015-04-20

Changed in fuel:
status:	New → Confirmed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-20: Fix proposed to fuel-library (master)

#12

Fix proposed to branch: master
Review: https://review.openstack.org/175615

Changed in fuel:
assignee:	Vasyl Saienko (vsaienko) → Igor Yozhikov (iyozhikov)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-21: Fix merged to fuel-library (master)

#13

Reviewed: https://review.openstack.org/175615
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c0893e6f14a03f1791fc9d2f7f1f038669bbb86b
Submitter: Jenkins
Branch: master

commit c0893e6f14a03f1791fc9d2f7f1f038669bbb86b
Author: Igor Yozhikov <email address hidden>
Date: Tue Apr 21 02:54:07 2015 +0300

Update in heat_domain_id_setter

* add keystone api version reset in auth_url
* changed used keystone api port 5000 -> 35357

Change-Id: I4c0df7600e10d57429eda854cd8aa6208e1c6539
Closes-Bug: #1445422

Changed in fuel:
status:	In Progress → Fix Committed

Anastasia Palkina (apalkina) on 2015-04-22

tags:

added: on-verification

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2015-04-23:

#14

Verified on ISO #338

"build_id": "2015-04-22_22-54-31", "build_number": "338", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-04-22_22-54-31", "build_number": "338", "api": "1.0", "fuel-library_sha": "9b26cb98e7ceb905fe85b744eee2f808f7e4737b", "nailgun_sha": "4972120d5a5d41efe9be2d10b81329adf3258a78", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "2b311b3b82a1e2df1dc3484a0f37e282273cd988", "astute_sha": "5be3506e6a4f8b7f0a858db2e162dabcc07347df", "fuel-ostf_sha": "dcda5a869d1780efc265b6f6b7b8c14c8324a68b", "release": "6.1", "fuelmain_sha": "2f8be90209e709810674f73abb02bb54a700ce5e"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "9b26cb98e7ceb905fe85b744eee2f808f7e4737b", "nailgun_sha": "4972120d5a5d41efe9be2d10b81329adf3258a78", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "2b311b3b82a1e2df1dc3484a0f37e282273cd988", "astute_sha": "5be3506e6a4f8b7f0a858db2e162dabcc07347df", "fuel-ostf_sha": "dcda5a869d1780efc265b6f6b7b8c14c8324a68b", "release": "6.1", "fuelmain_sha": "2f8be90209e709810674f73abb02bb54a700ce5e"

Changed in fuel:
status:	Fix Committed → Fix Released
tags:	removed: on-verification

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-27: Related fix merged to fuel-library (master)

#15

Reviewed: https://review.openstack.org/174913
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c4e052529ce1b6d3aae467c943aa05bb854c91c4
Submitter: Jenkins
Branch: master

commit c4e052529ce1b6d3aae467c943aa05bb854c91c4
Author: Dmitry Ilyin <email address hidden>
Date: Fri Apr 17 18:02:53 2015 +0300

Install generated horizon secret key

* Taken from horizon config hash

Change-Id: I321ba3acfa9efe3a4feedb12a68d6dd2cd6dd814
Related-Bug: 1445422

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-27: Related fix merged to fuel-web (master)

#16

Reviewed: https://review.openstack.org/174912
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=fda6850ebb3c20249d28e81a6f2f06fedb14c90e
Submitter: Jenkins
Branch: master

commit fda6850ebb3c20249d28e81a6f2f06fedb14c90e
Author: Dmitry Ilyin <email address hidden>
Date: Fri Apr 17 18:00:44 2015 +0300

Add horizon block with secret_key generator

* It will be used to encrypt cookies

Change-Id: If94cf4193473854658ca25201f4aff5f99760608
Related-Bug: 1445422

Fuel for OpenStack

Received error response from Keystone server at http://192.168.0.17:5000/v3/tokens: Not Found

Bug Description

Other bug subscribers

Remote bug watches