| 2014-09-24 19:01:57 |
Tyler Wilson |
bug |
|
|
added bug |
| 2014-09-24 19:21:10 |
Tyler Wilson |
description |
{"build_id": "2014-09-18_06-04-08", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "31", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["experimental"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-18_06-04-08", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "31", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["experimental"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}
1. Create new environment (Ubuntu, HA mode)
2. Choose GRE segmentation
3. Add controller x3 + Ceilometer
4. Add computes x3 + Ceph OSD
Shutdown primary controller, attempt to use the Resource Usage pages in horizon
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:38.372884+01:00 info: 2014-09-24 18:58:38.369 19184 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:58:38.374048+01:00 info: 2014-09-24 18:58:38.370 19184 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:40.411124+01:00 info: 2014-09-24 18:58:40.408 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:58:40.412381+01:00 info: 2014-09-24 18:58:40.409 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:43.394037+01:00 err: 2014-09-24 18:58:43.390 19184 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.3:5673 is unreachable: timed out. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:45.432197+01:00 err: 2014-09-24 18:58:45.429 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:55.426343+01:00 info: 2014-09-24 18:58:55.428 30562 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:58:55.427363+01:00 info: 2014-09-24 18:58:55.428 30562 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T19:59:00.449697+01:00 err: 2014-09-24 18:59:00.451 30562 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:04.452501+01:00 info: 2014-09-24 18:59:04.449 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:59:04.453660+01:00 info: 2014-09-24 18:59:04.451 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T19:59:09.469792+01:00 err: 2014-09-24 18:59:09.467 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:13.422414+01:00 info: 2014-09-24 18:59:13.419 19184 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:59:13.424022+01:00 info: 2014-09-24 18:59:13.420 19184 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T19:59:18.433425+01:00 err: 2014-09-24 18:59:18.430 19184 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.3:5673 is unreachable: timed out. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:28.490133+01:00 info: 2014-09-24 18:59:28.487 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:59:28.491235+01:00 info: 2014-09-24 18:59:28.488 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:30.479988+01:00 info: 2014-09-24 18:59:30.481 30562 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.3:5673
2014-09-24T19:59:30.481187+01:00 info: 2014-09-24 18:59:30.482 30562 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:33.515571+01:00 err: 2014-09-24 18:59:33.512 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:35.504238+01:00 err: 2014-09-24 18:59:35.505 30562 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:48.464659+01:00 info: 2014-09-24 18:59:48.461 19184 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:59:48.465735+01:00 info: 2014-09-24 18:59:48.462 19184 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:52.523420+01:00 info: 2014-09-24 18:59:52.520 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:59:52.524857+01:00 info: 2014-09-24 18:59:52.522 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:53.471741+01:00 err: 2014-09-24 18:59:53.468 19184 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.4:5673 is unreachable: timed out. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:57.533816+01:00 err: 2014-09-24 18:59:57.531 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T20:00:05.531734+01:00 info: 2014-09-24 19:00:05.533 30562 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.3:5673
2014-09-24T20:00:05.532827+01:00 info: 2014-09-24 19:00:05.534 30562 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T20:00:10.543608+01:00 err: 2014-09-24 19:00:10.545 30562 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T20:00:16.554651+01:00 info: 2014-09-24 19:00:16.551 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T20:00:16.556032+01:00 info: 2014-09-24 19:00:16.553 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
root@node-68:~# crm status
Last updated: Wed Sep 24 19:01:46 2014
Last change: Wed Sep 24 19:00:30 2014 via crm_attribute on node-69
Stack: classic openais (with plugin)
Current DC: node-69 - partition with quorum
Version: 1.1.10-42f2063
3 Nodes configured, 3 expected votes
22 Resources configured
Online: [ node-68 node-69 node-70 ]
vip__management_old (ocf::mirantis:ns_IPaddr2): Started node-69
vip__public_old (ocf::mirantis:ns_IPaddr2): Started node-70
p_ceilometer-alarm-evaluator (ocf::mirantis:ceilometer-alarm-evaluator): Started node-68
p_ceilometer-agent-central (ocf::mirantis:ceilometer-agent-central): Started node-69
Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
Masters: [ node-68 ]
Slaves: [ node-69 node-70 ]
Clone Set: clone_p_mysql [p_mysql]
Started: [ node-68 node-69 node-70 ]
Clone Set: clone_p_haproxy [p_haproxy]
Started: [ node-68 node-69 node-70 ]
p_heat-engine (ocf::mirantis:heat-engine): Started node-69
Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
Started: [ node-68 node-69 node-70 ]
Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
Started: [ node-68 node-69 node-70 ]
p_neutron-dhcp-agent (ocf::mirantis:neutron-agent-dhcp): Started node-69
p_neutron-l3-agent (ocf::mirantis:neutron-agent-l3): Started node-68
Failed actions:
p_mysql_monitor_120000 (node=node-68, call=97, rc=1, status=complete, last-rc-change=Wed Sep 24 15:17:52 2014
, queued=0ms, exec=0ms
): unknown error
p_neutron-metadata-agent_monitor_60000 (node=node-68, call=283, rc=7, status=complete, last-rc-change=Wed Sep 24 01:58:05 2014
, queued=0ms, exec=0ms
): not running |
{"build_id": "2014-09-18_06-04-08", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "31", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["experimental"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-18_06-04-08", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "31", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["experimental"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}
1. Create new environment (Ubuntu, HA mode)
2. Choose GRE segmentation
3. Add controller x3 + Ceilometer
4. Add computes x3 + Ceph OSD
Shutdown primary controller, attempt to use the Resource Usage pages in horizon
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:38.372884+01:00 info: 2014-09-24 18:58:38.369 19184 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:58:38.374048+01:00 info: 2014-09-24 18:58:38.370 19184 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:40.411124+01:00 info: 2014-09-24 18:58:40.408 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:58:40.412381+01:00 info: 2014-09-24 18:58:40.409 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:43.394037+01:00 err: 2014-09-24 18:58:43.390 19184 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.3:5673 is unreachable: timed out. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:45.432197+01:00 err: 2014-09-24 18:58:45.429 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:55.426343+01:00 info: 2014-09-24 18:58:55.428 30562 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:58:55.427363+01:00 info: 2014-09-24 18:58:55.428 30562 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T19:59:00.449697+01:00 err: 2014-09-24 18:59:00.451 30562 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:04.452501+01:00 info: 2014-09-24 18:59:04.449 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:59:04.453660+01:00 info: 2014-09-24 18:59:04.451 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T19:59:09.469792+01:00 err: 2014-09-24 18:59:09.467 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:13.422414+01:00 info: 2014-09-24 18:59:13.419 19184 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:59:13.424022+01:00 info: 2014-09-24 18:59:13.420 19184 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T19:59:18.433425+01:00 err: 2014-09-24 18:59:18.430 19184 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.3:5673 is unreachable: timed out. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:28.490133+01:00 info: 2014-09-24 18:59:28.487 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:59:28.491235+01:00 info: 2014-09-24 18:59:28.488 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:30.479988+01:00 info: 2014-09-24 18:59:30.481 30562 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.3:5673
2014-09-24T19:59:30.481187+01:00 info: 2014-09-24 18:59:30.482 30562 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:33.515571+01:00 err: 2014-09-24 18:59:33.512 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:35.504238+01:00 err: 2014-09-24 18:59:35.505 30562 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:48.464659+01:00 info: 2014-09-24 18:59:48.461 19184 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:59:48.465735+01:00 info: 2014-09-24 18:59:48.462 19184 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:52.523420+01:00 info: 2014-09-24 18:59:52.520 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:59:52.524857+01:00 info: 2014-09-24 18:59:52.522 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:53.471741+01:00 err: 2014-09-24 18:59:53.468 19184 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.4:5673 is unreachable: timed out. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:57.533816+01:00 err: 2014-09-24 18:59:57.531 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T20:00:05.531734+01:00 info: 2014-09-24 19:00:05.533 30562 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.3:5673
2014-09-24T20:00:05.532827+01:00 info: 2014-09-24 19:00:05.534 30562 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T20:00:10.543608+01:00 err: 2014-09-24 19:00:10.545 30562 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T20:00:16.554651+01:00 info: 2014-09-24 19:00:16.551 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T20:00:16.556032+01:00 info: 2014-09-24 19:00:16.553 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
root@node-68:~# crm status
Last updated: Wed Sep 24 19:01:46 2014
Last change: Wed Sep 24 19:00:30 2014 via crm_attribute on node-69
Stack: classic openais (with plugin)
Current DC: node-69 - partition with quorum
Version: 1.1.10-42f2063
3 Nodes configured, 3 expected votes
22 Resources configured
Online: [ node-68 node-69 node-70 ]
vip__management_old (ocf::mirantis:ns_IPaddr2): Started node-69
vip__public_old (ocf::mirantis:ns_IPaddr2): Started node-70
p_ceilometer-alarm-evaluator (ocf::mirantis:ceilometer-alarm-evaluator): Started node-68
p_ceilometer-agent-central (ocf::mirantis:ceilometer-agent-central): Started node-69
Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
Masters: [ node-68 ]
Slaves: [ node-69 node-70 ]
Clone Set: clone_p_mysql [p_mysql]
Started: [ node-68 node-69 node-70 ]
Clone Set: clone_p_haproxy [p_haproxy]
Started: [ node-68 node-69 node-70 ]
p_heat-engine (ocf::mirantis:heat-engine): Started node-69
Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
Started: [ node-68 node-69 node-70 ]
Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
Started: [ node-68 node-69 node-70 ]
p_neutron-dhcp-agent (ocf::mirantis:neutron-agent-dhcp): Started node-69
p_neutron-l3-agent (ocf::mirantis:neutron-agent-l3): Started node-68
Failed actions:
p_mysql_monitor_120000 (node=node-68, call=97, rc=1, status=complete, last-rc-change=Wed Sep 24 15:17:52 2014
, queued=0ms, exec=0ms
): unknown error
p_neutron-metadata-agent_monitor_60000 (node=node-68, call=283, rc=7, status=complete, last-rc-change=Wed Sep 24 01:58:05 2014
, queued=0ms, exec=0ms
): not running
Fuel Snapshot: http://23.80.0.2:8000/dump/fuel-snapshot-2014-09-24_20-01-30.tgz |
|
| 2014-09-24 19:30:05 |
Nastya Urlapova |
fuel: milestone |
|
6.0 |
|
| 2014-09-24 19:30:33 |
Nastya Urlapova |
fuel: importance |
Undecided |
High |
|
| 2014-09-24 19:30:51 |
Nastya Urlapova |
fuel: assignee |
|
Fuel Library Team (fuel-library) |
|
| 2014-09-24 19:47:53 |
Bogdan Dobrelya |
fuel: status |
New |
Incomplete |
|
| 2014-09-24 21:29:24 |
Bogdan Dobrelya |
fuel: status |
Incomplete |
Confirmed |
|
| 2014-09-24 21:33:37 |
Bogdan Dobrelya |
description |
{"build_id": "2014-09-18_06-04-08", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "31", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["experimental"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-18_06-04-08", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "31", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["experimental"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}
1. Create new environment (Ubuntu, HA mode)
2. Choose GRE segmentation
3. Add controller x3 + Ceilometer
4. Add computes x3 + Ceph OSD
Shutdown primary controller, attempt to use the Resource Usage pages in horizon
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:38.372884+01:00 info: 2014-09-24 18:58:38.369 19184 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:58:38.374048+01:00 info: 2014-09-24 18:58:38.370 19184 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:40.411124+01:00 info: 2014-09-24 18:58:40.408 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:58:40.412381+01:00 info: 2014-09-24 18:58:40.409 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:43.394037+01:00 err: 2014-09-24 18:58:43.390 19184 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.3:5673 is unreachable: timed out. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:45.432197+01:00 err: 2014-09-24 18:58:45.429 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:55.426343+01:00 info: 2014-09-24 18:58:55.428 30562 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:58:55.427363+01:00 info: 2014-09-24 18:58:55.428 30562 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T19:59:00.449697+01:00 err: 2014-09-24 18:59:00.451 30562 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:04.452501+01:00 info: 2014-09-24 18:59:04.449 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:59:04.453660+01:00 info: 2014-09-24 18:59:04.451 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T19:59:09.469792+01:00 err: 2014-09-24 18:59:09.467 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:13.422414+01:00 info: 2014-09-24 18:59:13.419 19184 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:59:13.424022+01:00 info: 2014-09-24 18:59:13.420 19184 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T19:59:18.433425+01:00 err: 2014-09-24 18:59:18.430 19184 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.3:5673 is unreachable: timed out. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:28.490133+01:00 info: 2014-09-24 18:59:28.487 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:59:28.491235+01:00 info: 2014-09-24 18:59:28.488 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:30.479988+01:00 info: 2014-09-24 18:59:30.481 30562 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.3:5673
2014-09-24T19:59:30.481187+01:00 info: 2014-09-24 18:59:30.482 30562 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:33.515571+01:00 err: 2014-09-24 18:59:33.512 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:35.504238+01:00 err: 2014-09-24 18:59:35.505 30562 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:48.464659+01:00 info: 2014-09-24 18:59:48.461 19184 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:59:48.465735+01:00 info: 2014-09-24 18:59:48.462 19184 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:52.523420+01:00 info: 2014-09-24 18:59:52.520 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:59:52.524857+01:00 info: 2014-09-24 18:59:52.522 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:53.471741+01:00 err: 2014-09-24 18:59:53.468 19184 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.4:5673 is unreachable: timed out. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:57.533816+01:00 err: 2014-09-24 18:59:57.531 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T20:00:05.531734+01:00 info: 2014-09-24 19:00:05.533 30562 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.3:5673
2014-09-24T20:00:05.532827+01:00 info: 2014-09-24 19:00:05.534 30562 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T20:00:10.543608+01:00 err: 2014-09-24 19:00:10.545 30562 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T20:00:16.554651+01:00 info: 2014-09-24 19:00:16.551 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T20:00:16.556032+01:00 info: 2014-09-24 19:00:16.553 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
root@node-68:~# crm status
Last updated: Wed Sep 24 19:01:46 2014
Last change: Wed Sep 24 19:00:30 2014 via crm_attribute on node-69
Stack: classic openais (with plugin)
Current DC: node-69 - partition with quorum
Version: 1.1.10-42f2063
3 Nodes configured, 3 expected votes
22 Resources configured
Online: [ node-68 node-69 node-70 ]
vip__management_old (ocf::mirantis:ns_IPaddr2): Started node-69
vip__public_old (ocf::mirantis:ns_IPaddr2): Started node-70
p_ceilometer-alarm-evaluator (ocf::mirantis:ceilometer-alarm-evaluator): Started node-68
p_ceilometer-agent-central (ocf::mirantis:ceilometer-agent-central): Started node-69
Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
Masters: [ node-68 ]
Slaves: [ node-69 node-70 ]
Clone Set: clone_p_mysql [p_mysql]
Started: [ node-68 node-69 node-70 ]
Clone Set: clone_p_haproxy [p_haproxy]
Started: [ node-68 node-69 node-70 ]
p_heat-engine (ocf::mirantis:heat-engine): Started node-69
Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
Started: [ node-68 node-69 node-70 ]
Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
Started: [ node-68 node-69 node-70 ]
p_neutron-dhcp-agent (ocf::mirantis:neutron-agent-dhcp): Started node-69
p_neutron-l3-agent (ocf::mirantis:neutron-agent-l3): Started node-68
Failed actions:
p_mysql_monitor_120000 (node=node-68, call=97, rc=1, status=complete, last-rc-change=Wed Sep 24 15:17:52 2014
, queued=0ms, exec=0ms
): unknown error
p_neutron-metadata-agent_monitor_60000 (node=node-68, call=283, rc=7, status=complete, last-rc-change=Wed Sep 24 01:58:05 2014
, queued=0ms, exec=0ms
): not running
Fuel Snapshot: http://23.80.0.2:8000/dump/fuel-snapshot-2014-09-24_20-01-30.tgz |
{"build_id": "2014-09-18_06-04-08", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "31", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["experimental"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-18_06-04-08", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "31", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["experimental"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}
1. Create new environment (Ubuntu, HA mode)
2. Choose GRE segmentation
3. Add controller x3 + Ceilometer
4. Add computes x3 + Ceph OSD
Shutdown primary controller and bring it up again, then attempt to use the Resource Usage pages in horizon
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:38.372884+01:00 info: 2014-09-24 18:58:38.369 19184 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:58:38.374048+01:00 info: 2014-09-24 18:58:38.370 19184 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:40.411124+01:00 info: 2014-09-24 18:58:40.408 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:58:40.412381+01:00 info: 2014-09-24 18:58:40.409 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:43.394037+01:00 err: 2014-09-24 18:58:43.390 19184 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.3:5673 is unreachable: timed out. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:45.432197+01:00 err: 2014-09-24 18:58:45.429 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:58:55.426343+01:00 info: 2014-09-24 18:58:55.428 30562 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:58:55.427363+01:00 info: 2014-09-24 18:58:55.428 30562 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T19:59:00.449697+01:00 err: 2014-09-24 18:59:00.451 30562 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:04.452501+01:00 info: 2014-09-24 18:59:04.449 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:59:04.453660+01:00 info: 2014-09-24 18:59:04.451 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T19:59:09.469792+01:00 err: 2014-09-24 18:59:09.467 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:13.422414+01:00 info: 2014-09-24 18:59:13.419 19184 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:59:13.424022+01:00 info: 2014-09-24 18:59:13.420 19184 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T19:59:18.433425+01:00 err: 2014-09-24 18:59:18.430 19184 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.3:5673 is unreachable: timed out. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:28.490133+01:00 info: 2014-09-24 18:59:28.487 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:59:28.491235+01:00 info: 2014-09-24 18:59:28.488 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:30.479988+01:00 info: 2014-09-24 18:59:30.481 30562 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.3:5673
2014-09-24T19:59:30.481187+01:00 info: 2014-09-24 18:59:30.482 30562 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:33.515571+01:00 err: 2014-09-24 18:59:33.512 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:35.504238+01:00 err: 2014-09-24 18:59:35.505 30562 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:48.464659+01:00 info: 2014-09-24 18:59:48.461 19184 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673
2014-09-24T19:59:48.465735+01:00 info: 2014-09-24 18:59:48.462 19184 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:52.523420+01:00 info: 2014-09-24 18:59:52.520 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T19:59:52.524857+01:00 info: 2014-09-24 18:59:52.522 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
==> /var/log/docker-logs/remote/node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:53.471741+01:00 err: 2014-09-24 18:59:53.468 19184 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.4:5673 is unreachable: timed out. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T19:59:57.533816+01:00 err: 2014-09-24 18:59:57.531 10059 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19 seconds.
==> /var/log/docker-logs/remote/node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T20:00:05.531734+01:00 info: 2014-09-24 19:00:05.533 30562 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.3:5673
2014-09-24T20:00:05.532827+01:00 info: 2014-09-24 19:00:05.534 30562 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
2014-09-24T20:00:10.543608+01:00 err: 2014-09-24 19:00:10.545 30562 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30 seconds.
==> /var/log/docker-logs/remote/node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
2014-09-24T20:00:16.554651+01:00 info: 2014-09-24 19:00:16.551 10059 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.0.5:5673
2014-09-24T20:00:16.556032+01:00 info: 2014-09-24 19:00:16.553 10059 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds...
root@node-68:~# crm status
Last updated: Wed Sep 24 19:01:46 2014
Last change: Wed Sep 24 19:00:30 2014 via crm_attribute on node-69
Stack: classic openais (with plugin)
Current DC: node-69 - partition with quorum
Version: 1.1.10-42f2063
3 Nodes configured, 3 expected votes
22 Resources configured
Online: [ node-68 node-69 node-70 ]
vip__management_old (ocf::mirantis:ns_IPaddr2): Started node-69
vip__public_old (ocf::mirantis:ns_IPaddr2): Started node-70
p_ceilometer-alarm-evaluator (ocf::mirantis:ceilometer-alarm-evaluator): Started node-68
p_ceilometer-agent-central (ocf::mirantis:ceilometer-agent-central): Started node-69
Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
Masters: [ node-68 ]
Slaves: [ node-69 node-70 ]
Clone Set: clone_p_mysql [p_mysql]
Started: [ node-68 node-69 node-70 ]
Clone Set: clone_p_haproxy [p_haproxy]
Started: [ node-68 node-69 node-70 ]
p_heat-engine (ocf::mirantis:heat-engine): Started node-69
Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
Started: [ node-68 node-69 node-70 ]
Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
Started: [ node-68 node-69 node-70 ]
p_neutron-dhcp-agent (ocf::mirantis:neutron-agent-dhcp): Started node-69
p_neutron-l3-agent (ocf::mirantis:neutron-agent-l3): Started node-68
Failed actions:
p_mysql_monitor_120000 (node=node-68, call=97, rc=1, status=complete, last-rc-change=Wed Sep 24 15:17:52 2014
, queued=0ms, exec=0ms
): unknown error
p_neutron-metadata-agent_monitor_60000 (node=node-68, call=283, rc=7, status=complete, last-rc-change=Wed Sep 24 01:58:05 2014
, queued=0ms, exec=0ms
): not running
Fuel Snapshot: http://23.80.0.2:8000/dump/fuel-snapshot-2014-09-24_20-01-30.tgz |
|
| 2014-09-24 22:00:48 |
Bogdan Dobrelya |
fuel: status |
Confirmed |
Invalid |
|
| 2014-09-24 22:10:18 |
Bogdan Dobrelya |
fuel: status |
Invalid |
Confirmed |
|
| 2014-09-25 17:16:22 |
Bogdan Dobrelya |
fuel: assignee |
Fuel Library Team (fuel-library) |
Bogdan Dobrelya (bogdando) |
|
| 2014-09-25 17:59:03 |
OpenStack Infra |
fuel: status |
Confirmed |
In Progress |
|
| 2014-09-25 17:59:27 |
Bogdan Dobrelya |
nominated for series |
|
fuel/6.0.x |
|
| 2014-09-25 17:59:27 |
Bogdan Dobrelya |
bug task added |
|
fuel/6.0.x |
|
| 2014-09-25 17:59:27 |
Bogdan Dobrelya |
nominated for series |
|
fuel/5.1.x |
|
| 2014-09-25 17:59:27 |
Bogdan Dobrelya |
bug task added |
|
fuel/5.1.x |
|
| 2014-09-25 17:59:39 |
Bogdan Dobrelya |
fuel/5.1.x: status |
New |
Triaged |
|
| 2014-09-25 17:59:44 |
Bogdan Dobrelya |
fuel/5.1.x: importance |
Undecided |
High |
|
| 2014-09-25 17:59:49 |
Bogdan Dobrelya |
fuel/5.1.x: assignee |
|
Bogdan Dobrelya (bogdando) |
|
| 2014-09-25 18:00:01 |
Bogdan Dobrelya |
fuel/5.1.x: milestone |
|
5.1.2 |
|
| 2014-09-25 18:00:09 |
Bogdan Dobrelya |
fuel/5.1.x: milestone |
5.1.2 |
5.1.1 |
|
| 2014-09-25 19:36:44 |
Dmitry Mescheryakov |
tags |
|
ceilometer |
|
| 2014-10-08 09:01:50 |
Bogdan Dobrelya |
fuel/6.0.x: assignee |
Bogdan Dobrelya (bogdando) |
MOS Ceilometer (mos-ceilometer) |
|
| 2014-10-08 09:02:10 |
Bogdan Dobrelya |
fuel/5.1.x: assignee |
Bogdan Dobrelya (bogdando) |
MOS Ceilometer (mos-ceilometer) |
|
| 2014-10-13 12:24:54 |
OpenStack Infra |
fuel: assignee |
MOS Ceilometer (mos-ceilometer) |
Bogdan Dobrelya (bogdando) |
|
| 2014-10-14 10:12:23 |
Bogdan Dobrelya |
fuel/5.1.x: assignee |
MOS Ceilometer (mos-ceilometer) |
Bogdan Dobrelya (bogdando) |
|
| 2014-10-14 10:12:27 |
Bogdan Dobrelya |
fuel/6.0.x: status |
In Progress |
Fix Committed |
|
| 2014-10-14 14:30:30 |
Bogdan Dobrelya |
fuel/5.1.x: status |
Triaged |
In Progress |
|
| 2014-10-14 14:30:41 |
Bogdan Dobrelya |
fuel/6.0.x: assignee |
Bogdan Dobrelya (bogdando) |
MOS Ceilometer (mos-ceilometer) |
|
| 2014-10-14 14:30:45 |
Bogdan Dobrelya |
fuel/6.0.x: status |
Fix Committed |
In Progress |
|
| 2014-10-20 09:58:15 |
Bogdan Dobrelya |
fuel/5.1.x: assignee |
Bogdan Dobrelya (bogdando) |
MOS Ceilometer (mos-ceilometer) |
|
| 2014-10-20 09:58:33 |
Bogdan Dobrelya |
fuel/6.0.x: status |
In Progress |
Confirmed |
|
| 2014-10-20 09:58:40 |
Bogdan Dobrelya |
fuel/5.1.x: status |
In Progress |
Confirmed |
|
| 2014-10-23 12:34:32 |
Bogdan Dobrelya |
fuel/5.1.x: assignee |
MOS Ceilometer (mos-ceilometer) |
Bogdan Dobrelya (bogdando) |
|
| 2014-10-23 12:34:36 |
Bogdan Dobrelya |
fuel/6.0.x: assignee |
MOS Ceilometer (mos-ceilometer) |
Bogdan Dobrelya (bogdando) |
|
| 2014-10-23 12:34:40 |
Bogdan Dobrelya |
fuel/5.1.x: status |
Confirmed |
Fix Committed |
|
| 2014-10-23 12:34:43 |
Bogdan Dobrelya |
fuel/6.0.x: status |
Confirmed |
Fix Committed |
|
| 2014-12-02 12:53:14 |
Vadim Rovachev |
fuel/6.0.x: status |
Fix Committed |
Fix Released |
|
| 2014-12-09 16:39:00 |
Stanislav Makar |
fuel/5.1.x: status |
Fix Committed |
In Progress |
|
| 2014-12-10 09:21:27 |
Bogdan Dobrelya |
fuel/5.1.x: status |
In Progress |
Fix Committed |
|
| 2014-12-10 10:01:56 |
Stanislav Makar |
fuel/5.1.x: status |
Fix Committed |
Fix Released |
|
