I've restarted 'service ceilometer-agent-notification restart', how are the
murano services restarted as they arnt in init.d?

On Thu, Sep 25, 2014 at 11:50 AM, Bogdan Dobrelya <email address hidden>
wrote:

> As far as I can see from current state of the env, Tyler, there is no
> conntrack issues with traffic. The issue is that murano-api, murano-
> engine, ceilometer-agent-notification at all controller nodes have a
> problem with closed socket they're trying to connect and fail. So I
> believe these are just have to be restarted to complete the fixing.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1373569
>
> Title:
>   Ceilometer Cant Connect to AMQP after Controller Down
>
> Status in Fuel: OpenStack installer that works:
>   In Progress
> Status in Fuel for OpenStack 5.1.x series:
>   Triaged
> Status in Fuel for OpenStack 6.0.x series:
>   In Progress
>
> Bug description:
>   {"build_id": "2014-09-18_06-04-08", "ostf_sha":
>   "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "31",
>   "auth_required": true, "api": "1.0", "nailgun_sha":
>   "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker",
>   "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd",
>   "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
>   "feature_groups": ["experimental"], "release": "5.1",
>   "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id":
>   "2014-09-18_06-04-08", "ostf_sha":
>   "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "31",
>   "api": "1.0", "nailgun_sha":
>   "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker",
>   "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd",
>   "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13",
>   "feature_groups": ["experimental"], "release": "5.1", "fuellib_sha":
>   "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha":
>   "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}
>
>   1. Create new environment (Ubuntu, HA mode)
>   2. Choose GRE segmentation
>   3. Add controller x3 + Ceilometer
>   4. Add computes x3 + Ceph OSD
>
>   Shutdown primary controller and bring it up again, then attempt to use
>   the Resource Usage pages in horizon
>
>   ==> /var/log/docker-logs/remote/
> node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:58:38.372884+01:00 info:  2014-09-24 18:58:38.369 19184
> INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server
> on 127.0.0.1:5673
>   2014-09-24T19:58:38.374048+01:00 info:  2014-09-24 18:58:38.370 19184
> INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0
> seconds...
>
>   ==> /var/log/docker-logs/remote/
> node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:58:40.411124+01:00 info:  2014-09-24 18:58:40.408 10059
> INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server
> on 192.168.0.5:5673
>   2014-09-24T19:58:40.412381+01:00 info:  2014-09-24 18:58:40.409 10059
> INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0
> seconds...
>
>   ==> /var/log/docker-logs/remote/
> node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:58:43.394037+01:00 err:  2014-09-24 18:58:43.390 19184
> ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on
> 192.168.0.3:5673 is unreachable: timed out. Trying again in 30 seconds.
>
>   ==> /var/log/docker-logs/remote/
> node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:58:45.432197+01:00 err:  2014-09-24 18:58:45.429 10059
> ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on
> 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19
> seconds.
>
>   ==> /var/log/docker-logs/remote/
> node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:58:55.426343+01:00 info:  2014-09-24 18:58:55.428 30562
> INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server
> on 127.0.0.1:5673
>   2014-09-24T19:58:55.427363+01:00 info:  2014-09-24 18:58:55.428 30562
> INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0
> seconds...
>   2014-09-24T19:59:00.449697+01:00 err:  2014-09-24 18:59:00.451 30562
> ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on
> 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30
> seconds.
>
>   ==> /var/log/docker-logs/remote/
> node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:59:04.452501+01:00 info:  2014-09-24 18:59:04.449 10059
> INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server
> on 192.168.0.5:5673
>   2014-09-24T19:59:04.453660+01:00 info:  2014-09-24 18:59:04.451 10059
> INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0
> seconds...
>   2014-09-24T19:59:09.469792+01:00 err:  2014-09-24 18:59:09.467 10059
> ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on
> 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19
> seconds.
>
>   ==> /var/log/docker-logs/remote/
> node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:59:13.422414+01:00 info:  2014-09-24 18:59:13.419 19184
> INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server
> on 127.0.0.1:5673
>   2014-09-24T19:59:13.424022+01:00 info:  2014-09-24 18:59:13.420 19184
> INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0
> seconds...
>   2014-09-24T19:59:18.433425+01:00 err:  2014-09-24 18:59:18.430 19184
> ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on
> 192.168.0.3:5673 is unreachable: timed out. Trying again in 30 seconds.
>
>   ==> /var/log/docker-logs/remote/
> node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:59:28.490133+01:00 info:  2014-09-24 18:59:28.487 10059
> INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server
> on 192.168.0.5:5673
>   2014-09-24T19:59:28.491235+01:00 info:  2014-09-24 18:59:28.488 10059
> INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0
> seconds...
>
>   ==> /var/log/docker-logs/remote/
> node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:59:30.479988+01:00 info:  2014-09-24 18:59:30.481 30562
> INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server
> on 192.168.0.3:5673
>   2014-09-24T19:59:30.481187+01:00 info:  2014-09-24 18:59:30.482 30562
> INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0
> seconds...
>
>   ==> /var/log/docker-logs/remote/
> node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:59:33.515571+01:00 err:  2014-09-24 18:59:33.512 10059
> ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on
> 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19
> seconds.
>
>   ==> /var/log/docker-logs/remote/
> node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:59:35.504238+01:00 err:  2014-09-24 18:59:35.505 30562
> ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on
> 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30
> seconds.
>
>   ==> /var/log/docker-logs/remote/
> node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:59:48.464659+01:00 info:  2014-09-24 18:59:48.461 19184
> INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server
> on 127.0.0.1:5673
>   2014-09-24T19:59:48.465735+01:00 info:  2014-09-24 18:59:48.462 19184
> INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0
> seconds...
>
>   ==> /var/log/docker-logs/remote/
> node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:59:52.523420+01:00 info:  2014-09-24 18:59:52.520 10059
> INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server
> on 192.168.0.5:5673
>   2014-09-24T19:59:52.524857+01:00 info:  2014-09-24 18:59:52.522 10059
> INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0
> seconds...
>
>   ==> /var/log/docker-logs/remote/
> node-70.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:59:53.471741+01:00 err:  2014-09-24 18:59:53.468 19184
> ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on
> 192.168.0.4:5673 is unreachable: timed out. Trying again in 30 seconds.
>
>   ==> /var/log/docker-logs/remote/
> node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T19:59:57.533816+01:00 err:  2014-09-24 18:59:57.531 10059
> ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on
> 127.0.0.1:5673 is unreachable: [Errno 32] Broken pipe. Trying again in 19
> seconds.
>
>   ==> /var/log/docker-logs/remote/
> node-69.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T20:00:05.531734+01:00 info:  2014-09-24 19:00:05.533 30562
> INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server
> on 192.168.0.3:5673
>   2014-09-24T20:00:05.532827+01:00 info:  2014-09-24 19:00:05.534 30562
> INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0
> seconds...
>   2014-09-24T20:00:10.543608+01:00 err:  2014-09-24 19:00:10.545 30562
> ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on
> 192.168.0.5:5673 is unreachable: Socket closed. Trying again in 30
> seconds.
>
>   ==> /var/log/docker-logs/remote/
> node-68.lax3.ubiquity.io/ceilometer-agent-notification.log <==
>   2014-09-24T20:00:16.554651+01:00 info:  2014-09-24 19:00:16.551 10059
> INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server
> on 192.168.0.5:5673
>   2014-09-24T20:00:16.556032+01:00 info:  2014-09-24 19:00:16.553 10059
> INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0
> seconds...
>
>   root@node-68:~# crm status
>   Last updated: Wed Sep 24 19:01:46 2014
>   Last change: Wed Sep 24 19:00:30 2014 via crm_attribute on node-69
>   Stack: classic openais (with plugin)
>   Current DC: node-69 - partition with quorum
>   Version: 1.1.10-42f2063
>   3 Nodes configured, 3 expected votes
>   22 Resources configured
>
>   Online: [ node-68 node-69 node-70 ]
>
>    vip__management_old    (ocf::mirantis:ns_IPaddr2):     Started node-69
>    vip__public_old        (ocf::mirantis:ns_IPaddr2):     Started node-70
>    p_ceilometer-alarm-evaluator
>  (ocf::mirantis:ceilometer-alarm-evaluator):     Started node-68
>    p_ceilometer-agent-central
>  (ocf::mirantis:ceilometer-agent-central):       Started node-69
>    Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
>        Masters: [ node-68 ]
>        Slaves: [ node-69 node-70 ]
>    Clone Set: clone_p_mysql [p_mysql]
>        Started: [ node-68 node-69 node-70 ]
>    Clone Set: clone_p_haproxy [p_haproxy]
>        Started: [ node-68 node-69 node-70 ]
>    p_heat-engine  (ocf::mirantis:heat-engine):    Started node-69
>    Clone Set: clone_p_neutron-plugin-openvswitch-agent
> [p_neutron-plugin-openvswitch-agent]
>        Started: [ node-68 node-69 node-70 ]
>    Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
>        Started: [ node-68 node-69 node-70 ]
>    p_neutron-dhcp-agent   (ocf::mirantis:neutron-agent-dhcp):     Started
> node-69
>    p_neutron-l3-agent     (ocf::mirantis:neutron-agent-l3):       Started
> node-68
>
>   Failed actions:
>       p_mysql_monitor_120000 (node=node-68, call=97, rc=1,
> status=complete, last-rc-change=Wed Sep 24 15:17:52 2014
>   , queued=0ms, exec=0ms
>   ): unknown error
>       p_neutron-metadata-agent_monitor_60000 (node=node-68, call=283,
> rc=7, status=complete, last-rc-change=Wed Sep 24 01:58:05 2014
>   , queued=0ms, exec=0ms
>   ): not running
>
>   Fuel Snapshot: http://23.80.0.2:8000/dump/fuel-
>   snapshot-2014-09-24_20-01-30.tgz
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1373569/+subscriptions
>