Precondition: HA Cluster Fuel 4.0 All controllers and services are up as reported by crm status One controller recently rebooted or had mysql restarted When I use the command "nova service-list" (as an example) to get a list of openstack nodes and roles I expect output that contains a list of openstack services. Instead I get a timeout. For reference: node-16=10.0.5.2 node-17=10.0.5.3 node-18=10.0.5.4 vip__management_old=10.0.5.10 [root@node-18 ~]# nova service-list ERROR: HTTPConnectionPool(host='10.0.5.10', port=5000): Max retries exceeded with url: /v2.0/tokens [root@node-18 ~]# crm status Last updated: Tue Mar 18 06:50:48 2014 Last change: Tue Mar 18 06:39:53 2014 via crmd on node-16.domain.com Stack: openais Current DC: node-16.domain.com - partition with quorum Version: 1.1.8-1.el6-1f8858c 3 Nodes configured, 3 expected votes 19 Resources configured. Online: [ node-16.domain.com node-17.domain.com node-18.domain.com ] vip__management_old (ocf::heartbeat:IPaddr2): Started node-16.domain.com vip__public_old (ocf::heartbeat:IPaddr2): Started node-17.domain.com Clone Set: clone_p_haproxy [p_haproxy] Started: [ node-16.domain.com node-17.domain.com node-18.domain.com ] Clone Set: clone_p_mysql [p_mysql] Started: [ node-16.domain.com node-17.domain.com node-18.domain.com ] Clone Set: clone_p_neutron-openvswitch-agent [p_neutron-openvswitch-agent] Started: [ node-16.domain.com node-17.domain.com node-18.domain.com ] Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent] Started: [ node-16.domain.com node-17.domain.com node-18.domain.com ] p_neutron-dhcp-agent (ocf::mirantis:neutron-agent-dhcp): Started node-18.domain.com p_neutron-l3-agent (ocf::mirantis:neutron-agent-l3): Started node-16.domain.com openstack-heat-engine (ocf::mirantis:openstack-heat-engine): Started node-17.domain.com p_openstack-ceilometer-central (ocf::mirantis:ceilometer-agent-central): Started node-18.domain.com p_openstack-ceilometer-alarm-evaluator (ocf::mirantis:ceilometer-alarm-evaluator): Started node-16.domain.com Failed actions: p_mysql_monitor_60000 (node=node-18.domain.com, call=95, rc=7, status=complete): not running [root@node-18 ~]# crm resource restart clone_p_haproxy INFO: ordering clone_p_haproxy to stop INFO: ordering clone_p_haproxy to start [root@node-18 ~]# mysql -e "show status like 'wsrep%';" +----------------------------+----------------------------------------------+ | Variable_name | Value | +----------------------------+----------------------------------------------+ | wsrep_local_state_uuid | | | wsrep_protocol_version | 4 | | wsrep_last_committed | 18446744073709551615 | | wsrep_replicated | 0 | | wsrep_replicated_bytes | 0 | | wsrep_received | 1 | | wsrep_received_bytes | 274 | | wsrep_local_commits | 0 | | wsrep_local_cert_failures | 0 | | wsrep_local_bf_aborts | 0 | | wsrep_local_replays | 0 | | wsrep_local_send_queue | 0 | | wsrep_local_send_queue_avg | 0.000000 | | wsrep_local_recv_queue | 40 | | wsrep_local_recv_queue_avg | 0.000000 | | wsrep_flow_control_paused | 0.000000 | | wsrep_flow_control_sent | 0 | | wsrep_flow_control_recv | 0 | | wsrep_cert_deps_distance | 0.000000 | | wsrep_apply_oooe | 0.000000 | | wsrep_apply_oool | 0.000000 | | wsrep_apply_window | 0.000000 | | wsrep_commit_oooe | 0.000000 | | wsrep_commit_oool | 0.000000 | | wsrep_commit_window | 0.000000 | | wsrep_local_state | 1 | | wsrep_local_state_comment | Joining: receiving State Transfer | | wsrep_cert_index_size | 0 | | wsrep_causal_reads | 0 | | wsrep_incoming_addresses | 10.0.5.4:3307,10.0.5.2:3307,10.0.5.3:3307 | | wsrep_cluster_conf_id | 32 | | wsrep_cluster_size | 3 | | wsrep_cluster_state_uuid | 1757119a-ae06-11e3-0800-3a0e97c7179f | | wsrep_cluster_status | Primary | | wsrep_connected | ON | | wsrep_local_index | 0 | | wsrep_provider_name | Galera | | wsrep_provider_vendor | Codership Oy