Juniper Openstack

SM:3.1build 23 HA : ceilometer sample floating ip packets failed

Bug #1612774 reported by sundarkh on 2016-08-12

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R3.0	Won't Fix	Medium	Nikhil Bansal	Juniper Openstack r3.0.3.0
R3.1	Fix Committed	Medium	Nikhil Bansal	Juniper Openstack r3.1.1.0
R3.2	Fix Committed	Medium	Nitish Krishna Kaveri
Trunk	Fix Committed	Medium	Nikhil Bansal	Juniper Openstack r3.2.0.0-fcs "r3.2.0.0"

Bug Description

test_sample_floating_ip_transmit_packets failed with connection error

HA setup

   "server": [
        {
            "cluster_id": "cluster5sanity",
            "id": "nodeg21",
            "ip_address": "10.204.217.61",
            "roles": [
                "control",
                "webui",
                "openstack",
                "database",
                "collector",
                "config"
            ]
        },
        {
            "cluster_id": "cluster5sanity",
            "id": "nodec38",
            "ip_address": "10.204.217.23",
            "roles": [
                "compute"
            ]
        },
        {
            "cluster_id": "cluster5sanity",
            "id": "nodeg17",
            "ip_address": "10.204.217.57",
            "roles": [
                "compute"
            ]
        },
        {
            "cluster_id": "cluster5sanity",
            "id": "nodec58",
            "ip_address": "10.204.217.98",
            "roles": [
                "control",
                "webui",
                "openstack",
                "database",
                "collector",
                "config"
            ]
        },
        {
            "cluster_id": "cluster5sanity",
            "id": "nodeg33",
            "ip_address": "10.204.217.73",
            "roles": [
                "control",
                "webui",
                "openstack",
                "database",
                "collector",
                "config"
            ]
        }
    ]
}

cfgm 0 nodeg21(ceilometer collector log)
13532 2016-08-12 11:38:23.858 17232 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('10.204.217.61', 27017), ('10.204.217.98', 27017), ('10.204.217.73', 27017)]
  13533 2016-08-12 11:38:23.860 17232 WARNING ceilometer.storage.mongo.utils [-] Unable to connect to the database server: could not connect to 10.204.217.73:27017: [Errno 111] ECONNREFUSED, could not connect to 10.204.217.61:27017: [Errno 111] ECONNREFUSED, could not connect to 10.204.217.98:27017: [Errno 111] ECONNREFUSED.
  13534 2016-08-12 11:38:33.860 17232 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.7/dist-packages/ceilometer/storage/__init__.py:130
  13535 2016-08-12 11:38:33.861 17232 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('10.204.217.61', 27017), ('10.204.217.98', 27017), ('10.204.217.73', 27017)]

test_sample_floating_ip_transmit_packets failed with connection error

ceilometer compute log in nodec58 (compute)

Traceback (most recent call last):
  File "tcutils/wrappers.py", line 73, in wrapper
    result = function(self, *args, **kwargs)
  File "scripts/ceilometer_tests/test_ceilometer.py", line 104, in test_sample_floating_ip_transmit_packets
    result = ceilometer_client.sample_list(self.cclient,m)
  File "/contrail-test/fixtures/ceilometer/ceilometer_client.py", line 207, in sample_list
    q=query, limit=limit)
  File "/usr/lib/python2.7/dist-packages/ceilometerclient/v2/samples.py", line 49, in list
    return self._list(options.build_url(path, q, params))
  File "/usr/lib/python2.7/dist-packages/ceilometerclient/common/base.py", line 68, in _list
    resp = self.api.get(url)
  File "/usr/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 355, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 189, in request
    resp = self.http.request(method, url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 375, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='10.204.217.176', port=8777): Max retries exceeded with url: /v2/meters/ip.floating.transmit.packets (Caused by <class 'socket.error'>: [Errno 111] Connection refused)

ceilometer compute log

122527 2016-08-12 11:36:37.175 1580 ERROR ceilometer.nova_client [-] HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by ProtocolError('Connection aborted.', error(111, 'ECONNREFUSED')))
122528 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client Traceback (most recent call last):
122529 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 48, in with_logging
122530 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client return func(*args, **kwargs)
122531 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 151, in instance_get_all_by_host
122532 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client search_opts=search_opts))
122533 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 617, in list
122534 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client return self._list("/servers%s%s" % (detail, query_string), "servers")
122535 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
122536 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client _resp, body = self.api.client.get(url)
122537 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 440, in get
122538 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client return self._cs_request(url, 'GET', **kwargs)
122539 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 399, in _cs_request
122540 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client self.authenticate()
122541 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 543, in authenticate
122542 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client auth_url = self._v2_auth(auth_url)
122543 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 634, in _v2_auth
122544 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client return self._authenticate(url, body)
122545 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 647, in _authenticate
122546 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client **kwargs)
122547 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 392, in _time_request
122548 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client resp, body = self.request(url, method, **kwargs)
122549 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 363, in request
122550 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client **kwargs)
122551 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
122552 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client return session.request(method=method, url=url, **kwargs)
122553 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
122554 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client resp = self.send(prep, **send_kwargs)
122555 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
122556 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client r = adapter.send(request, **kwargs)
122557 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
122558 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client raise ConnectionError(e)
122559 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by ProtocolError('Connection aborted.', error(111, 'ECONNREFUSED')))
122560 2016-08-12 11:36:37.175 1580 TRACE ceilometer.nova_client
122561 2016-08-12 11:36:37.176 1580 ERROR ceilometer.agent.base [-] Unable to discover resources: None: Max retries exceeded with url: /v2.0/tokens (Caused by redirect)
122562 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base Traceback (most recent call last):
122563 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/ceilometer/agent/base.py", line 346, in discover
122564 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base discovered = discoverer.discover(self, param)
122565 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/ceilometer/compute/discovery.py", line 41, in discover
122566 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base cfg.CONF.host, self.last_run)
122567 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 48, in with_logging

122568 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base return func(*args, **kwargs)
122569 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/ceilometer/nova_client.py", line 151, in instance_get_all_by_host
122570 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base search_opts=search_opts))
122571 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 617, in list
122572 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base return self._list("/servers%s%s" % (detail, query_string), "servers")
122573 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
122574 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base _resp, body = self.api.client.get(url)
122575 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 440, in get
122576 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base return self._cs_request(url, 'GET', **kwargs)
122577 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 399, in _cs_request
122578 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base self.authenticate()
122579 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 543, in authenticate
122580 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base auth_url = self._v2_auth(auth_url)
122581 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 634, in _v2_auth
122582 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base return self._authenticate(url, body)
122583 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 647, in _authenticate
122584 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base **kwargs)
122585 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 392, in _time_request
122586 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base resp, body = self.request(url, method, **kwargs)
122587 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 363, in request
122588 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base **kwargs)
122589 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
122590 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base return session.request(method=method, url=url, **kwargs)
122591 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
122592 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base resp = self.send(prep, **send_kwargs)
122593 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
122594 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base r = adapter.send(request, **kwargs)
122595 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
122596 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base raise ConnectionError(e)
122597 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by ProtocolError('Connection aborted.', error(111, 'ECONNREFUSED')))
122598 2016-08-12 11:36:37.176 1580 TRACE ceilometer.agent.base
122599 2016-08-12 11:36:37.177 1580 INFO ceilometer.agent.base [-] Skip polling pollster network.outgoing.bytes, no resources found
~
~
~ https://bugs.launchpad.net/juniperopenstack/+bug/1610506

Tags:

Revision history for this message

Nitish Krishna Kaveri (nitishk) wrote on 2016-08-31:

Not sure if Ceilometer is supported yet in HA

Changed in juniperopenstack:
importance:	High → Medium

Revision history for this message

sundarkh (sundar-kh) wrote on 2016-09-14:

Seen with R3.0 build 60, Kilo

Revision history for this message

Abhay Joshi (abhayj) wrote on 2016-09-15:

Nikhil is looking into this.

Raj Reddy (rajreddy) on 2016-09-19

tags:	added: analytics
Changed in juniperopenstack:
assignee:	Abhay Joshi (abhayj) → Nikhil Bansal (nikhilb-u)

Revision history for this message

Nikhil Bansal (nikhilb-u) wrote on 2016-09-21:

ceilometer HA is not supported in 3.0
It is supported 3.1 onwards.

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-09-22: [Review update] master

Review in progress for https://review.opencontrail.org/24368
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-09-22: A change has been merged

Reviewed: https://review.opencontrail.org/24368
Committed: http://github.org/Juniper/contrail-puppet/commit/7e63d1229b0c5beb5f1eca72ebead476b55560be
Submitter: Zuul
Branch: master

commit 7e63d1229b0c5beb5f1eca72ebead476b55560be
Author: nitishkrishna <email address hidden>
Date: Wed Sep 21 22:02:35 2016 -0700

Closes-Bug: #1612774 - Ceilometer HA support

Needs integration with tooz + couple missing params

Change-Id: I89f0478455544a25d9fc66a2ec8e622a51cfa897

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-11-05: [Review update] R3.1

Review in progress for https://review.opencontrail.org/25749
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-11-05: [Review update] R3.2

#10

Review in progress for https://review.opencontrail.org/25750
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-11-05: A change has been merged

#12

Reviewed: https://review.opencontrail.org/25749
Committed: http://github.org/Juniper/contrail-puppet/commit/a951d15596c3e54dc7c02f9c5b6500452fbca35b
Submitter: Zuul
Branch: R3.1

commit a951d15596c3e54dc7c02f9c5b6500452fbca35b
Author: nitishkrishna <email address hidden>
Date: Fri Nov 4 17:47:40 2016 -0700

Closes-Bug: #1638814 - Ceilometer processes restart too much

The ordering was missing and rpc_backend being removed also caused too many restarts

Closes-Bug: #1612774 - Ceilometer HA support

Needs integration with tooz + couple missing params

Change-Id: I8640c01881836b8c61c4943dd8003d08def6b647

Revision history for this message

Nitish Krishna Kaveri (nitishk) wrote on 2016-11-10:

#13

will close in 3.2 as part of :
https://review.opencontrail.org/#/c/25813/

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.