Gnocchi

devstack installation failed when enabling ceilometer/gnocchi at the same time

Bug #1649259 reported by Lianhao Lu on 2016-12-12

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	Ceilometer	Fix Released	Undecided	Lianhao Lu
	Gnocchi	Invalid	Undecided	Unassigned

Bug Description

When have the following settings in devstack's local.conf,

enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer
CEILOMETER_BACKEND=gnocchi
enable_plugin aodh https://git.openstack.org/openstack/aodh.git
enable_plugin gnocchi https://git.openstack.org/openstack/gnocchi

running ./stack.sh would fail with the following output:

-12-12 06:17:07.392 | ++/opt/stack/ceilometer/devstack/plugin.sh:init_ceilometer:372 '[' gnocchi = gnocchi ']'
2016-12-12 06:17:07.402 | ++/opt/stack/ceilometer/devstack/plugin.sh:init_ceilometer:373 /usr/local/bin/ceilometer-upgrade --skip-metering-databa
se --skip-event-database
2016-12-12 06:17:09.359 | WARNING:oslo_config.cfg:Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the
future.
2016-12-12 06:17:09.363 | 2016-12-12 14:17:09.363 25061 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by
default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2016-12-12 06:17:09.363 | 2016-12-12 14:17:09.363 25061 INFO ceilometer.cmd.storage [-] Skipping metering database upgrade
2016-12-12 06:17:09.364 | 2016-12-12 14:17:09.363 25061 INFO ceilometer.cmd.storage [-] Skipping event database upgrade
2016-12-12 06:17:09.364 | 2016-12-12 14:17:09.364 25061 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /opt/stack/ceilometer/ceil
ometer/cmd/storage.py:60
2016-12-12 06:17:09.668 | 2016-12-12 14:17:09.666 25061 CRITICAL ceilometer [-] ConnectFailure: Unable to establish connection to http://10.239.159.49:8041/
v1/resource_type/ceph_account: HTTPConnectionPool(host='10.239.159.49', port=8041): Max retries exceeded with url: /v1/resource_type/ceph_account (Caused by
NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f118ea8fad0>: Failed to establish a new connection: [Errno 111] Conne
ction refused',))
2016-12-12 06:17:09.668 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer Traceback (most recent call last):
2016-12-12 06:17:09.668 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer File "/usr/local/bin/ceilometer-upgrade", line 10, in <module>
2016-12-12 06:17:09.668 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer sys.exit(upgrade())
2016-12-12 06:17:09.668 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer File "/opt/stack/ceilometer/ceilometer/cmd/storage.py", line 62, in upgrade
2016-12-12 06:17:09.668 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer gnocchi_client.upgrade_resource_types(conf)
2016-12-12 06:17:09.668 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer File "/opt/stack/ceilometer/ceilometer/gnocchi_client.py", line 105, in upgrade_r
esource_types
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer gnocchi.resource_type.get(name=name)
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer File "/usr/local/lib/python2.7/dist-packages/gnocchiclient/v1/resource_type.py",
line 44, in get
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer headers={'Content-Type': "application/json"}).json()
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer File "/usr/local/lib/python2.7/dist-packages/gnocchiclient/v1/base.py", line 37,
in _get
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer return self.client.api.get(*args, **kwargs)
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 216,
in get
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer return self.request(url, 'GET', **kwargs)
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer File "/usr/local/lib/python2.7/dist-packages/gnocchiclient/client.py", line 35, i
n request
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer **kwargs)
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 141,
in request
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer return self.session.request(url, method, **kwargs)
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer File "/usr/local/lib/python2.7/dist-packages/positional/__init__.py", line 101, i
n inner
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer return wrapped(*args, **kwargs)
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 592,
in request
2016-12-12 06:17:09.669 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer resp = send(**kwargs)
2016-12-12 06:17:09.670 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 642,
in _send_request
2016-12-12 06:17:09.670 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer raise exceptions.ConnectFailure(msg)
2016-12-12 06:17:09.670 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer ConnectFailure: Unable to establish connection to http://10.239.159.49:8041/v1/reso
urce_type/ceph_account: HTTPConnectionPool(host='10.239.159.49', port=8041): Max retries exceeded with url: /v1/resource_type/ceph_account (Caused by NewCon
nectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f118ea8fad0>: Failed to establish a new connection: [Errno 111] Connection r
efused',))
2016-12-12 06:17:09.670 | 2016-12-12 14:17:09.666 25061 ERROR ceilometer
2016-12-12 06:17:09.741 | +/opt/stack/ceilometer/devstack/plugin.sh:init_ceilometer:1 exit_trap
2016-12-12 06:17:09.751 | +./stack.sh:exit_trap:487 local r=1
2016-12-12 06:17:09.763 | ++./stack.sh:exit_trap:488 jobs -p
2016-12-12 06:17:09.775 | +./stack.sh:exit_trap:488 jobs=
2016-12-12 06:17:09.785 | +./stack.sh:exit_trap:491 [[ -n '' ]]
2016-12-12 06:17:09.795 | +./stack.sh:exit_trap:497 kill_spinner
2016-12-12 06:17:09.805 | +./stack.sh:kill_spinner:383 '[' '!' -z '' ']'
2016-12-12 06:17:09.816 | +./stack.sh:exit_trap:499 [[ 1 -ne 0 ]]
2016-12-12 06:17:09.825 | +./stack.sh:exit_trap:500 echo 'Error on exit'
2016-12-12 06:17:09.825 | Error on exit

Revision history for this message

TonyCheng (tony-pig) wrote on 2016-12-13:

Download full text (8.1 KiB)

Here is my local.conf

# Sample ``local.conf`` for user-configurable variables in ``stack.sh``

# NOTE: Copy this file to the root DevStack directory for it to work properly.

# ``local.conf`` is a user-maintained settings file that is sourced from ``stackrc``.
# This gives it the ability to override any variables set in ``stackrc``.
# Also, most of the settings in ``stack.sh`` are written to only be set if no
# value has already been set; this lets ``local.conf`` effectively override the
# default values.

# This is a collection of some of the settings we have found to be useful
# in our DevStack development environments. Additional settings are described
# in http://devstack.org/local.conf.html
# These should be considered as samples and are unsupported DevStack code.

# The ``localrc`` section replaces the old ``localrc`` configuration file.
# Note that if ``localrc`` is present it will be used in favor of this section.
[[local|localrc]]

# Minimal Contents
# ----------------

# While ``stack.sh`` is happy to run without ``localrc``, devlife is better when
# there are a few minimal variables set:

# If the ``*_PASSWORD`` variables are not set here you will be prompted to enter
# values for them by ``stack.sh``and they will be added to ``local.conf``.
ADMIN_PASSWORD=Abc12345
DATABASE_PASSWORD=Abc12345
RABBIT_PASSWORD=Abc12345
SERVICE_PASSWORD=$ADMIN_PASSWORD

#GIT_BASE=http://git.trystack.cn
# ``HOST_IP`` and ``HOST_IPV6`` should be set manually for best results if
# the NIC configuration of the host is unusual, i.e. ``eth1`` has the default
# route but ``eth0`` is the public interface. They are auto-detected in
# ``stack.sh`` but often is indeterminate on later runs due to the IP moving
# from an Ethernet interface to a bridge on the host. Setting it here also
# makes it available for ``openrc`` to include when setting ``OS_AUTH_URL``.
# Neither is set by default.
#HOST_IP=w.x.y.z
#HOST_IPV6=2001:db8::7
MULTI_HOST=True

# Logging
# -------

# By default ``stack.sh`` output only goes to the terminal where it runs. It can
# be configured to additionally log to a file by setting ``LOGFILE`` to the full
# path of the destination log file. A timestamp will be appended to the given name.
LOGFILE=$DEST/logs/stack.sh.log

# Old log files are automatically removed after 7 days to keep things neat. Change
# the number of days by setting ``LOGDAYS``.
LOGDAYS=2

# Nova logs will be colorized if ``SYSLOG`` is not set; turn this off by setting
# ``LOG_COLOR`` false.
LOG_COLOR=False

# http://docs.openstack.org/developer/devstack/configuration.html#logging-the-service-output
LOGDIR=$DEST/logs

# Using milestone-proposed branches
# ---------------------------------
GIT_BASE=https://github.com

NOVA_REPO=$GIT_BASE/openstack/nova.git
NOVA_BRANCH=stable/newton

CIDER_REPO=$GIT_BASE/openstack/cinder.git
CINDER_BRANCH=stable/newton

NEUTRON_REPO=$GIT_BASE/openstack/neutron.git
NEUTRON_BRANCH=stable/newton

HEAT_REPO=$GIT_BASE/openstack/heat.git
HEAT_BRANCH=stable/newton

HORIZON_REPO=$GIT_BASE/openstack/horizon.git
HORIZON_BRANCH=stable/newton

KEYSTONE_REPO=$GIT_BASE/openstack/keystone.git
KEYSTONE_BRANCH=stable/newton

GLANCE_REPO=$GIT_BASE/openstack/g...

Here is my local.conf

# Sample ``local.conf`` for user-configurable variables in ``stack.sh``

# NOTE: Copy this file to the root DevStack directory for it to work properly.

# The ``localrc`` section replaces the old ``localrc`` configuration file.
# Note that if ``localrc`` is present it will be used in favor of this section.
[[local|localrc]]

# Minimal Contents
# ----------------

# While ``stack.sh`` is happy to run without ``localrc``, devlife is better when
# there are a few minimal variables set:

#GIT_BASE=http://git.trystack.cn
# ``HOST_IP`` and ``HOST_IPV6`` should be set manually for best results if
# the NIC configuration of the host is unusual, i.e. ``eth1`` has the default
# route but ``eth0`` is the public interface.  They are auto-detected in
# ``stack.sh`` but often is indeterminate on later runs due to the IP moving
# from an Ethernet interface to a bridge on the host. Setting it here also
# makes it available for ``openrc`` to include when setting ``OS_AUTH_URL``.
# Neither is set by default.
#HOST_IP=w.x.y.z
#HOST_IPV6=2001:db8::7
MULTI_HOST=True

# Logging
# -------

# By default ``stack.sh`` output only goes to the terminal where it runs.  It can
# be configured to additionally log to a file by setting ``LOGFILE`` to the full
# path of the destination log file.  A timestamp will be appended to the given name.
LOGFILE=$DEST/logs/stack.sh.log

# Old log files are automatically removed after 7 days to keep things neat.  Change
# the number of days by setting ``LOGDAYS``.
LOGDAYS=2

# Nova logs will be colorized if ``SYSLOG`` is not set; turn this off by setting
# ``LOG_COLOR`` false.
LOG_COLOR=False

# http://docs.openstack.org/developer/devstack/configuration.html#logging-the-service-output
LOGDIR=$DEST/logs

# Using milestone-proposed branches
# ---------------------------------
GIT_BASE=https://github.com

NOVA_REPO=$GIT_BASE/openstack/nova.git
NOVA_BRANCH=stable/newton

CIDER_REPO=$GIT_BASE/openstack/cinder.git
CINDER_BRANCH=stable/newton

NEUTRON_REPO=$GIT_BASE/openstack/neutron.git
NEUTRON_BRANCH=stable/newton

HEAT_REPO=$GIT_BASE/openstack/heat.git
HEAT_BRANCH=stable/newton

HORIZON_REPO=$GIT_BASE/openstack/horizon.git
HORIZON_BRANCH=stable/newton

KEYSTONE_REPO=$GIT_BASE/openstack/keystone.git
KEYSTONE_BRANCH=stable/newton

GLANCE_REPO=$GIT_BASE/openstack/glance.git
GLANCE_BRANCH=stable/newton

CEILOMETER_REPO=$GIT_BASE/openstack/ceilometer.git
CEILOMETER_BRANCH=stable/newton

# Uncomment these to grab the milestone-proposed branches from the
# repos:
#CINDER_BRANCH=milestone-proposed
#GLANCE_BRANCH=milestone-proposed
#HORIZON_BRANCH=milestone-proposed
#KEYSTONE_BRANCH=milestone-proposed
#KEYSTONECLIENT_BRANCH=milestone-proposed
#NOVA_BRANCH=milestone-proposed
#NOVACLIENT_BRANCH=milestone-proposed
#NEUTRON_BRANCH=milestone-proposed
#SWIFT_BRANCH=milestone-proposed
## Branches
#

# Using git versions of clients
# -----------------------------
# By default clients are installed from pip.  See LIBS_FROM_GIT in
# stackrc for details on getting clients from specific branches or
# revisions.  e.g.
# LIBS_FROM_GIT="python-ironicclient"
# IRONICCLIENT_BRANCH=refs/changes/44/2.../1
# Identity
ENABLE_IDENTITY_V2=False

# Image
IMAGE_URLS="http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img"

# Swift
# -----

# Swift is now used as the back-end for the S3-like object store. Setting the
# hash value is required and you will be prompted for it if Swift is enabled
# so just set it to something already:
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5

# For development purposes the default of 3 replicas is usually not required.
# Set this to 1 to save some resources:
# SWIFT_REPLICAS=1

# The data for Swift is stored by default in (``$DEST/data/swift``),
# or (``$DATA_DIR/swift``) if ``DATA_DIR`` has been set, and can be
# moved by setting ``SWIFT_DATA_DIR``. The directory will be created
# if it does not exist.
# SWIFT_DATA_DIR=$DEST/data

# Disable nova-network and enable neutron
disable_service n-net
enable_service q-svc
enable_service q-dhcp
enable_service q-meta
enable_service q-agt
enable_service q-l3
#enable_service q-vpn

# Neutron options
HOST_IP=192.168.230.180
PUBLIC_INTERFACE=ens33
PUBLIC_NETWORK_GATEWAY=192.168.230.2
FLOATING_RANGE=192.168.230.0/24
Q_FLOATING_ALLOCATION_POOL=start=192.168.230.200,end=192.168.230.230
Q_USE_SECGROUP=True
Q_L3_ENABLED=True

# Neutron options :: Open vSwitch provider networking configuration
PUBLIC_BRIDGE=br-ex
Q_USE_PROVIDERNET_FOR_PUBLIC=True
OVS_PHYSICAL_BRIDGE=br-ex
OVS_BRIDGE_MAPPINGS=public:br-ex

# Enable Heat
enable_service h-eng
enable_service h-api
enable_service h-api-cfn
enable_service h-api-cw

# Enable Gnocchi with Grafana
enable_plugin gnocchi https://github.com/openstack/gnocchi.git stable/3.0
enable_service gnocchi-api
enable_service gnocchi-metricd
#enable_service gnocchi-grafana
GNOCCHI_ARCHIVE_POLICY=medium
GNOCCHI_STORAGE_BACKEND=file

# Enable Ceilometer
#enable_plugin ceilometer http://git.trystack.cn/openstack/ceilometer.git stable/newton
enable_plugin ceilometer https://github.com/openstack/ceilometer.git stable/newton
enable_service ceilometer-acompute ceilometer-acentral ceilometer-anotification ceilometer-collector ceilometer-api
#enable_service ceilometer-alarm-notifier ceilometer-alarm-evaluator
CEILOMETER_EVENT_ALARM=True
#CEILOMETER_BACKEND=mongodb
CEILOMETER_BACKEND=gnocchi
CEILOMETER_PIPELINE_INTERVAL=60

# Enable Adoh
#enable_plugin aodh http://git.trystack.cn/openstack/aodh.git stable/newton
enable_plugin aodh https://github.com/openstack/aodh.git stable/newton
enable_service aodh-evaluator aodh-notifier aodh-api aodh-listener
disable_service Ceilometer-alarm-notifier Ceilometer-alarm-evaluator

# Enable VPNaaS
#enable_plugin neutron-vpnaas http://git.trystack.cn/openstack/neutron-vpnaas.git stable/newton
#enable_plugin neutron-vpnaas https://github.com/openstack/neutron-vpnaas.git  stable/newton

# Enable FWaaS
#enable_plugin neutron-fwaas http://git.trystack.cn/openstack/neutron-fwaas.git stable/newton
#enable_plugin neutron-fwaas https://github.com/openstack/neutron-fwaas.git stable/newton
#enable_service q-fwaas

# Enable LBaaS
#enable_plugin neutron-lbaas http://git.trystack.cn/openstack/neutron-lbaas.git stable/newton
#enable_plugin neutron-lbaas https://github.com/openstack/neutron-lbaas.git stable/newton
#enable_service q-lbaasv2

#enable_plugin octavia http://git.trystack.cn/openstack/octavia.git master
#enable_plugin octavia https://github.com/openstack/octavia.git stable/newton
#enable_service octavia
#enable_service o-cw
#enable_service o-hk
#enable_service o-hm
#enable_service o-api

# Enable SkyDive
#enable_plugin skydive https://github.com/skydive-project/skydive.git
#enable_service skydive-agent skydive-analyzer

# Enable Panko
enable_plugin panko https://github.com/openstack/panko.git stable/newton

# Capacity
VOLUME_BACKING_FILE_SIZE=48000M

# Dashboard for LBaaS
#enable_plugin neutron-lbaas-dashboard https://github.com/openstack/neutron-lbaas-dashboard.git stable/newton
#enable_service horizon

# SPICE
enable_service n-spice
disable_service n-novnc n-xvnc

SPICESERVER_PROXYCLIENT_ADDRESS=192.168.230.180
SPICEHTML5PROXY_URL=http://192.168.230.180:6082/spice_auto.html
SPICE_REPO=https://github.com/SPICE/spice-html5.git
SPICE_FROM_PACKAGE=True
SPICESERVER_LISTEN=0.0.0.0

# OpenStack requirement
#USE_CONSTRAINTS=True

[[post-config|/$Q_PLUGIN_CONF_FILE]]
global_physnet_mtu = 1400

Revision history for this message

gordon chung (chungg) wrote on 2016-12-13:

you should enable gnocchi befoer ceilometer. it's probably attempting to connect to gnocchi but times out before gnocchi service is started.

Changed in ceilometer:
status:	New → Incomplete
Changed in gnocchi:
status:	New → Incomplete

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-12-14: Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/410505

Changed in ceilometer:
status:	Incomplete → In Progress

gordon chung (chungg) on 2016-12-14

Changed in gnocchi:
status:	Incomplete → Invalid

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-12-20: Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/410505
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=34731eed8bd6ec4cde40465aab7d12d35e63c83b
Submitter: Jenkins
Branch: master

commit 34731eed8bd6ec4cde40465aab7d12d35e63c83b
Author: Lianhao Lu <email address hidden>
Date: Wed Dec 14 09:59:41 2016 +0800

Make sure gnocchi is enabled before ceilometer

Ceilometer upgrade will contact gnocchi so we must make sure gnocchi is
enabled before ceilometer in devstack.

Change-Id: I876b6b7551b00991685c3f9dee7610668b8a8864
Closes-Bug: #1649259

Changed in ceilometer:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-02: Fix included in openstack/ceilometer 8.0.0

This issue was fixed in the openstack/ceilometer 8.0.0 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.