Bug #1434013 “MongoDB 3.X cannot be used as database backend” : Bugs : Ceilometer

gordon chung (chungg) on 2015-03-19

Changed in ceilometer:
status:	New → Incomplete
status:	Incomplete → Triaged

Revision history for this message

gordon chung (chungg) wrote on 2015-03-19:

#1

is this with or without these steps: http://docs.mongodb.org/v3.0/release-notes/3.0-scram/#upgrade-scenarios

Revision history for this message

gordon chung (chungg) wrote on 2015-03-19:

#2

can you also post logs from collector? and pymongo version

Revision history for this message

Marcin Zbik (zbikmarc+launchpad) wrote on 2015-03-19:

#3

@ Comment #1
this is about updating from 2.6 to 3.0 when that problem will not occure due to old users having MONGODB-CR hashes already existing in databse.

Bug is about totally new installation with Mongo 3.0

@ Comment #2
I will do this at morning (CET), dont have access to logs now. But there should be no logs so far. It is new installation, so it happens during init of database.

Revision history for this message

Marcin Zbik (zbikmarc+launchpad) wrote on 2015-03-20:

#4

# ceilometer-dbsync --debug
2015-03-19 10:33:32.147 18284 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-19 10:33:32.232 18284 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('127.0.0.1', 27017)]
2015-03-19 10:33:42.240 18284 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-19 10:33:42.241 18284 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('127.0.0.1', 27017)]

logs from mongodb.log
2015-03-19T10:33:32.234+0100 I NETWORK [initandlisten] connection accepted from 127.0.0.1:57663 #14 (1 connection now open)
2015-03-19T10:33:32.236+0100 I ACCESS [conn14] authenticate db: ceilometer { authenticate: 1, user: "ceilometer", nonce: "xxx", key: "xxx" }
2015-03-19T10:33:32.237+0100 I ACCESS [conn14] Failed to authenticate ceilometer@ceilometer with mechanism MONGODB-CR: AuthenticationFailed MONGODB-CR credentials missing in the user document
2015-03-19T10:33:42.242+0100 I NETWORK [initandlisten] connection accepted from 127.0.0.1:57664 #15 (2 connections now open)
2015-03-19T10:33:42.244+0100 I ACCESS [conn15] authenticate db: ceilometer { authenticate: 1, user: "ceilometer", nonce: "xxx", key: "xxx" }
2015-03-19T10:33:42.245+0100 I ACCESS [conn15] Failed to authenticate ceilometer@ceilometer with mechanism MONGODB-CR: AuthenticationFailed MONGODB-CR credentials missing in the user document
2015-03-19T10:33:42.245+0100 I NETWORK [conn14] end connection 127.0.0.1:57663 (1 connection now open)

Revision history for this message

gordon chung (chungg) wrote on 2015-03-20:

#5

can you try updating to pymongo 2.8+.... i was able to connect with pymongo2.8 but got a similar error when using pymongo2.7.2

Revision history for this message

Marcin Zbik (zbikmarc+launchpad) wrote on 2015-03-23:

#6

Latest pymongo for RHEL 7 is 2.5.2 so far. I will try to install from outside repository.

Revision history for this message

Marcin Zbik (zbikmarc+launchpad) wrote on 2015-03-23:

#7

Download full text (11.4 KiB)

I updated to 2.8 but cilometer-dbsync fails:

]# ceilometer-dbsync --debug
2015-03-23 09:40:08.251 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:08.340 25985 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('127.0.0.1', 27017)]
2015-03-23 09:40:08.579 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.alarm.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:08.732 25985 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('127.0.0.1', 27017)]
2015-03-23 09:40:08.842 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:08.844 25985 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('127.0.0.1', 27017)]
2015-03-23 09:40:18.954 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:28.970 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:38.980 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:48.995 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:59.011 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:41:09.026 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:41:19.040 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:41:29.050 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:41:39.066 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:41:39.071 25985 CRITICAL ceilometer [-] OperationFailure: command SON([('listIndexes', u'event'), ('cursor', {})]) on namespace ceilometer.$cmd failed: no collection
2015-03-23 09:41:39.071 25985 TRACE ceilometer Traceback (most recent call last):
2015-03-23 09:41:39.071 25985 TRACE ceilometer File "/usr/bin/ceilometer-dbsync", line 10, i...

I updated to 2.8 but cilometer-dbsync fails:

]# ceilometer-dbsync --debug
2015-03-23 09:40:08.251 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:08.340 25985 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('127.0.0.1', 27017)]
2015-03-23 09:40:08.579 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.alarm.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:08.732 25985 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('127.0.0.1', 27017)]
2015-03-23 09:40:08.842 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:08.844 25985 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('127.0.0.1', 27017)]
2015-03-23 09:40:18.954 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:28.970 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:38.980 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:48.995 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:40:59.011 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:41:09.026 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:41:19.040 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:41:29.050 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:41:39.066 25985 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.event.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:127
2015-03-23 09:41:39.071 25985 CRITICAL ceilometer [-] OperationFailure: command SON([('listIndexes', u'event'), ('cursor', {})]) on namespace ceilometer.$cmd failed: no collection
2015-03-23 09:41:39.071 25985 TRACE ceilometer Traceback (most recent call last):
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/bin/ceilometer-dbsync", line 10, in <module>
2015-03-23 09:41:39.071 25985 TRACE ceilometer     sys.exit(dbsync())
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/cmd/storage.py", line 32, in dbsync
2015-03-23 09:41:39.071 25985 TRACE ceilometer     storage.get_connection_from_config(cfg.CONF, 'event').upgrade()
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py", line 117, in get_connection_from_config
2015-03-23 09:41:39.071 25985 TRACE ceilometer     return _inner()
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/retrying.py", line 49, in wrapped_f
2015-03-23 09:41:39.071 25985 TRACE ceilometer     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/retrying.py", line 212, in call
2015-03-23 09:41:39.071 25985 TRACE ceilometer     raise attempt.get()
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/retrying.py", line 247, in get
2015-03-23 09:41:39.071 25985 TRACE ceilometer     six.reraise(self.value[0], self.value[1], self.value[2])
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/retrying.py", line 200, in call
2015-03-23 09:41:39.071 25985 TRACE ceilometer     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py", line 115, in _inner
2015-03-23 09:41:39.071 25985 TRACE ceilometer     return get_connection(url, namespace)
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py", line 129, in get_connection
2015-03-23 09:41:39.071 25985 TRACE ceilometer     return mgr.driver(url)
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/event/storage/impl_mongodb.py", line 53, in __init__
2015-03-23 09:41:39.071 25985 TRACE ceilometer     self.upgrade()
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/event/storage/impl_mongodb.py", line 59, in upgrade
2015-03-23 09:41:39.071 25985 TRACE ceilometer     self.db.event)
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/storage/impl_mongodb.py", line 421, in update_ttl
2015-03-23 09:41:39.071 25985 TRACE ceilometer     indexes = coll.index_information()
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/storage/mongo/utils.py", line 399, in closure
2015-03-23 09:41:39.071 25985 TRACE ceilometer     return call(*args, **kwargs)
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/storage/mongo/utils.py", line 421, in __call__
2015-03-23 09:41:39.071 25985 TRACE ceilometer     return self.method(*args, **kwargs)
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1276, in index_information
2015-03-23 09:41:39.071 25985 TRACE ceilometer     cursor={}, read_preference=ReadPreference.PRIMARY)
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 345, in _command
2015-03-23 09:41:39.071 25985 TRACE ceilometer     msg, allowable_errors)
2015-03-23 09:41:39.071 25985 TRACE ceilometer   File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 182, in _check_command_response
2015-03-23 09:41:39.071 25985 TRACE ceilometer     raise OperationFailure(msg % errmsg, code, response)
2015-03-23 09:41:39.071 25985 TRACE ceilometer OperationFailure: command SON([('listIndexes', u'event'), ('cursor', {})]) on namespace ceilometer.$cmd failed: no collection

and monog.log
2015-03-23T09:40:08.343+0100 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:34072 #2 (1 connection now open)
2015-03-23T09:40:08.448+0100 I ACCESS   [conn2] Successfully authenticated as principal ceilometer on ceilometer
2015-03-23T09:40:08.451+0100 I INDEX    [conn2] allocating new ns file /var/lib/mongo/ceilometer.ns, filling with zeroes...
2015-03-23T09:40:08.551+0100 I STORAGE  [FileAllocator] allocating new datafile /var/lib/mongo/ceilometer.0, filling with zeroes...
2015-03-23T09:40:08.552+0100 I STORAGE  [FileAllocator] done allocating datafile /var/lib/mongo/ceilometer.0, size: 64MB,  took 0 secs
2015-03-23T09:40:08.560+0100 I INDEX    [conn2] build index on: ceilometer.resource properties: { v: 1, key: { user_id: 1, source: 1 }, name: "resource_idx", ns: "ceilometer.resource", background: false }
2015-03-23T09:40:08.560+0100 I INDEX    [conn2]          building index using bulk method
2015-03-23T09:40:08.561+0100 I INDEX    [conn2] build index done.  scanned 0 total records. 0 secs
2015-03-23T09:40:08.561+0100 I COMMAND  [conn2] command ceilometer.$cmd command: createIndexes { createIndexes: "resource", indexes: [ { background: false, name: "resource_idx", key: { user_id: 1, source: 1 } } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:113 locks:{} 109ms
2015-03-23T09:40:08.563+0100 I INDEX    [conn2] build index on: ceilometer.meter properties: { v: 1, key: { resource_id: 1, user_id: 1, counter_name: 1, timestamp: 1, source: 1 }, name: "meter_idx", ns: "ceilometer.meter", background: false }
2015-03-23T09:40:08.563+0100 I INDEX    [conn2]          building index using bulk method
2015-03-23T09:40:08.563+0100 I INDEX    [conn2] build index done.  scanned 0 total records. 0 secs
2015-03-23T09:40:08.565+0100 I INDEX    [conn2] build index on: ceilometer.resource properties: { v: 1, key: { project_id: 1, source: 1 }, name: "resource_project_idx", ns: "ceilometer.resource", background: true }
2015-03-23T09:40:08.565+0100 I INDEX    [conn2] build index done.  scanned 0 total records. 0 secs
2015-03-23T09:40:08.567+0100 I INDEX    [conn2] build index on: ceilometer.meter properties: { v: 1, key: { resource_id: 1, project_id: 1, counter_name: 1, timestamp: 1, source: 1 }, name: "meter_project_idx", ns: "ceilometer.meter", background: true }
2015-03-23T09:40:08.567+0100 I INDEX    [conn2] build index done.  scanned 0 total records. 0 secs
2015-03-23T09:40:08.568+0100 I INDEX    [conn2] build index on: ceilometer.resource properties: { v: 1, key: { last_sample_timestamp: -1 }, name: "last_sample_timestamp_idx", ns: "ceilometer.resource", sparse: true }
2015-03-23T09:40:08.568+0100 I INDEX    [conn2]          building index using bulk method
2015-03-23T09:40:08.568+0100 I INDEX    [conn2] build index done.  scanned 0 total records. 0 secs
2015-03-23T09:40:08.569+0100 I INDEX    [conn2] build index on: ceilometer.meter properties: { v: 1, key: { timestamp: -1 }, name: "timestamp_idx", ns: "ceilometer.meter" }
2015-03-23T09:40:08.569+0100 I INDEX    [conn2]          building index using bulk method
2015-03-23T09:40:08.569+0100 I INDEX    [conn2] build index done.  scanned 0 total records. 0 secs
2015-03-23T09:40:08.570+0100 I COMMAND  [conn2] CMD: drop ceilometer.user
2015-03-23T09:40:08.571+0100 I COMMAND  [conn2] CMD: drop ceilometer.project
2015-03-23T09:40:08.575+0100 I COMMAND  [conn2] CMD: drop ceilometer.user
2015-03-23T09:40:08.575+0100 I COMMAND  [conn2] CMD: drop ceilometer.project
2015-03-23T09:40:08.578+0100 I NETWORK  [conn2] end connection 127.0.0.1:34072 (0 connections now open)
2015-03-23T09:40:08.733+0100 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:34080 #3 (1 connection now open)
2015-03-23T09:40:08.839+0100 I ACCESS   [conn3] Successfully authenticated as principal ceilometer on ceilometer
2015-03-23T09:40:08.841+0100 I NETWORK  [conn3] end connection 127.0.0.1:34080 (0 connections now open)
2015-03-23T09:40:08.845+0100 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:34081 #4 (1 connection now open)
2015-03-23T09:40:08.950+0100 I ACCESS   [conn4] Successfully authenticated as principal ceilometer on ceilometer
2015-03-23T09:41:39.111+0100 I NETWORK  [conn4] end connection 127.0.0.1:34081 (0 connections now open)

Of course user ceilometer was created earlier using (as it is at http://docs.openstack.org/icehouse/install-guide/install/apt/content/ceilometer-install.html):
> db = db.getSiblingDB("ceilometer");
> db.createUser({user: "ceilometer", pwd: "ceilometer", roles: [ "readWrite", "dbAdmin" ]})

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-23: Fix proposed to ceilometer (master)

#8

Fix proposed to branch: master
Review: https://review.openstack.org/166860

Changed in ceilometer:
assignee:	nobody → gordon chung (chungg)
status:	Triaged → In Progress

Revision history for this message

gordon chung (chungg) wrote on 2015-03-23:

#9

ok, so i figured it out. it's not related so much to mongo3.x but just a gap in upgrade logic.

basically when creating indices, the mongo backend never checks to see if collection exists (it gets created when the first document is written)... i've added a check to create the collection beforehand.

Changed in ceilometer:
milestone:	none → kilo-rc1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-24: Fix merged to ceilometer (master)

#10

Reviewed: https://review.openstack.org/166860
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=5c0762534b6cceccff151001399e047ffa7f9498
Submitter: Jenkins
Branch: master

commit 5c0762534b6cceccff151001399e047ffa7f9498
Author: gordon chung <email address hidden>
Date: Mon Mar 23 10:35:59 2015 -0400

ensure collections created on upgrade

    upgrade does not check that collections exists. this can cause
    issues when running upgrade before any data is inserted. this patch
    creates collection on upgrade if it does not exist.

Change-Id: I1774d1c943050f1a87b386502392e18b0af8c840
Closes-Bug: #1434013

Changed in ceilometer:
status:	In Progress → Fix Committed

Eoghan Glynn (eglynn) on 2015-03-26

Changed in ceilometer:
importance:	Undecided → Medium

Thierry Carrez (ttx) on 2015-04-09

Changed in ceilometer:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2015-04-30

Changed in ceilometer:
milestone:	kilo-rc1 → 2015.1.0

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-24: Related fix proposed to ceilometer (master)

#11

Related fix proposed to branch: master
Review: https://review.openstack.org/185268

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-25: Change abandoned on ceilometer (master)

#12

Change abandoned by Julien Danjou (<email address hidden>) on branch: master
Review: https://review.openstack.org/185268
Reason: Duplicate of https://review.openstack.org/#/c/175418/

Ceilometer

MongoDB 3.X cannot be used as database backend

Bug Description

Other bug subscribers

Remote bug watches