ceilometer-agent-notification can't connect after rabbitmq restart

[Miroslav Anashkin]

{"build_id": "2014-11-20_21-01-00", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "28", "auth_required": true, "api": "1.0", "nailgun_sha": "7580f6341a726c2019f880ae23ff3f1c581fd850", "production": "docker", "fuelmain_sha": "eac9e2704424d1cb3f183c9f74567fd42a1fa6f3", "astute_sha": "51087c92a50be982071a074ff2bea01f1a5ddb76", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-11-20_21-01-00", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "28", "api": "1.0", "nailgun_sha": "7580f6341a726c2019f880ae23ff3f1c581fd850", "production": "docker", "fuelmain_sha": "eac9e2704424d1cb3f183c9f74567fd42a1fa6f3", "astute_sha": "51087c92a50be982071a074ff2bea01f1a5ddb76", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "b3d9f0e203f2f0faf3763e871a8dc31570777fed"}}}, "fuellib_sha": "b3d9f0e203f2f0faf3763e871a8dc31570777fed"}

This bug affected ceilometer-agent-notification in HA-mode 5.1.1.
Steps to reproduce:
1. Create and deploy env with HA mode
2. Restart rabbitmq
3. Notification agent may get "Socket closed", "Pipeline broken", "ECONNREFUSED" error then try to connect to rabbitmq.

Ceilometer doesn't use oslo.messaging in 5.1.1, so issue doesn't affect oslo.messaging.

Researching have been stopped when we found that socket error produced by GreenSocket.

Possible workaround:
Restart ceilometer-agent-notification after rabbitmq restarting.

[Miroslav Anashkin (manashkin)]

Upstream bug:
https://bugs.launchpad.net/ceilometer/+bug/1337715

[Ilya Pekelny (i159)]

I don't think that the problem with Ceilometer is directly related to oslo.messaging version 1.4.1. The Ceilomenter problem still exists even with 1.4.1 version, but didn't mentioned by customer as critical. The main issue with heartbeats was originally fixed in 5.1 by Roman Podolyaka, Stan Lagun and Serge Melikyan. The fixes were ported in all the MOS branches by me and Oleksii Zamiatin. This is easily seen by viewing late patches of the branches https://review.fuel-infra.org/#/admin/projects/openstack/oslo.messaging,branches.

Resume.

We (oslo.messaging team) need to pass Ceilometer bug to Ceilometer team and be informed of all the progress. Backporting shell not be helpful.

[Ilya Tyaptin (ityaptin)]

We are going to deploy 5.1.1 and try to research this bug.

[Ilya Tyaptin (ityaptin)]

Research confirm bug with handshake error in rabbit in next usecase:
In HA mode:
1. Start rabbitmq service: pcs resource enable p_rabbitmq-server
2. Start ceilometer-agent-notification: service ceilometer-agent-notification start
3. Restart rabbitmq: pcs resource disable p_rabbitmq-server; sleep 15; pcs resource enable p_rabbitmq-server
After this one instance of ceilometer-agent-notification may fails when connecting to rabbitmq. In this case rabbit log will contain errors with handshake timeout.

One of the probably fixes of it - increase notification_workers parameter in ceilometer.conf.

Next we'll make check of this fix.

[Vadim Rovachev (vrovachev)]

{"build_id": "2014-11-20_21-01-00", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "28", "auth_required": true, "api": "1.0", "nailgun_sha": "7580f6341a726c2019f880ae23ff3f1c581fd850", "production": "docker", "fuelmain_sha": "eac9e2704424d1cb3f183c9f74567fd42a1fa6f3", "astute_sha": "51087c92a50be982071a074ff2bea01f1a5ddb76", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-11-20_21-01-00", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "28", "api": "1.0", "nailgun_sha": "7580f6341a726c2019f880ae23ff3f1c581fd850", "production": "docker", "fuelmain_sha": "eac9e2704424d1cb3f183c9f74567fd42a1fa6f3", "astute_sha": "51087c92a50be982071a074ff2bea01f1a5ddb76", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "b3d9f0e203f2f0faf3763e871a8dc31570777fed"}}}, "fuellib_sha": "b3d9f0e203f2f0faf3763e871a8dc31570777fed"}

Bug reproduced on this iso.

[ruhe (ruhe)]

Not feasible in 5.1.1

Ilya T to update bug title and description

[Ilya Tyaptin (ityaptin)]

Today fixing of this bug in "In progress" status. We don't know source of this issue yet.
The only known workaround is restart service "ceilometer-agent-notification" after restarting rabbitmq.

[Dina Belova (dbelova)]

Known issue: ceilometer-agent-notification service may not connect to queue after RabbitMQ restart. This seems to be reproduced only in the HA mode, and we could reproduce one instance of ceilometer-agent-notification being failed.

Workaround: restart ceilometer-agent-notification after restarting RebbitMQ

[Ivan Berezovskiy (iberezovskiy)]

It's impossible to choose right value ttl for ceilometer data, because it depends on ceilometer usage. For Fuel 6.1 we chose one week for storing data, for Fuel 7.0 we want to provide ability to set ttl from Web UI . Default value will be the same (one week), but it could be possible for users to change it.

[Ivan Berezovskiy (iberezovskiy)]

Sorry, wrong comment

[Ilya Tyaptin (ityaptin)]

I'll going to continue work on this bug at next week.
Firstly I decide to check different rpc service executors in oslo.messaging library in MOS 6.x. Now in ceilometer messaging we use eventlet executor but also 'thread' and 'blocking' executors are available. We'll test performance and correct work for these executors and update bug status with results.
For MOS 5.1 relative aspects will be researched.

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2015.1.0b1, package release == fuel6.1.mira12.git.6d5d090.1f7aebf

Changeset: https://review.fuel-infra.org/2841
project: openstack/ceilometer
branch: master
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Change messaging listener executor to blocking
status: patchset-created

Files placed on repository:
openstack-ceilometer-alarm-2015.1.0b1-fuel6.1.mira12.git.6d5d090.1f7aebf.noarch.rpm
openstack-ceilometer-api-2015.1.0b1-fuel6.1.mira12.git.6d5d090.1f7aebf.noarch.rpm
openstack-ceilometer-central-2015.1.0b1-fuel6.1.mira12.git.6d5d090.1f7aebf.noarch.rpm
openstack-ceilometer-collector-2015.1.0b1-fuel6.1.mira12.git.6d5d090.1f7aebf.noarch.rpm
openstack-ceilometer-common-2015.1.0b1-fuel6.1.mira12.git.6d5d090.1f7aebf.noarch.rpm
openstack-ceilometer-compute-2015.1.0b1-fuel6.1.mira12.git.6d5d090.1f7aebf.noarch.rpm
openstack-ceilometer-ipmi-2015.1.0b1-fuel6.1.mira12.git.6d5d090.1f7aebf.noarch.rpm
openstack-ceilometer-notification-2015.1.0b1-fuel6.1.mira12.git.6d5d090.1f7aebf.noarch.rpm
python-ceilometer-2015.1.0b1-fuel6.1.mira12.git.6d5d090.1f7aebf.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-2841/centos

[OSCI Robot (oscirobot)]

package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2, package release == fuel6.1.mira22.git.bc66a4b.b6b17a4

Changeset: https://review.fuel-infra.org/2840
project: openstack/ceilometer
branch: openstack-ci/fuel-6.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Change messaging listener executor to blocking
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-2840/

[OSCI Robot (oscirobot)]

package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2, package release == fuel6.1~mira21+git.bc66a4b.b6b17a4

Changeset: https://review.fuel-infra.org/2840
project: openstack/ceilometer
branch: openstack-ci/fuel-6.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Change messaging listener executor to blocking
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /ubuntu-fuel-6.1-stable-2840/

[OSCI Robot (oscirobot)]

DEB package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2, package release == fuel6.1~mira21+git.bc66a4b.b6b17a4

Changeset: https://review.fuel-infra.org/2840
project: openstack/ceilometer
branch: openstack-ci/fuel-6.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Change messaging listener executor to blocking
status: patchset-created

Files placed on repository:
ceilometer-agent-central_2014.2-fuel6.1~mira21+git.bc66a4b.b6b17a4_all.deb
ceilometer-agent-compute_2014.2-fuel6.1~mira21+git.bc66a4b.b6b17a4_all.deb
ceilometer-agent-notification_2014.2-fuel6.1~mira21+git.bc66a4b.b6b17a4_all.deb
ceilometer-alarm-evaluator_2014.2-fuel6.1~mira21+git.bc66a4b.b6b17a4_all.deb
ceilometer-alarm-notifier_2014.2-fuel6.1~mira21+git.bc66a4b.b6b17a4_all.deb
ceilometer-api_2014.2-fuel6.1~mira21+git.bc66a4b.b6b17a4_all.deb
ceilometer-collector_2014.2-fuel6.1~mira21+git.bc66a4b.b6b17a4_all.deb
ceilometer-common_2014.2-fuel6.1~mira21+git.bc66a4b.b6b17a4_all.deb
python-ceilometer_2014.2-fuel6.1~mira21+git.bc66a4b.b6b17a4_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.1-stable-2840/ubuntu

[OSCI Robot (oscirobot)]

DEB package ceilometer has been built for project openstack/ceilometer
Package version == 2015.1.0b1, package release == fuel6.1~mira22+git.6d5d090.1f7aebf

Changeset: https://review.fuel-infra.org/2841
project: openstack/ceilometer
branch: master
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Change messaging listener executor to blocking
status: patchset-created

Files placed on repository:
ceilometer-agent-central_2015.1.0b1-fuel6.1~mira22+git.6d5d090.1f7aebf_all.deb
ceilometer-agent-compute_2015.1.0b1-fuel6.1~mira22+git.6d5d090.1f7aebf_all.deb
ceilometer-agent-ipmi_2015.1.0b1-fuel6.1~mira22+git.6d5d090.1f7aebf_all.deb
ceilometer-agent-notification_2015.1.0b1-fuel6.1~mira22+git.6d5d090.1f7aebf_all.deb
ceilometer-alarm-evaluator_2015.1.0b1-fuel6.1~mira22+git.6d5d090.1f7aebf_all.deb
ceilometer-alarm-notifier_2015.1.0b1-fuel6.1~mira22+git.6d5d090.1f7aebf_all.deb
ceilometer-api_2015.1.0b1-fuel6.1~mira22+git.6d5d090.1f7aebf_all.deb
ceilometer-collector_2015.1.0b1-fuel6.1~mira22+git.6d5d090.1f7aebf_all.deb
ceilometer-common_2015.1.0b1-fuel6.1~mira22+git.6d5d090.1f7aebf_all.deb
ceilometer-polling_2015.1.0b1-fuel6.1~mira22+git.6d5d090.1f7aebf_all.deb
python-ceilometer_2015.1.0b1-fuel6.1~mira22+git.6d5d090.1f7aebf_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-master-2841/ubuntu

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2, package release == fuel6.1.mira22.git.bc66a4b.b6b17a4

Changeset: https://review.fuel-infra.org/2840
project: openstack/ceilometer
branch: openstack-ci/fuel-6.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Change messaging listener executor to blocking
status: patchset-created

Files placed on repository:
openstack-ceilometer-alarm-2014.2-fuel6.1.mira22.git.bc66a4b.b6b17a4.noarch.rpm
openstack-ceilometer-api-2014.2-fuel6.1.mira22.git.bc66a4b.b6b17a4.noarch.rpm
openstack-ceilometer-central-2014.2-fuel6.1.mira22.git.bc66a4b.b6b17a4.noarch.rpm
openstack-ceilometer-collector-2014.2-fuel6.1.mira22.git.bc66a4b.b6b17a4.noarch.rpm
openstack-ceilometer-common-2014.2-fuel6.1.mira22.git.bc66a4b.b6b17a4.noarch.rpm
openstack-ceilometer-compute-2014.2-fuel6.1.mira22.git.bc66a4b.b6b17a4.noarch.rpm
openstack-ceilometer-notification-2014.2-fuel6.1.mira22.git.bc66a4b.b6b17a4.noarch.rpm
python-ceilometer-2014.2-fuel6.1.mira22.git.bc66a4b.b6b17a4.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.1-stable-2840/centos

[OSCI Robot (oscirobot)]

package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2, package release == fuel6.1.mira23

Changeset: https://review.fuel-infra.org/2840
project: openstack/ceilometer
branch: openstack-ci/fuel-6.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Change messaging listener executor to blocking
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /centos-fuel-6.1-stable/

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2, package release == fuel6.0.1.mira24

Changeset: https://review.fuel-infra.org/2945
project: openstack/ceilometer
branch: openstack-ci/fuel-6.0.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Change messaging listener executor to blocking
status: change-merged

Files placed on repository:
openstack-ceilometer-alarm-2014.2-fuel6.0.1.mira24.noarch.rpm
openstack-ceilometer-api-2014.2-fuel6.0.1.mira24.noarch.rpm
openstack-ceilometer-central-2014.2-fuel6.0.1.mira24.noarch.rpm
openstack-ceilometer-collector-2014.2-fuel6.0.1.mira24.noarch.rpm
openstack-ceilometer-common-2014.2-fuel6.0.1.mira24.noarch.rpm
openstack-ceilometer-compute-2014.2-fuel6.0.1.mira24.noarch.rpm
openstack-ceilometer-notification-2014.2-fuel6.0.1.mira24.noarch.rpm
python-ceilometer-2014.2-fuel6.0.1.mira24.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable/centos

[OSCI Robot (oscirobot)]

package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2, package release == fuel6.1~mira22

Changeset: https://review.fuel-infra.org/2840
project: openstack/ceilometer
branch: openstack-ci/fuel-6.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Change messaging listener executor to blocking
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /ubuntu-fuel-6.1-stable/

[OSCI Robot (oscirobot)]

DEB package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2, package release == fuel6.0.1~mira23

Changeset: https://review.fuel-infra.org/2945
project: openstack/ceilometer
branch: openstack-ci/fuel-6.0.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Change messaging listener executor to blocking
status: change-merged

Files placed on repository:
ceilometer-agent-central_2014.2-fuel6.0.1~mira23_all.deb
ceilometer-agent-compute_2014.2-fuel6.0.1~mira23_all.deb
ceilometer-agent-notification_2014.2-fuel6.0.1~mira23_all.deb
ceilometer-alarm-evaluator_2014.2-fuel6.0.1~mira23_all.deb
ceilometer-alarm-notifier_2014.2-fuel6.0.1~mira23_all.deb
ceilometer-api_2014.2-fuel6.0.1~mira23_all.deb
ceilometer-collector_2014.2-fuel6.0.1~mira23_all.deb
ceilometer-common_2014.2-fuel6.0.1~mira23_all.deb
python-ceilometer_2014.2-fuel6.0.1~mira23_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

[OSCI Robot (oscirobot)]

DEB package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2, package release == fuel6.0.1~mira24

Changeset: https://review.fuel-infra.org/2945
project: openstack/ceilometer
branch: openstack-ci/fuel-6.0.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Change messaging listener executor to blocking
status: change-merged

Files placed on repository:
ceilometer-agent-central_2014.2-fuel6.0.1~mira24_all.deb
ceilometer-agent-compute_2014.2-fuel6.0.1~mira24_all.deb
ceilometer-agent-notification_2014.2-fuel6.0.1~mira24_all.deb
ceilometer-alarm-evaluator_2014.2-fuel6.0.1~mira24_all.deb
ceilometer-alarm-notifier_2014.2-fuel6.0.1~mira24_all.deb
ceilometer-api_2014.2-fuel6.0.1~mira24_all.deb
ceilometer-collector_2014.2-fuel6.0.1~mira24_all.deb
ceilometer-common_2014.2-fuel6.0.1~mira24_all.deb
python-ceilometer_2014.2-fuel6.0.1~mira24_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

[OSCI Robot (oscirobot)]

DEB package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2, package release == fuel6.0.1~mira25

Changeset: https://review.fuel-infra.org/2945
project: openstack/ceilometer
branch: openstack-ci/fuel-6.0.1/2014.2
author: Ilya Tyaptin
committer: Ilya Tyaptin
subject: Change messaging listener executor to blocking
status: change-merged

Files placed on repository:
ceilometer-agent-central_2014.2-fuel6.0.1~mira25_all.deb
ceilometer-agent-compute_2014.2-fuel6.0.1~mira25_all.deb
ceilometer-agent-notification_2014.2-fuel6.0.1~mira25_all.deb
ceilometer-alarm-evaluator_2014.2-fuel6.0.1~mira25_all.deb
ceilometer-alarm-notifier_2014.2-fuel6.0.1~mira25_all.deb
ceilometer-api_2014.2-fuel6.0.1~mira25_all.deb
ceilometer-collector_2014.2-fuel6.0.1~mira25_all.deb
ceilometer-common_2014.2-fuel6.0.1~mira25_all.deb
python-ceilometer_2014.2-fuel6.0.1~mira25_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2015.1.0b2, package release == fuel6.1.mira13.git.fa9a8dd.4463029

Changeset: https://review.fuel-infra.org/2841
project: openstack/ceilometer
branch: master
author: Ilya Tyaptin
committer: Ivan Berezovskiy
subject: Change messaging listener executor to blocking
status: patchset-created

Files placed on repository:
openstack-ceilometer-alarm-2015.1.0b2-fuel6.1.mira13.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-api-2015.1.0b2-fuel6.1.mira13.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-central-2015.1.0b2-fuel6.1.mira13.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-collector-2015.1.0b2-fuel6.1.mira13.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-common-2015.1.0b2-fuel6.1.mira13.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-compute-2015.1.0b2-fuel6.1.mira13.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-ipmi-2015.1.0b2-fuel6.1.mira13.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-notification-2015.1.0b2-fuel6.1.mira13.git.fa9a8dd.4463029.noarch.rpm
python-ceilometer-2015.1.0b2-fuel6.1.mira13.git.fa9a8dd.4463029.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-2841/centos

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2015.1.0b2, package release == fuel6.1.mira14.git.fa9a8dd.4463029

Changeset: https://review.fuel-infra.org/2841
project: openstack/ceilometer
branch: master
author: Ilya Tyaptin
committer: Ivan Berezovskiy
subject: Change messaging listener executor to blocking
status: patchset-created

Files placed on repository:
openstack-ceilometer-alarm-2015.1.0b2-fuel6.1.mira14.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-api-2015.1.0b2-fuel6.1.mira14.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-central-2015.1.0b2-fuel6.1.mira14.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-collector-2015.1.0b2-fuel6.1.mira14.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-common-2015.1.0b2-fuel6.1.mira14.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-compute-2015.1.0b2-fuel6.1.mira14.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-ipmi-2015.1.0b2-fuel6.1.mira14.git.fa9a8dd.4463029.noarch.rpm
openstack-ceilometer-notification-2015.1.0b2-fuel6.1.mira14.git.fa9a8dd.4463029.noarch.rpm
python-ceilometer-2015.1.0b2-fuel6.1.mira14.git.fa9a8dd.4463029.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-2841/centos

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2015.1.0b2, package release == fuel6.1.mira16.git.f06996b.4463029

Changeset: https://review.fuel-infra.org/2841
project: openstack/ceilometer
branch: master
author: Ilya Tyaptin
committer: Ivan Berezovskiy
subject: Change messaging listener executor to blocking
status: patchset-created

Files placed on repository:
openstack-ceilometer-alarm-2015.1.0b2-fuel6.1.mira16.git.f06996b.4463029.noarch.rpm
openstack-ceilometer-api-2015.1.0b2-fuel6.1.mira16.git.f06996b.4463029.noarch.rpm
openstack-ceilometer-central-2015.1.0b2-fuel6.1.mira16.git.f06996b.4463029.noarch.rpm
openstack-ceilometer-collector-2015.1.0b2-fuel6.1.mira16.git.f06996b.4463029.noarch.rpm
openstack-ceilometer-common-2015.1.0b2-fuel6.1.mira16.git.f06996b.4463029.noarch.rpm
openstack-ceilometer-compute-2015.1.0b2-fuel6.1.mira16.git.f06996b.4463029.noarch.rpm
openstack-ceilometer-ipmi-2015.1.0b2-fuel6.1.mira16.git.f06996b.4463029.noarch.rpm
openstack-ceilometer-notification-2015.1.0b2-fuel6.1.mira16.git.f06996b.4463029.noarch.rpm
python-ceilometer-2015.1.0b2-fuel6.1.mira16.git.f06996b.4463029.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-2841/centos

[Vitaly Gusev (vgusev)]

Verified on MOS ISO:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0.1"
  api: "1.0"
  build_number: "127"
  build_id: "2015-03-10_09-14-43"
  astute_sha: "f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0"
  fuellib_sha: "dc9c193039f3e12c12a1b460a4840228b995cfb9"
  ostf_sha: "b38332e6741fc4b0ef40a9b4fe9d2804ee6437cc"
  nailgun_sha: "83f8c83b5e77973bd391dde3b97d22999ff02bd2"
  fuelmain_sha: "0868371a91e72b5a6720281f02055165b6c81d44"

Steps To Verify:
1. Login to controller node
2. Start rabbitmq service: pcs resource enable p_rabbitmq-server
3. Start ceilometer-agent-notification: service ceilometer-agent-notification start
4. Restart rabbitmq: pcs resource disable p_rabbitmq-server; sleep 15; pcs resource enable p_rabbitmq-server
5. Check that ceilometer-agent-notification is connected

[Vitaly Gusev (vgusev)]

Verified on ISO:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "260"
  build_id: "2015-03-31_22-54-44"
  nailgun_sha: "51ddbe0a2e379e82af531b1c6d45fd386d2f894e"
  python-fuelclient_sha: "9b0b6ab2f09ff75445550b7210104e21df5d8cd6"
  astute_sha: "7292fc2a673cb1c32a688a46fd4836ca0500a957"
  fuellib_sha: "27459a89cdac078ec4046a442a11d750c5eccc2e"
  ostf_sha: "1c7f799cb0e159737d1f910b13e4073b16d1eb8c"
  fuelmain_sha: "178812b1971a900c49a8afc1688afd7475a6ffbb"

[OSCI Robot (oscirobot)]

RPM package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2, package release == fuel6.0.mira19.git.72dda6d.b6b17a4

Changeset: https://review.fuel-infra.org/5253
project: openstack/ceilometer
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Alex Ermolov
committer: Alex Ermolov
subject: Change messaging listener executor to blocking
status: patchset-created

Files placed on repository:
openstack-ceilometer-alarm-2014.2-fuel6.0.mira19.git.72dda6d.b6b17a4.noarch.rpm
openstack-ceilometer-api-2014.2-fuel6.0.mira19.git.72dda6d.b6b17a4.noarch.rpm
openstack-ceilometer-central-2014.2-fuel6.0.mira19.git.72dda6d.b6b17a4.noarch.rpm
openstack-ceilometer-collector-2014.2-fuel6.0.mira19.git.72dda6d.b6b17a4.noarch.rpm
openstack-ceilometer-common-2014.2-fuel6.0.mira19.git.72dda6d.b6b17a4.noarch.rpm
openstack-ceilometer-compute-2014.2-fuel6.0.mira19.git.72dda6d.b6b17a4.noarch.rpm
openstack-ceilometer-notification-2014.2-fuel6.0.mira19.git.72dda6d.b6b17a4.noarch.rpm
python-ceilometer-2014.2-fuel6.0.mira19.git.72dda6d.b6b17a4.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable-5253/centos

[OSCI Robot (oscirobot)]

DEB package ceilometer has been built for project openstack/ceilometer
Package version == 2014.2, package release == fuel6.0~mira18+git.72dda6d.b6b17a4

Changeset: https://review.fuel-infra.org/5253
project: openstack/ceilometer
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Alex Ermolov
committer: Alex Ermolov
subject: Change messaging listener executor to blocking
status: patchset-created

Files placed on repository:
ceilometer-agent-central_2014.2-fuel6.0~mira18+git.72dda6d.b6b17a4_all.deb
ceilometer-agent-compute_2014.2-fuel6.0~mira18+git.72dda6d.b6b17a4_all.deb
ceilometer-agent-notification_2014.2-fuel6.0~mira18+git.72dda6d.b6b17a4_all.deb
ceilometer-alarm-evaluator_2014.2-fuel6.0~mira18+git.72dda6d.b6b17a4_all.deb
ceilometer-alarm-notifier_2014.2-fuel6.0~mira18+git.72dda6d.b6b17a4_all.deb
ceilometer-api_2014.2-fuel6.0~mira18+git.72dda6d.b6b17a4_all.deb
ceilometer-collector_2014.2-fuel6.0~mira18+git.72dda6d.b6b17a4_all.deb
ceilometer-common_2014.2-fuel6.0~mira18+git.72dda6d.b6b17a4_all.deb
python-ceilometer_2014.2-fuel6.0~mira18+git.72dda6d.b6b17a4_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable-5253/ubuntu

[Ivan Berezovskiy (iberezovskiy)]

This issue wasn't fixed for 5.1.2. It was mistake with changing status to "fix released".

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/5253
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: 72dda6dba9bc18b34746de91e17ba78d87ad2aa7
Author: Alex Ermolov <email address hidden>
Date: Thu Apr 2 17:30:41 2015

Change messaging listener executor to blocking

Now we use 'eventlet' notification_listener executor. it's
provide socket errors with rabbitmq. Blocking executor avoid
these issues.
Notification service performance does not change much for these
cases. In HA mode with 3 controllers "eventlet" 3
ceilometer-agent-notification processed ~150 samples per second,
"blocking" ~120 samples/sec.

Closes-bug: #1393505
Change-Id: I7f2bf518453b4f3706fca92ac0b09c70f041ed9c

[OSCI Robot (oscirobot)]

Reviewed: https://review.fuel-infra.org/5253
Committed: https://review.fuel-infra.org/gitweb?p=openstack/ceilometer.git;a=commitdiff;h=72dda6dba9bc18b34746de91e17ba78d87ad2aa7
Submitter: Vitaly Sedelnik
Branch: openstack-ci/fuel-6.0-updates/2014.2

commit 72dda6dba9bc18b34746de91e17ba78d87ad2aa7
Author: Alex Ermolov <email address hidden>

Change messaging listener executor to blocking

Now we use 'eventlet' notification_listener executor. it's
provide socket errors with rabbitmq. Blocking executor avoid
these issues.
Notification service performance does not change much for these
cases. In HA mode with 3 controllers "eventlet" 3
ceilometer-agent-notification processed ~150 samples per second,
"blocking" ~120 samples/sec.

Closes-bug: #1393505
Change-Id: I7f2bf518453b4f3706fca92ac0b09c70f041ed9c

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Ilya Tyaptin <email address hidden>
Review: https://review.fuel-infra.org/8061

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ilya Tyaptin <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8061

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ivan Berezovskiy <email address hidden> on branch: master
Review: https://review.fuel-infra.org/2841