ceilometer-common updates are breaking swift-proxy-server

[Serhii Ovsianikov]

Steps to reproduce
Install Fuel-6.1, 1-controller, 1-mongo, 2-compute+cinder, ceilometer

On controller node
root@node-1:~# ls -al /var/log/ceilometer | grep swift
-rw-r--r-- 1 swift swift 463655 Jan 24 19:35 swift-proxy-server.log

After updating ceilometer packages swift-proxy-server doesn't work because it changes the owner and group of the swift-proxy-server.log file.

Install the following packages and check owner of the swift-proxy-server.log file:

root@node-1:~# dpkg -i python-ceilometer_2014.2.2-1-u14.04+mos10+git.d342290.96c6c5d_all.deb
root@node-1:~# dpkg -i python-ceilometerclient_1.0.12-1-u14.04+mos9+git.131a5ee.15bb1e3_all.deb
root@node-1:~# dpkg -i ceilometer-common_2014.2.2-1-u14.04+mos10+git.d342290.96c6c5d_all.deb

root@node-1:~# ls -al /var/log/ceilometer | grep swift
-rw-r--r-- 1 ceilometer ceilometer 492535 Jan 24 19:39 swift-proxy-server.log

[Roman Podoliaka (rpodolyaka)]

Can't reproduce on 8.x. Alyona, please give it a try on 7.0

[Alyona Kiseleva (akiselyova)]

On 7.1 bug is reproduced. But why it's swift's problem?
Installation of package ceilometer-common (I check, that exactly this package broke all) chown folder /var/log/ceilometer. Swift cannot access its files and don't work, isn't it a deployment problem?

[Serhii Ovsianikov (sovsianikov)]

>But why it's swift's problem?
IMHO Why swift-proxy-server.log is located in ceilometer directory?

[Alyona Kiseleva (akiselyova)]

>IMHO Why swift-proxy-server.log is located in ceilometer directory?
Because ceilometer middleware for swift is used, I think. I know too little about ceilometer working process to explain this necessity.
In ceilometer installing manual there is note:
"Please make sure that ceilometer’s logging directory (if it’s configured) is read and write accessible for the user swift is started by."
http://docs.openstack.org/developer/ceilometer/install/manual.html

[Alexey Stupnikov (astupnikov)]

I can't reproduce this issue on environment deployed using Fuel with latest MU. I will try to manually update existing clean MOS 6.1 environment.

[Alexey Stupnikov (astupnikov)]

OK, this issue came up when packages with specific bugfixes were applied. For more information check the links below:
https://bugs.launchpad.net/mos/+bug/1500426
http://obs-1.mirantis.com:82/trusty-fuel-6.1-stable-updates-13516/ubuntu/

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Alexey Stupnikov <email address hidden>
Review: https://review.fuel-infra.org/20508

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Alexey Stupnikov <email address hidden>
Review: https://review.fuel-infra.org/20509

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20509
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 53f9b44e964ac5bb68f09a31cb22f7f2530885a6
Author: Alexey Stupnikov <email address hidden>
Date: Wed May 11 17:24:47 2016

Change back the owner of swift-proxy log file

Ceilometer's postinstall script recursively changes the owner of
/var/log/ceilometer dir. Swift-proxy uses that dir to store its log
file swift-proxy-server.log, which will not be available after
update. I have added the if statement to postinst scripts change
back the file's owner after update.

Closes-bug: #1537657
Change-Id: I2d30fdc9726a4866a90e59eabf841791f3bd77a0

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20508
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 5f5cc1a792524c4f5268655cbaa45127ea54bf7b
Author: Alexey Stupnikov <email address hidden>
Date: Wed May 11 17:26:23 2016

Change back the owner of swift-proxy log file

Ceilometer's postinstall script recursively changes the owner of
/var/log/ceilometer dir. Swift-proxy uses that dir to store its log
file swift-proxy-server.log, which will not be available after
update. I have added the if statement to postinst scripts change
back the file's owner after update.

Closes-bug: #1537657
Change-Id: I2d30fdc9726a4866a90e59eabf841791f3bd77a0

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Alexey Stupnikov <email address hidden>
Review: https://review.fuel-infra.org/20581

[Alexey Stupnikov (astupnikov)]

It is possible not to push empty commit to openstack-ci/fuel-7.0/2015.1.0 branch, since I will push another commits there during 7.0-mu4 release cycle. But I will do it anyway for consistency purposes.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Alexey Stupnikov <email address hidden>
Review: https://review.fuel-infra.org/20582

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20581
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: f7f3e0f1ffacf20d8f250bde9f049b892ad654cb
Author: Alexey Stupnikov <email address hidden>
Date: Fri May 13 10:18:56 2016

Rebuild ceilometer-common package in MOS 6.1 MU

There was a bug in ceilometer's build manifests. As a result,
current ceilometer-common package is breaking swift-proxy-server
process (see bug #1537657). We need to rebuild those package and
put correct version of ceilometer-common package in our
mos-updates repo.

Spec patch: https://review.fuel-infra.org/#/c/20508/

Related-bug: #1537657
Change-Id: I21c661dce50c8f267e3e44340a26406b179e123d

[Alexey Stupnikov (astupnikov)]

There is a problem with py27 tests in 7.0 ceilometer's branch. I have to fix those tests first, opened bug #1581507

[Alexey Stupnikov (astupnikov)]

New ceilometer-common package will be put into repo after https://review.fuel-infra.org/#/c/11049/ will be merged. Since it is impossible to merge dedicated empty patch due to a broken CI, we will not merge it after (it is useless job).

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Alexey Stupnikov <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/20582
Reason: New packages will be built after merge of https://review.fuel-infra.org/#/c/11049/

[Alexander Gromov (agromov)]

Verified on MOS 6.1 + mu6 updates.

Without updates after updating ceilometer packages:
root@node-1:~# ls -al /var/log/ceilometer | grep swift
-rw-r--r-- 1 ceilometer ceilometer 384988 May 13 14:52 swift-proxy-server.log

With updates updating ceilometer packages:
root@node-1:~# ls -al /var/log/ceilometer | grep swift
-rw-r--r-- 1 swift swift 328711 May 17 14:19 swift-proxy-server.log

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11049
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 2818186eb82d74e8474d8fbc0a7fe6df4121d2aa
Author: Ilya Tyaptin <email address hidden>
Date: Tue May 17 06:45:11 2016

[MongoDB] Use a aggregate pipeline in statistics

Currently, we use a native mapreduce jobs for highload requests
in MongoDb backend, but this approach have insufficient performance.
For example, at test environment mapreduce job processes a 10k samples/sec.
It's not good result for big collections.
Aggregation framework has a better performance and enough syntax
for our needs. At test environment it shows a improving "statistics" request
performance up to 10 times.

Closes-Bug: #1506877
Closes-bug: #1581507
Related-bug: #1537657
Implements: blueprint mongodb-aggregation-pipeline
Change-Id: I60040b4472529d460f03cc4d1b55d883ea339124
cherry-picked from c41a8856b98a435bc9af4ea80d42f5d8139b7fe9

[Alexey Stupnikov (astupnikov)]

New ceilometer-common package is build for MOS7 after https://review.fuel-infra.org/#/c/11049/ commit.

[Alexander Gromov (agromov)]

Verified on MOS 7.0 + mu4 updates.

Without updates after updating ceilometer packages:
root@node-1:~# ls -al /var/log/ceilometer | grep swift
-rw-rw-r-- 1 ceilometer ceilometer 406139 May 27 08:32 swift-proxy-server.log

With updates after updating ceilometer packages:
root@node-1:~# ls -al /var/log/ceilometer | grep swift
-rw-rw-r-- 1 swift swift 479938 May 27 09:09 swift-proxy-server.log