Issues with Heat and Murano remote logs and multiline records

[Bogdan Dobrelya]

{"build_id": "2014-02-26_13-39-45", "mirantis": "yes", "build_number": "211", "nailgun_sha": "ea08cef3e06a72f47cfaa8cd8fe6d034e2cf722e", "ostf_sha": "8e6681b6d06c7cb20a84c1cc740d5f2492fb9d85", "fuelmain_sha": "baa8bb07393698f1186cb67bb65f1b93907c59bd", "astute_sha": "10cccc87f2ee35510e43c8fa19d2bf916ca1fced", "release": "4.1", "fuellib_sha": "0a2e5bdc01c1e3bb285acb7b39125101e950ac72"}

Reproduce:
* Deploy Centos simple, Neutron with GRE, Murano and Ceilometer, Cinder LVM for volumes, Ceph RBD for images;
nodes: 1 controller, 1 compute, 1 cinder, 2 ceph
* Prepare Murano OSTF tests https://github.com/stackforge/fuel-docs/blob/master/pages/user-guide/post-install-healthchecks.rst#preparing-murano-for-testing and run Murano OSTF
* Watch for local and remote logs for Heat and Murano

Issue1: Multiline/TRACE log messages in Heat are incorrectly parsed by remote logs node
Issue2: TRACE messages in Murano are incorrectly parsed by remote logs node

Expected:
- remote log files for Heat & Murano should be named like /var/log/remote/*/heat*.log & /var/log/remote/*/murano*.log
- multiline messages for local log records in /var/log/heat-all.log should look similar to /var/log/murano-all.log (see logs attached), e.g.:
-- Murano multiline is:
<131>Feb 27 11:40:04 node-5 murano-conductor 2014-02-27 11:40:04 ERROR muranoconductor.app [-] Unexpected stack state NOT_FOUND
2014-02-27 11:40:04.581 24447 TRACE muranoconductor.app Traceback (most recent call last):
2014-02-27 11:40:04.581 24447 TRACE muranoconductor.app File "/usr/lib/python2.6/site-packages/muranoconductor/app.py", line 129, in _task_received
-- Heat multiline should look:
<131>Feb 26 15:26:19 node-5 heat-heat.engine.resource ERROR: CREATE : WaitCondition "WaitCondition"
Traceback (most recent call last):
2014-02-26 15:26:19.581 24447 TRACE heatengine.app File "/usr/lib/python2.6/site-packages/heat/engine/resource.py", line 371, in _do_action

Actual:
- TRACE records for Heat & Murano are parsed to the wrong ***NO_MATCHED*** file
- Heat multiline is:
<131>Feb 26 15:26:19 node-5 heat-heat.engine.resource ERROR: CREATE : WaitCondition "WaitCondition"
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/heat/engine/resource.py", line 371, in _do_action

Solution: puppet modules for Murano and Heat should define correct debug and normal logging formatters.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/77171

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/77172

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/77172
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=44718c88f59a2078c72a7ebe26e23e4a3b4fbb1c
Submitter: Jenkins
Branch: master

commit 44718c88f59a2078c72a7ebe26e23e4a3b4fbb1c
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Feb 28 13:49:02 2014 +0200

    Fix heat, murano logging

    CI retest

    Closes-bug: #1285639

    Change-Id: I40f879a8ad1c4f87d7232f3bbc0439afb3fcab8f
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/77171
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2e442b1b600a4be146679ea689070b83dc69209e
Submitter: Jenkins
Branch: stable/4.1

commit 2e442b1b600a4be146679ea689070b83dc69209e
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Feb 28 13:49:02 2014 +0200

    Fix heat, murano logging

    4.1.1 (stable/4.1) targeted

    Closes-bug: #1285639

    Change-Id: I40f879a8ad1c4f87d7232f3bbc0439afb3fcab8f
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Bogdan Dobrelya (bogdando)]

Note: the fix is partial. Multiline traceback issue will be addressed in https://blueprints.launchpad.net/fuel/+spec/refactor-logging-puppet-openstack-services , part2

[Anastasia Palkina (apalkina)]

Verified on ISO #51
"build_id": "2014-03-26_13-22-07",
"mirantis": "yes",
"build_number": "51",
"nailgun_sha": "cb27e483815aa5ee09fcb69fe25e45b0e5fd6a7e",
"ostf_sha": "b54c48a50ea815d4e4aee1ed97c68601a2f458e8",
"fuelmain_sha": "27a04526eb1a0596b03dc9805435d2a3147d216e",
"astute_sha": "d7c6c4d00ffd6e2fa74da442f573e6f39049961e",
"release": "5.0",
"fuellib_sha": "d7fb2fe73788cae5241f321b11a90235d297b493"