Fuel logging works incorrectly for Savanna

[Vadim Rovachev]

{"build_id": "2014-02-26_17-35-21", "mirantis": "yes", "build_number": "214", "nailgun_sha": "ea08cef3e06a72f47cfaa8cd8fe6d034e2cf722e", "ostf_sha": "8e6681b6d06c7cb20a84c1cc740d5f2492fb9d85", "fuelmain_sha": "6e6f6b31b8880b7600a1af5cc94d48ac01d60a07", "astute_sha": "10cccc87f2ee35510e43c8fa19d2bf916ca1fced", "release": "4.1", "fuellib_sha": "0a2e5bdc01c1e3bb285acb7b39125101e950ac72"}

Steps to reproduce.
1. Deploy openstack via KVM, Nova Network.
2. Create any Savanna object.
3. See logs in controller, master fuel.
Expected result:
Savanna logs have
Actual result:
Savanna logs did not have

------------------------------------------------------------------------------------------

In controller in files /var/log/syslog, /var/log/sasvanna-all.log is no logs

But

If remove all rsyslog rules in /etc/rsuslog.d/ directory except savanna rules
And in file /etc/init.d/savanna-api in line for start savanna remove all settings for savanna logging
And in file /etc/savanna/savanna.conf comment out line with 'log_config' parameter
and restart rsyslog and savanna-api
Then in files /var/log/syslog, /var/log/sasvanna-all.log logs entries start to appear.

[Vladimir Kuklin (vkuklin)]

please add diagnostic snapshot here.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/77548

[Andrew Lazarev (alazarev)]

Note, the issue occurs only if 'OpenStack debug logging' is not checked.

Workaround for the issue:

Perform the following changes in '/etc/savanna/savanna.conf'.
Delete (or comment) the following lines from [DEFAULT] section:

use_syslog=True
use_stderr=False
syslog_log_facility=LOG_LOCAL0
log_config=/etc/savanna/logging.conf

Adde the following line to [DEFAULT] section:

log_dir=/var/log/savanna

[Bogdan Dobrelya (bogdando)]

Note: the w/a would provide local logs only, remote logs would be absent.
Instead, the patch suggested above fixes the issue and makes local copy of logs to be equal to the remote one.

Thus, some useful informational log records could be skipped though, because of the known log_config issue (see https://bugs.launchpad.net/fuel/+bug/1284867 for details)

[Andrew Lazarev (alazarev)]

@Bogdan I understand that w/a enables local logs only. Are any solution for remote logs?

[Bogdan Dobrelya (bogdando)]

Yes, it is provided here https://bugs.launchpad.net/fuel/+bug/1284867 in the comments.

[Andrew Lazarev (alazarev)]

"redeploy or reapply puppet using debug= true in astute.yaml or patch OS configuration files appropriately" doesn't look like suitable workaround:
1. redeploy is required
2. it will write debug logs

Am I missing something?

[Bogdan Dobrelya (bogdando)]

Another option is for savanna-api Openstack service, use log_config_append instead of log_config and correct
log_format=%(asctime)s.%(msecs)06d+00:00 savanna-api %(levelname)s: %(name)s: %(message)s
and log_date_format=%Y-%m-%dT%H:%M:%S
For example, patch for nova config could be http://pastebin.com/akN4J2EN. That would make all nova subservices to log locally as is, and remotely to /var/log/remote/node-baz.qux/nova.log

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/80233

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/77548
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=be71153e31ec058905ae667a37954d30bc72e49c
Submitter: Jenkins
Branch: master

commit be71153e31ec058905ae667a37954d30bc72e49c
Author: iberezovskiy <email address hidden>
Date: Mon Mar 3 14:07:12 2014 +0400

    Fix Savanna manifests to configure logging

    Change-Id: Iffae82d347a4bd6bb0170bfae6b3d1501bc5f344
    Closes-bug: #1285766

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/80233
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a9c07424c2f6abf6e2c59499da25a0d71ca3517e
Submitter: Jenkins
Branch: stable/4.1

commit a9c07424c2f6abf6e2c59499da25a0d71ca3517e
Author: iberezovskiy <email address hidden>
Date: Mon Mar 3 14:07:12 2014 +0400

    Fix Savanna manifests to configure logging

    4.1.1 (stable/4.1) targeted

    Closes-bug: #1285766
    Closes-bug: #1288475

    Change-Id: I96e78220feabca9ccfa9ea3c1ece079645449f9b
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Anastasia Palkina (apalkina)]

Verified on ISO #34
"build_id": "2014-03-19_01-01-16",
"mirantis": "yes",
"build_number": "34",
"nailgun_sha": "ffd36bc479a6d5e1ce285982259ddefa9dce2602",
"ostf_sha": "31c8316d4d9762f280da9e360d3d42e88464aac1",
"fuelmain_sha": "d494e5cf2b8399899f00b5079eec1c5f70c0b66f",
"astute_sha": "f52db1262401cfb2beaf41e7d6eeaeb456d5fe95",
"release": "5.0",
"fuellib_sha": "da114aa7b484ea62a34843c54353a012743b9d96"

[Meg McRoberts (dreidellhasa)]

Marked as "Resolved" in 5.0 Release Notes

[Meg McRoberts (dreidellhasa)]

Marked as "Resolved" in 4.1.1 Release Notes