Syslog truncates traceback in logs

[Ilya Shakhat]

Detailed bug description:
  Python traceback is truncated in *-all log file and at the master node. E.g.: /var/log/heat/heat-api.log contains full traceback, but /var/log/heat-all.log has only the beginning. An example is attached below.

Steps to reproduce:
  Check log file at controller and master node. No specific actions required.

Expected results:
  The traceback is preserved (as it was before)

Actual result:
  The traceback is truncated

Reproducibility:
  100%

Workaround:
  no

Impact:
  debuggability & support

Description of the environment:
# fuel2 fuel-version
api: '1'
auth_required: true
feature_groups: []
openstack_version: mitaka-9.0
release: '9.0'

[Ilya Shakhat (shakhat)]

/var/log/heat/heat-api.log
~~~~~~~~~~~~~~~~~~~~~~~~~~

2016-04-21 13:24:53.066 13361 INFO eventlet.wsgi.server [-] 192.168.0.2 - - [21/Apr/2016 13:24:53] "OPTIONS / HTTP/1.0" 200 134 0.002654
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi [req-1f32be4a-f249-49f4-9f7a-f987d6056f28 - admin - default default] Unexpected error occurred serving API: Timed out waiting for a reply to message ID
 0c244c8d150648a1b928817031c15f8d
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi Traceback (most recent call last):
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi File "/usr/lib/python2.7/dist-packages/heat/common/wsgi.py", line 842, in __call__
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi request, **action_args)
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi File "/usr/lib/python2.7/dist-packages/heat/common/wsgi.py", line 916, in dispatch
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi return method(*args, **kwargs)
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi File "/usr/lib/python2.7/dist-packages/heat/api/openstack/v1/util.py", line 38, in handle_stack_method
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi return handler(controller, req, **kwargs)
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi File "/usr/lib/python2.7/dist-packages/heat/api/openstack/v1/stacks.py", line 410, in lookup
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi stack_name)
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi File "/usr/lib/python2.7/dist-packages/heat/rpc/client.py", line 104, in identify_stack
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi stack_name=stack_name))
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi File "/usr/lib/python2.7/dist-packages/heat/rpc/client.py", line 69, in call
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi return client.call(ctxt, method, **kwargs)
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 413, in call
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi return self.prepare().call(ctxt, method, **kwargs)
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi retry=self.retry)
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi timeout=timeout, retry=retry)
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 512, in send
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi retry=retry)
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 501, in _send
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi result = self._waiter.wait(msg_id, timeout)
2016-04-21 13:24:58.957 13361 ERROR heat.common.wsgi File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py"...

[Dmitry Klenov (dklenov)]

Having issue in place will impact troubleshooting. Raising to high.

[Aleksey Zvyagintsev (azvyagintsev)]

+1 to issue, looks like similar behavior with fuel-agent (bootstrap provison logs):

On master:
head -n 10 /var/log/remote/node-1.test.domain.local/bootstrap/fuel-agent.log
http://paste.openstack.org/show/495101/

On node:
head -n 10 /var/log/fuel-agent.log
http://paste.openstack.org/show/495100/

(long sting was cutter)

[Dmitry Pyzhov (dpyzhov)]

This is an issue on syslog side. Moving to the library team

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/314186

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/314186
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=8f10ba3e0bf83f249b44b79a4f9f1b9ae416e866
Submitter: Jenkins
Branch: master

commit 8f10ba3e0bf83f249b44b79a4f9f1b9ae416e866
Author: Maksim Malchuk <email address hidden>
Date: Mon May 9 18:34:28 2016 +0300

    Do not omit syncing the '*-all.log' syslog files

    While parsing syslogtag from many sources we always should sync.

    Change-Id: If9859950fb4c8704da0df5a6c715eaeb08ad3b51
    Closes-Bug: #1573047
    Signed-off-by: Maksim Malchuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/314961

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/314961
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4cbf6b95a4023b55dd88e3761c1028af2b754ade
Submitter: Jenkins
Branch: stable/mitaka

commit 4cbf6b95a4023b55dd88e3761c1028af2b754ade
Author: Maksim Malchuk <email address hidden>
Date: Mon May 9 18:34:28 2016 +0300

    Do not omit syncing the '*-all.log' syslog files

    While parsing syslogtag from many sources we always should sync.

    Change-Id: If9859950fb4c8704da0df5a6c715eaeb08ad3b51
    Closes-Bug: #1573047
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit 8f10ba3e0bf83f249b44b79a4f9f1b9ae416e866)

[Ekaterina Shutova (eshutova)]

Log on controller /var/log/heat-all.log contains full traceback from deploy moment till now.
https://paste.mirantis.net/show/2345/

Verified on:
cat /etc/fuel_build_id:
 443
cat /etc/fuel_build_number:
 443
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0