Empty response for logs in UI: Source and Min level

[Anastasia Palkina]

"build_id": "2014-06-11_00-30-36",
"mirantis": "yes",
"build_number": "250",
"ostf_sha": "67b61ed3788297fa5d985afec32498d8c0f812db",
"nailgun_sha": "f851199db57d9561b51c429956c056b463ad3f0b",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "e59cb325d99fba49085ed5b0b6cfb0d9a83d2fc8",
"astute_sha": "17b1afa5f0dc8f4fca5ed4eb03ec566fbfb5ed19",
"release": "5.1",
"fuellib_sha": "d9b8daa61e131e3c346e93d3870620897a71c593"

1. Create new environment (Ubuntu, HA mode)
2. Choose VLAN segmentation
3. Choose both ceph
4. Choose rados
5. Add 3 controller+ceph, 2 compute+ceph
6. Start deployment. It was successful
7. Click button "View logs" on controllers
8. Source and Min level in logs are empty (see screen). Response is empty

[Bogdan Dobrelya (bogdando)]

The issue is caused by the commit to fuel-library:

https://github.com/stackforge/fuel-library/commit/a3ec3ce2b85c4c13fb310a36b133d6b8235546e5

When 'rsyslog' container starts it applies '/etc/puppet/modules/rsyslog/manifests/server.pp' manifest and necessary configuration file '30-server-remote-log.conf' isn't created anymore, so all logs from remote nodes are being writed to /var/log/messages inside docker container. As I understand currently this file should be created in 'openstack::logging' puppet class, but it isn't applied for 'rsyslog' container.

As temporary workaround to fix logging you can try to follow next steps before deploying new environment(on master):

        # dockerctl shell rsyslog
        # curl -s http://paste.openstack.org/raw/84168/ > /etc/puppet/modules/rsyslog/manifests/server.pp
        # cp /etc/puppet/modules/{openstack,rsyslog}/templates/30-server-remote-log.conf.erb
 # exit
 # dockerctl stop rsyslog && supervisorctl restart docker-rsyslog

[Artem Panchenko (apanchenko-8)]

Also, that workaround doesn't fix displaying of remote logs on UI, because timestamp format in logs was also changed from

 '%Y-%m-%dT%H:%M:%s%:z'
to
 '%Y-%m-%d %H:%M:%S'

and nailgun handler can't parse them. It expects the following logs format:

    regexp: '^(?P<date>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})(?P<secfrac>\.\d{1,})?(?P<timezone>(Z|[+-]\d{2}:\d{2}))?\s(?P<level>[a-z]{3,7}):\s(?P<text>.*)$'
    date_format: '%Y-%m-%dT%H:%M:%S'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/100273

[Bogdan Dobrelya (bogdando)]

Artem, the date format is ensured by 30-server-remote-log.conf template as well, so the main issue is the absense of the template.
The fix suggested should resolve it.

[Bogdan Dobrelya (bogdando)]

Patch was tested manually on {"build_id": "2014-06-16_00-31-15", "mirantis": "yes", "build_number": "255", "ostf_sha": "67b61ed3788297fa5d985afec32498d8c0f812db", "nailgun_sha": "984aa7a86487f1488c2f83c052904abd9f589b7f", "production": "docker", "api": "1.0", "fuelmain_sha": "6f355160366475d52050d7898a1080a95ecb9cbf", "astute_sha": "17b1afa5f0dc8f4fca5ed4eb03ec566fbfb5ed19", "release": "5.1", "fuellib_sha": "99d74172887ab81d38132655d6e5d180e8726437"}

issue resolved

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/100273
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=46c06ec3bd8350e67cbbbc8dea9d2aee7d74a75a
Submitter: Jenkins
Branch: master

commit 46c06ec3bd8350e67cbbbc8dea9d2aee7d74a75a
Author: Bogdan Dobrelya <email address hidden>
Date: Mon Jun 16 18:02:06 2014 +0300

    Add missed 30-remote-log.conf for docker container

    Add missed 30-remote.conf template for rsyslog docker container
    Fix existing dependency issue (rsyslog::server wipes templates dir)
    Fix default access and notification for rsyslog.d file resources
    Purge confusing nailgun templates for syslog (not used anymore)

    Closes-bug: #1328877

    Change-Id: If6f256cfeeba5bee486dd45c34cbc56c9e721730
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Artem Panchenko (apanchenko-8)]

Bogdan, there is a condition in 30-server-remote-log.conf file template and if show_timezone variable isn't equals true, rsyslogd writes logs with "short" timestamp which isn't recognized by nailgun. It is possible to workaround this by adding

$show_timezone = true

in /etc/puppet/modules/nailgun/examples/rsyslog-only.pp, but probably this can be fixed in another way (nailgun part of someting else).

[Bogdan Dobrelya (bogdando)]

One more issue discovered
the patch provides timestamps as
# Use traditional timestamps (date-rfc3164, Dec 5 02:21:13)
$Template RemoteLog, "%timegenerated:1:15:date-rfc3164% %syslogseverity-text%: %msg%\n"

but we actualy wanted
# Use high precision timestamps (date-rfc3339, 2010-12-05T02:21:41.889482+01:00)
$Template RemoteLog, "%timegenerated:1:32:date-rfc3339% %syslogseverity-text%: %msg%\n"

cuz UI relies on this in its regex patterns for logs matching.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/100456

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/100456
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a451e123113b436caa53555b4bd97875d647bca5
Submitter: Jenkins
Branch: master

commit a451e123113b436caa53555b4bd97875d647bca5
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Jun 17 11:58:46 2014 +0300

    Fix high precision timestamps for UI logs matching

    Fuel UI relies on the fixed timestamp pattern, so we have to
    enable it back in syslog logging config.

    Closes-bug: #1328877

    Change-Id: Ib12779470f6bf91d26fbfa271c2af14e61535c54
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Anastasia Palkina (apalkina)]

Verified on ISO #274
"build_id": "2014-06-27_00-31-14",
"mirantis": "yes",
"build_number": "274",
"ostf_sha": "a4978638de3951dbc229276608a839a19ece2b70",
"nailgun_sha": "5f2944a8d5077a1c96acb076ba9194f670b818e8",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "bf8660309601cee2f8f3e1bb881d272e638dcffa",
"astute_sha": "694b5a55695e01e1c42185bfac9cc7a641a9bd48",
"release": "5.1",
"fuellib_sha": "acc99fcd0ba9eeef0a504dc26507eb91ce757220"