Multi-app server logging solution

[Paul Everitt]

Now that we are multi-process, multi-VM for staging and soon production, we will need a better logging solution. Both for collection of log data and presentation into reports.

[Chris Rossi (chris-archimedeanco)]

Started in m102 but will need to finish in m103.

[Chris Rossi (chris-archimedeanco)]

our installation of Redis has the authentication layer[1] enabled. You have to issue the "AUTH" command followed by the password given in /etc/redis.conf before issuing any other commands.

The file /etc/redis.conf is readable for service users. You may easily find the password by grepping for "requirepass" in the file.

Hope this helps

[Chris Rossi (chris-archimedeanco)]

This is implemented it looks a little different from what we had before.

Unlike the old system which only tried to capture errors, this captures any logging statements. Logging statements expire automatically after 7 days (which is configurable) so we don't have to worry about the log growing infinitely large. Currently there aren't a lot of logging statements in Karl, so mostly this will wind up capturing errors, but this gives us the flexibility to start dumping performance and other data into here and have the possibility of writing tools to pull the data back out and analyze it. When an error is logged, the error triggers an alarm which puts the log into an alarm state which must be cleared by a human. This is very similar to how the old system worked, except that here you can clear the alarm state without erasing the log.

In the admin interface, there is now a 'System Log' link in the sidebar which takes you to the new view. This view currently kind of sucks but it is functional, at least. The view displays the 100 most recent entries all on a single page. Each entry has a 'Details' link that will show/hide more detailed info, like the stack trace for errors. Entries are divided into levels and categories. Levels are things like, INFO, WARNING, ERROR. Categories are things like karl, mailin, gsa_sync, etc... There are links at the top to narrow the display to a particular level and/or a particular category. The links display only levels/categories that the redislog has seen before. The redislog is not aware, a priori, which levels and categories it might see, so it just keeps track of the ones it has seen.

If an alarm is set for a particular category, that will be visible at the top of the screen along with a link to clear the alarm. The alarm should impact gocept's nagios monitoring in the same way as the old system.

[Chris Rossi (chris-archimedeanco)]

Oh, this is all available for test on karldev on the 'redislog' branch.

[Paul Everitt (paul-agendaless)]

I think it's cool that it captures more than just errors, for the use cases that you describe.

Thanks for the usability improvements. Is it even worth trying to engage Nagios in a better way, so that we can tell it whether something is "CRITICAL", "MAJOR", etc.?

--Paul

On May 29, 2012, at 11:01 AM, Chris Rossi wrote:

> This is implemented it looks a little different from what we had before.
>
> Unlike the old system which only tried to capture errors, this captures
> any logging statements. Logging statements expire automatically after 7
> days (which is configurable) so we don't have to worry about the log
> growing infinitely large. Currently there aren't a lot of logging
> statements in Karl, so mostly this will wind up capturing errors, but
> this gives us the flexibility to start dumping performance and other
> data into here and have the possibility of writing tools to pull the
> data back out and analyze it. When an error is logged, the error
> triggers an alarm which puts the log into an alarm state which must be
> cleared by a human. This is very similar to how the old system worked,
> except that here you can clear the alarm state without erasing the log.
>
> In the admin interface, there is now a 'System Log' link in the sidebar
> which takes you to the new view. This view currently kind of sucks but
> it is functional, at least. The view displays the 100 most recent
> entries all on a single page. Each entry has a 'Details' link that will
> show/hide more detailed info, like the stack trace for errors. Entries
> are divided into levels and categories. Levels are things like, INFO,
> WARNING, ERROR. Categories are things like karl, mailin, gsa_sync,
> etc... There are links at the top to narrow the display to a particular
> level and/or a particular category. The links display only
> levels/categories that the redislog has seen before. The redislog is
> not aware, a priori, which levels and categories it might see, so it
> just keeps track of the ones it has seen.
>
> If an alarm is set for a particular category, that will be visible at
> the top of the screen along with a link to clear the alarm. The alarm
> should impact gocept's nagios monitoring in the same way as the old
> system.
>
>
> ** Changed in: karl3
> Status: In Progress => Fix Committed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/960107
>
> Title:
> Multi-app server logging solution
>
> Status in KARL3:
> Fix Committed
>
> Bug description:
> Now that we are multi-process, multi-VM for staging and soon
> production, we will need a better logging solution. Both for
> collection of log data and presentation into reports.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/960107/+subscriptions

[Paul Everitt (paul-agendaless)]

I just poked around on it. Obviously, since I can't trigger system errors, I can't really do too much testing. What's a good way for me to contribute to testing?

--Paul

On May 29, 2012, at 11:02 AM, Chris Rossi wrote:

> Oh, this is all available for test on karldev on the 'redislog' branch.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/960107
>
> Title:
> Multi-app server logging solution
>
> Status in KARL3:
> Fix Committed
>
> Bug description:
> Now that we are multi-process, multi-VM for staging and soon
> production, we will need a better logging solution. Both for
> collection of log data and presentation into reports.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/960107/+subscriptions

[Chris Rossi (chris-archimedeanco)]

/raise_error.html

On Tue, May 29, 2012 at 4:49 PM, Paul Everitt <email address hidden> wrote:
> I just poked around on it. Obviously, since I can't trigger system errors, I can't really do too much testing. What's a good way for me to contribute to testing?
>
> --Paul
>
> On May 29, 2012, at 11:02 AM, Chris Rossi wrote:
>
>> Oh, this is all available for test on karldev on the 'redislog' branch.
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/960107
>>
>> Title:
>>  Multi-app server logging solution
>>
>> Status in KARL3:
>>  Fix Committed
>>
>> Bug description:
>>  Now that we are multi-process, multi-VM for staging and soon
>>  production, we will need a better logging solution. Both for
>>  collection of log data and presentation into reports.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/karl3/+bug/960107/+subscriptions
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/960107
>
> Title:
>  Multi-app server logging solution
>
> Status in KARL3:
>  Fix Committed
>
> Bug description:
>  Now that we are multi-process, multi-VM for staging and soon
>  production, we will need a better logging solution. Both for
>  collection of log data and presentation into reports.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/960107/+subscriptions

[JimPGlenn (jpglenn09)]

Yep you are right it is in production on last release