Fix middleware to allow both slowlog and DataDog

[Paul Everitt]

(Note to Chris: I forgot to mention the timeit changes)

In December we did three things related to performance monitoring:

1) DataDog. Get perfmetrics wired up with logging to DataDog. Along the way, refined some of the perfmetrics to collect better data. (Work in progress.)

2) slowlog. Change slowlog to write files in var for requests that exceeded a certain threshold. Along the way, we also log ZODB cache metrics on each log entry.

3) timeit. I changed the timeit middleware to let us use it in production. It is now in the footer of every page, in a visible:hidden line above the copyright. Mouse over that space to see it. I show elapsed time in the stack on the server, but also which app server the request went to. I hoped to also show ZODB cache numbers.

Unfortunately it was left in a bad state. Shane's RelStorage perfmetric swolled the cache counter, meaning we could either use perfmetrics/DataDog or slowlog, but not both. Also, I have the slowlog var files accumulating indefinitely, plus we're not really looking at them.

Actions
=========

- Fix RelStorage perfmetrics to not kill the counter
- Get DataDog wired back up
- Analyze the slowlog
- (Optional) Arrange for the slowlog files to be part of the normal system logrotate

[Chris McDonough (chrism-plope)]

A fix was committed for this.

[Paul Everitt (paul-agendaless)]

Once we put it into production we can mark it as Fix Released.

--Paul

On Jan 29, 2014, at 3:05 PM, Chris McDonough <email address hidden> wrote:

> A fix was committed for this.
>
> ** Changed in: karl3
> Status: New => Fix Committed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1272422
>
> Title:
> Fix middleware to allow both slowlog and DataDog
>
> Status in KARL3:
> Fix Committed
>
> Bug description:
> (Note to Chris: I forgot to mention the timeit changes)
>
> In December we did three things related to performance monitoring:
>
> 1) DataDog. Get perfmetrics wired up with logging to DataDog. Along
> the way, refined some of the perfmetrics to collect better data. (Work
> in progress.)
>
> 2) slowlog. Change slowlog to write files in var for requests that
> exceeded a certain threshold. Along the way, we also log ZODB cache
> metrics on each log entry.
>
> 3) timeit. I changed the timeit middleware to let us use it in
> production. It is now in the footer of every page, in a visible:hidden
> line above the copyright. Mouse over that space to see it. I show
> elapsed time in the stack on the server, but also which app server the
> request went to. I hoped to also show ZODB cache numbers.
>
> Unfortunately it was left in a bad state. Shane's RelStorage
> perfmetric swolled the cache counter, meaning we could either use
> perfmetrics/DataDog or slowlog, but not both. Also, I have the slowlog
> var files accumulating indefinitely, plus we're not really looking at
> them.
>
> Actions
> =========
>
> - Fix RelStorage perfmetrics to not kill the counter
> - Get DataDog wired back up
> - Analyze the slowlog
> - (Optional) Arrange for the slowlog files to be part of the normal system logrotate
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/1272422/+subscriptions

[Paul Everitt (paul-agendaless)]

I think this was slated to go into production on a Saturday a number of weeks ago. Chris, did you wind up doing a production update?