Investigate splitting search ghetto stats in Datadog

[Tres Seaver]

As with connection stats (#1302111), the statistics logged from each appserver to Datadog mingle the 'search' and the 'main' instances. Figure out how to split those out.

[Tres Seaver (tseaver)]

Experimenting with using the Supervisord 'environment' setting to influence the appserver's logging.

[Tres Seaver (tseaver)]

Running supervisord to show environment variables being used::

    $ bin/supervisord -c supervisord.conf
    $ bin/supervisorctl -c supervisord.conf
    test1 RUNNING pid 20356, uptime 0:00:04
    test2 RUNNING pid 20357, uptime 0:00:04
    supervisor> tail test1
    ENVTEST: test1
    ENVTEST: test1

    supervisor> tail test2
    ENVTEST: test2
    ENVTEST: test2

    supervisor> shutdown
    Really shut the remote supervisord process down y/N? y
    Shut down
    supervisor>

[Tres Seaver (tseaver)]

Rather than hack in stuff via the environment, we should be able to
define a new '[pipeline:search]' and '[app:karlserve-search]' sections
in the 'etc/karlserve.ini' file. Those sections would be based on
'[pipeline:main]' and '[app:karlserve], overriding the configuration
settings we need. E.g.::

    [app:karlserve-search]
    use = karlserve
    connection_stats_filename = %(here)s/../var/log/search_connection_stats.csv
    connection_stats_threshhold = 0.1

    [pipeline:search]
    pipeline =
        egg:guillotine#guillotine
        egg:Paste#evalerror
        egg:karl#timeit
        browserid
        karlserve-search

We can then pass '--application-name=search' in via the command line
entry in 'etc/supervisord.conf' to select that pipeline.

[Tres Seaver (tseaver)]

Of course, that example is actually for https://bugs.launchpad.net/karl3/+bug/1302111.

[Tres Seaver (tseaver)]

From looking at the 'perfmetrics' code, the tweak we need here is to add a '?prefix=search' to the 'statsd_uri' for the search appserver. E.g., instead of::

  statsd_uri = http://localhost:8125/

use::

 statsd_uri = http://localhost:8125/?prefix=search

[Tres Seaver (tseaver)]

The configuration change is now checked into osideploy and rolled out to staging.

Paul, can you give me access to the Datadog instance to verify that the search stats are being split out?

[Paul Everitt (paul-agendaless)]

FWIW, the DataDog agent wasn't running on staging, but I suspect you know that and started it.

--Paul

On Apr 4, 2014, at 3:06 PM, Tres Seaver <email address hidden> wrote:

> The configuration change is now checked into osideploy and rolled out to
> staging.
>
> Paul, can you give me access to the Datadog instance to verify that the
> search stats are being split out?
>
> ** Changed in: karl3
> Status: In Progress => Fix Committed
>
> ** Changed in: karl3
> Assignee: Tres Seaver (tseaver) => Paul Everitt (paul-agendaless)
>
> --
> You received this bug notification because you are subscribed to KARL3.
> https://bugs.launchpad.net/bugs/1302121
>
> Title:
> Investigate splitting search ghetto stats in Datadog
>
> Status in KARL3:
> Fix Committed
>
> Bug description:
> As with connection stats (#1302111), the statistics logged from each
> appserver to Datadog mingle the 'search' and the 'main' instances.
> Figure out how to split those out.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/1302121/+subscriptions

[Tres Seaver (tseaver)]

This change is *not* checked in or rolled out anywhere (I appear to have confused #130211's work).

[Tres Seaver (tseaver)]

*Now* the change is checked in and rolled out to staging.

[Tres Seaver (tseaver)]

Rolled out to production.