'debug-log' fails when logs are large when using db-log

[John A Meinel]

While doing a scale test I ran into this failure:
2015-12-09 00:29:21 ERROR juju.apiserver debuglog.go:96 debug-log handler error: tailer stopped: too much data for sort() with no index. add an index or specify a smaller limit

I have 1500 ubuntu units, so there is a significant level of logging going on. Apparently the code which tracks the log content is getting overloaded and Mongo is refusing to do time sorting for us.

This is in an environment bootstrapped with "JUJU_DEV_FEATURE_FLAG=db-log,disable-rsyslog"

[John A Meinel (jameinel)]

It turns out we have an index on environment UUID and time (e,t) but we are doing a filter e and sort on (t, _id), but Mongo doesn't seem to do a good job noticing that it could use that index. So we have to explicitly match our index in order for sort to use it.
One option is to sort on (e, t) or maybe add an index so we could sort on (e, t, _id).

We're discussing whether we need to sort based on "_id".

[John A Meinel (jameinel)]

The fix should have landed in master.

[Menno Finlay-Smits (menno.smits)]

Thanks very much for both finding and fixing this.

On 11 December 2015 at 08:30, John A Meinel <email address hidden> wrote:

> The fix should have landed in master.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1524135
>
> Title:
> 'debug-log' fails when logs are large when using db-log
>
> Status in juju-core:
> Fix Committed
>
> Bug description:
> While doing a scale test I ran into this failure:
> 2015-12-09 00:29:21 ERROR juju.apiserver debuglog.go:96 debug-log
> handler error: tailer stopped: too much data for sort() with no index. add
> an index or specify a smaller limit
>
> I have 1500 ubuntu units, so there is a significant level of logging
> going on. Apparently the code which tracks the log content is getting
> overloaded and Mongo is refusing to do time sorting for us.
>
>
> This is in an environment bootstrapped with
> "JUJU_DEV_FEATURE_FLAG=db-log,disable-rsyslog"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1524135/+subscriptions
>