warning: log line attempted over max size - leadership related

[David Britton]

This environment is nothing too special. Juju 1.23.0, been running for over a month. Now I'm getting lots of errors in my syslog, meanwhile jujud and mongo are both spinning on cpu taking up in the 60% range all the time, while using up in the 1G range of memory.

Restarting the services is not enough to clear things up.

I attached a full machine-0.log (it's over 9G uncompressed) and a mongo database dump.

Getting messages repeating every minute or so like this:

May 21 18:44:44 juju-lcy01-machine-0 mongod.37017[7124]: Thu May 21 18:44:44.077 [conn50] warning: log line attempted (677k) over max size(10k), printing beginning and end ... update juju.txns.stash query: { _id: { c: "lease", id: "trusty-server-leadership" }, n: "d5b7100b" } update: { $set: { txn-queue: [ "55352def91a7b10f460007b4_127104b4", "55352e1091a7b10f46000802_01edfd48", "55352e2a91a7b10f4600085a_dbb7318e", "55352e4891a7b10f460008ce_01a667f7", "55352e6591a7b10f460009cd_3991da34",
...
"555e26bc91a7b11ba5029c5f_113c738e", "555e271f91a7b11ba5029c6e_9c81bef2", "555e279591a7b11ba5029c7d_06584eb9" ], txn-revno: -37441 }, $unset: { n: 1 } } nscanned:1 nmoved:1 nupdated:1 keyUpdates:0 locks(micros) w:246896 126ms

[Menno Finlay-Smits (menno.smits)]

I've had a look at the DB and the lease collection looks like it has problems. All the lease documents have huge numbers of txn-queue entries, which all refer to completed transactions. Typically mgo/txn will leave only the last completed transaction to touch a document in the txn-queue field so this is quite unusual.

Here's the txn-queue sizes of each of the lease docs in the dump:

landscape-client-precise-leadership: 20746
landscape-client-trusty-leadership: 21381
landscape-client-utopic-leadership: 20685
landscape-client-vivid-leadership: 5942
precise-client-leadership: 18785
precise-server-leadership: 18777
swap-leadership: 18690
trusty-client-leadership: 18769
trusty-server-leadership: 18771
ubuntu-leadership: 18782
utopic-client-leadership: 18747
utopic-server-leadership: 18751
vivid-client-leadership: 18714
vivid-server-leadership: 5918

I'm not sure if this is related to bug 1454891 but it certainly could be.