txn suddenly went missing in a txn queue
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Fix Released
|
High
|
John A Meinel | ||
2.3 |
Fix Released
|
High
|
John A Meinel | ||
2.4 |
Fix Released
|
High
|
John A Meinel |
Bug Description
Hi,
Last night, we suddenly got the following logs on our Juju 2.3.7 controller :
2018-06-12 01:45:23 ERROR juju.state cleanup.go:150 cleanup failed for units("ubuntu"): cannot destroy unit "ubuntu/37264": cannot find transaction 5b1f25665f5ce83
I have no idea how it happened. There was some load on the mongodb primary, presumably because we were in the process of backing it up (mongodump).
Also, this happened during an internal prune :
2018-06-12 01:43:50 INFO juju.txn prune.go:127 txns after last prune: 209061, txns now: 356448, pruning: too many new transactions
2018-06-12 01:49:19 INFO juju.txn prune.go:152 txn pruning complete after 5m29.68502757s. txns now: 168206, inspected 68 collections, 24978 docs (623 cleaned)
removed 334 stash docs and 208935 txn docs
The txn was indeed the first in the queue for the unit. However, it wasn't in the queue for the annotation document (which was already in txns.stash). jam advised to try to add it, which I did, using :
db.txns.
{"_id.id": "6cd2c273-
{$push: {
$each: ["5b1f25665f5ce
}
}}
)
(with the controllers down). After that I ran an mgopurge, and the problem went away.
Changed in juju: | |
status: | In Progress → Fix Committed |
Changed in juju: | |
status: | Fix Committed → Fix Released |
Note that another possibility would be to update the txn from status "2" ==
"prepared" to status "1" which is "preparing", in which case it should try
to add the txn id to all the queues. I think you also need to delete the
"n" field. So something like:
db.txns. update( {_id: " 5b1f25665f5ce83 3efeff19b" }, {$set: {"s": 1}, $unset:
{"n": 1}})
On Wed, Jun 13, 2018 at 5:21 PM, Junien Fridrick <<email address hidden>
> wrote:
> Public bug reported: 3efeff19b_ 0e03bf64 in queue for document 658c-44c9- 8cff-42f47eb586 ea:a#ubuntu} stash.update( 658c-44c9- 8cff-42f47eb586 ea:a#ubuntu" }, 833efeff19b_ 0e03bf64" ], /bugs.launchpad .net/bugs/ 1776673 /bugs.launchpad .net/juju/ +bug/1776673/ +subscriptions
>
> Hi,
>
> Last night, we suddenly got the following logs on our Juju 2.3.7
> controller :
>
> 2018-06-12 01:45:23 ERROR juju.state cleanup.go:150 cleanup failed for
> units("ubuntu"): cannot destroy unit "ubuntu/37264": cannot find
> transaction 5b1f25665f5ce83
> {annotations 6cd2c273-
>
> I have no idea how it happened. There was some load on the mongodb
> primary, presumably because we were in the process of backing it up
> (mongodump).
>
> Also, this happened during an internal prune :
>
> 2018-06-12 01:43:50 INFO juju.txn prune.go:127 txns after last prune:
> 209061, txns now: 356448, pruning: too many new transactions
>
> 2018-06-12 01:49:19 INFO juju.txn prune.go:152 txn pruning complete after
> 5m29.68502757s. txns now: 168206, inspected 68 collections, 24978 docs (623
> cleaned)
> removed 334 stash docs and 208935 txn docs
>
>
> The txn was indeed the first in the queue for the unit. However, it wasn't
> in the queue for the annotation document (which was already in txns.stash).
> jam advised to try to add it, which I did, using :
>
> db.txns.
>
> {"_id.id": "6cd2c273-
> {$push: {
> "txn-queue": {
> $each: ["5b1f25665f5ce
> $position: 0
> }
> }}
> )
>
> (with the controllers down). After that I ran an mgopurge, and the
> problem went away.
>
> ** Affects: juju
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https:/
>
> Title:
> txn suddenly went missing in a txn queue
>
> To manage notifications about this bug go to:
> https:/
>