model destruction failed because modelEntityRefs was missing some fields

[Junien F]

Hi,

When trying to destroy a model on our 2.4.3 controller, we got the following error message :

ERROR cannot destroy model: failed to destroy model: state changing too quickly; try again soon

which is generally a sign that a transaction got aborted. When looking at aborted transactions related to the destruction of this model, I found the following :

https://pastebin.canonical.com/p/q4rgqT694N/

Looking at this txn, the second and third assertions were true, but for the first one, the document looked like https://pastebin.canonical.com/p/sJ4YSTRjKf/

When I added "volumes" and "filesystems" fields to the modelEntityRefs document (set to []), the model destruction worked.

A lot of modelEntityRefs are missing these fields. Should every single document in this collection have these fields ? Do you have any idea how we got into this situation ?

Thanks !

[Jamon Camisso (jamon)]

I ran into the same issue with a different model, exact same symptoms and missing fields in mongodb.

I ran 'db.modelEntityRefs.update({_id: "13c240e8-04a7-4ae0-8568-d3fdbfd6cf72"}, { $set: { "volumes": [], "filesystems": [] }})', and after those volumes and filesystems fields were added, the modelEntityRefs documet for the model in question showed them and I could then delete it.

[John A Meinel (jameinel)]

My supposition was wrong. Even in Mongo 2.4 $size: 0 only matches if you have an array:
> db.t.insert({"a": []})
> db.t.find()
{ "_id" : ObjectId("5bdea1bb8952436077782726"), "a" : [ ] }
> db.t.find({"a": {$size: 0}})
{ "_id" : ObjectId("5bdea1bb8952436077782726"), "a" : [ ] }
> db.t.find({"b": {$size: 0}})
> exit

We'll need to check how you can get into this situation. I don't know what would be deleting the arrays vs just removing items from it, and if we do need that behavior, then we should handle it in the cleanup with an $or clause.

[Junien F (axino)]

@jam : we just ran into this bug again. Should we proactively add "volumes" and "filesystems" fields for all modelEntityRefs documents that don't have them ?

[Junien F (axino)]

More data to debug : https://pastebin.canonical.com/p/vr6XVyvYtb/ (good and bad models creation date), https://pastebin.canonical.com/p/XzPRgBCJhT/ (db.upgradeInfo dump) - sorry, Canonical-only links

[John A Meinel (jameinel)]

Looking at the code, the description of the modelEntityRefs document is:
type modelEntityRefsDoc struct {
 UUID string `bson:"_id"`

 // Machines contains the names of the top-level machines in the model.
 Machines []string `bson:"machines"`

 // Applications contains the names of the applications in the model.
 Applications []string `bson:"applications"`

 // Volumes contains the IDs of the volumes in the model.
 Volumes []string `bson:"volumes"`

 // Filesystems contains the IDs of the filesystems in the model.
 Filesystems []string `bson:"filesystems"`
}

However, there is no reference in 'upgrades/' to 'volumes' or 'filesystems' that would be adding those values to the documents in the database. Nor is there any reference to 'modelEntityRefs' in upgrades.

And I can confirm that "{$size: 0}" does *not* match non-existent members.

So I think we just have models that have existed before Andrew's change (6e00efc) in 2016-12 that introduced the fields. (It appears to have been introduced in Juju 2.2 so the model would have been created w juju 2.1).

It seems we have a couple possible fixes:

a) Introduce an upgrade step that adds the arrays
b) Change the cleanup code from:
   {"machines", bson.D{{"$size", 0}}},
   {"applications", bson.D{{"$size", 0}}},
   {"volumes", bson.D{{"$size", 0}}},
   {"filesystems", bson.D{{"$size", 0}}},

to something like:
   {"machines", bson.D{{"$size", 0}}},
   {"applications", bson.D{{"$size", 0}}},
   {"$or", bson.D{{"volumes", bson.D{{"$size", 0}}}, bson.D{{"volumes", bson.D{{"$exists", false}}}},
   {"$or", bson.D{{"filesystems", bson.D{{"$size", 0}}}, bson.D{{"filesystems", bson.D{{"$exists", false}}}},

or possibly:
   {"machines", bson.D{{"$size", 0}}},
   {"applications", bson.D{{"$size", 0}}},
   {"$or", bson.D{{bson.M{"volumes": bson.M{"$size": 0}}},
...
Forgive the formatting, as I'm just typing this here without trying to be precise.
The DB query would look something like:
db.modelEntityRefs.find({ $or: [ { "volumes": { $exists: false}}, { "volumes": {$size: 0}}]})
an object with an array of objects is clumsy to type in bson.D objects.

An upgrade step means we don't have to worry about them being different in future versions, a change to the query makes it easier to be compatible. (It feels a little odd to add an upgrade step in 2.5.2 for a data change in 2.2.0).

[Anastasia (anastasia-macmood)]

I agree and believe that we should do both the upgrade step and the change to the query.

[John A Meinel (jameinel)]

https://github.com/juju/juju/pull/9793 is a fix for 2.5

[John A Meinel (jameinel)]

The patch against 2.5 has landed, but it only fixes the Query, it doesn't introduce an upgrade step.
It is also unclear whether this will end up in 2.5.2 or 2.5.3 as we won't block 2.5.2 for it, but it may be that we won't finish getting a clean report for 2.5.2 until after this patch is released.

[John A Meinel (jameinel)]

https://github.com/juju/juju/pull/9819 merges this into develop