backup-restore failed creating collection EOF

[Curtis Hovey]

As seen in
    http://reports.vapour.ws/releases/issue/57922732749a5624aac9f7b8

backup-restore failed creating collection juju.blockdevices

22T12:59:33.908+0000 Failed: juju.blockdevices: error creating collection juju.blockdevices: error running create command: EOF;

or

Failed: presence.presence.pings: error creating collection presence.presence.pings: error running create command: EOF;

[Alexis Bruemmer (alexis-bruemmer)]

William, this may be related to lp:1604641

[William Reade (fwereade)]

I don't think it is related; and the closest I can see is https://jira.mongodb.org/browse/TOOLS-939 -- which presents slightly differently, but the workaround is something we should be able to use anyway.

I haven't reproed it, and there might well be something about the backed-up collection sizes that triggers it; do we have the backup artifact from a failed run, by any chance? an unexpectedly huge backup may be pointing to another issue...

http://reviews.vapour.ws/r/5300/

[Curtis Hovey (sinzui)]

This issue can also happen dropping a collection.

[William Reade (fwereade)]

Above PR has landed; so this *might* be fixed on master. (But if we see it again, it's definitely not...)

[Curtis Hovey (sinzui)]

The EOF is still seen restoring to a xenial host

[Christian Muirhead (2-xtian)]

Are we still seeing this happening? On http://reports.vapour.ws/releases/issue/57922732749a5624aac9f7b8 the last occurrence is 2016-07-27, which is 3 weeks ago. Are instances not being added to the page, or is it being seen in some other context? It was happening multiple times a day before then.

That said, that last failure is definitely from a revision after Will's commit adding the batch size.

[Anastasia (anastasia-macmood)]

The fix went in

[Christopher Lee (veebers)]

https://bugs.launchpad.net/juju/+bug/1606308 is blocking all restore tests so we're not getting a run of the test that will exercise this bug.

[Anastasia (anastasia-macmood)]

@Christian
As per Christopher's comment - the bug still exists. The test results you've looked at are not reliable as this tests have not run recently.

[Christian Muirhead (2-xtian)]

Ok, sorry. I'm looking at this again today.

[Nicholas Skaggs (nskaggs)]

I've updated the issue with to better catch examples of this type of failure that are very close to the original EOF. It's still happening now in CI; the issue page is recording the new failures better. Have a new look.

[Nicholas Skaggs (nskaggs)]

Also, there's been a couple instances of rather similar restore failures;

http://reports.vapour.ws/releases/issue/57c72994749a56206988839a
http://reports.vapour.ws/releases/issue/57c72a21749a56206988839b

For now I'm assuming this bug covers it. We'll watch and open a new bug if we feel it's a new strain of error not covered here.

[Horacio Durán (hduran-8)]

I proposed http://reviews.vapour.ws/r/5605/

[Curtis Hovey (sinzui)]

This bug is still seen by CI. IT is not fixed yet. Manual testing shows a restore can succeed. We are investigating if the test script introduces timing or network issues.

[Curtis Hovey (sinzui)]

The assess_recovery script is updated. It verifies that the hosted model is operational when restore reports it is successful. This bug is still observed. As the error is happening within a single juju command (juju restore-backup) there is nothing the test can change to mitigate that restore on xenial with mongo 3.x fails, unlike restore of trusty with mongio 2.4. The test passes when we use an older juju 2.

Manual tests of a restore of xenial have been observed to report a pass, but that leads to the next bug, juju thinks it is upgrading, which is impossible. If juju were to think it was upgrading during a restore, EOF my might be a symptom.

See bug 1606265 (juju thinks it is upgrade, but cannot possibly be upgrading) and bug 1625258 (juju is not permitted to auto upgrade, but it thinks it is).

[Tim Penhey (thumper)]

Curtis, this may have been fixed by my latest restore branch. Can we keep an eye on it?