| 2019-03-22 18:49:42 |
Simon Richardson |
bug |
|
|
added bug |
| 2019-03-22 19:08:42 |
Andreas Hasenack |
bug |
|
|
added subscriber Andreas Hasenack |
| 2019-03-22 19:08:54 |
Andreas Hasenack |
mongodb (Ubuntu): assignee |
|
Andreas Hasenack (ahasenack) |
|
| 2019-03-22 19:08:57 |
Andreas Hasenack |
mongodb (Ubuntu): status |
New |
In Progress |
|
| 2019-03-22 19:09:01 |
Andreas Hasenack |
mongodb (Ubuntu): importance |
Undecided |
Critical |
|
| 2019-03-22 19:39:27 |
Simon Richardson |
description |
[Impact]
When an operator of Juju attempts to restore a backup, it fails with a mongorestore related error[0]. Under the hood, Juju uses the supplied mongo related command line interfaces (mongodump, mongorestore etc), using these in conjunction with the oplog related flags for the former CLIs, the mongorestore fails with the following error:
> Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
The only work around for Juju at the moment is to update mongod, so that it carries the following commit[1]. MongoDB have an identical issue, under a different name, which fixes this[2], but there is unfortunately no work around without upgrading mongod so it caries the former commit.
The original investigation around Juju failing to restore a previous dump can be seen here[3]
I've verified with the 3.6.11 and the same restore, worked fine.
[Test Case]
The following test case doesn't require Juju, and can be replicated quite easily inside a lxc container. It may require 2 or 3 goes, depending on timings of the creation of the index and the mongodump, but I've been getting this consistently.
You can also find instructions in one pastebin[4]
// Launch a container
lxc launch ubuntu:18.04 mongo
lxc exec mongo bash
// Install a version of mongo that Juju is using.
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 58712A2291FA4AD5
sudo bash -c 'echo "deb http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" > /etc/apt/sources.list.d/mongodb-org-3.6.list'
sudo apt update
sudo apt-get install -y mongodb-org=3.6.3 mongodb-org-server=3.6.3 mongodb-org-shell=3.6.3 mongodb-org-mongos=3.6.3 mongodb-org-tools=3.6.3
// Apply the patch to /etc/mongod.conf to enable the replication - this is required.
--- /etc/mongod.conf 2019-03-22 17:17:10.530432761 +0000
+++ /etc/mongod.conf 2019-03-22 17:16:51.602528908 +0000
@@ -32,8 +32,7 @@
#operationProfiling:
-replication:
- replSetName: "rs0"
+#replication:
#sharding:
// Start the mongod service
systemctl enable mongod.service
systemctl start mongod.service
// not required, but helps diagnose bsondump
snap install jq
// launch the mongo CLI, running each command
mongo --eval "rs.initiate()"
mongo --eval "rs.status()"
mongo --eval "db.foo.insert({'foo':'bar'})"
mongo --eval "db.foo.createIndex({'bar':1})"
// Get the dump
mongodump --oplog
// check if it's going to fail
// if the output contains a createIndex, but doesn't contain an oid then it'll fail.
bsondump dump/oplog.bson | jq
// Drop the foo collection, so we don't get any duplication errors
mongo --eval "db.foo.drop()"
// Now restore the dump
mongorestore --oplogReplay dump/
// It should fail with
2019-03-22T17:26:28.034+0000 preparing collections to restore from
2019-03-22T17:26:28.035+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T17:26:28.053+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T17:26:28.056+0000 restoring indexes for collection test.foo from metadata
2019-03-22T17:26:28.062+0000 finished restoring test.foo (1 document)
2019-03-22T17:26:28.062+0000 replaying oplog
2019-03-22T17:26:28.064+0000 Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
[Other Info]
0. https://paste.ubuntu.com/p/9r9pYc9g9D/
1. https://github.com/mongodb/mongo/commit/0d0ba866052fd2b9ceaaa66c2b725a02822b102d
2. https://jira.mongodb.org/browse/SERVER-36951
3. https://bugs.launchpad.net/juju/+bug/1818831
4. https://paste.ubuntu.com/p/HJsDf5yNkS/ |
[Impact]
When an operator of Juju attempts to restore a backup, it fails with a mongorestore related error[0]. Under the hood, Juju uses the supplied mongo related command line interfaces (mongodump, mongorestore etc), using these in conjunction with the oplog related flags for the former CLIs, the mongorestore fails with the following error:
> Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
The only work around for Juju at the moment is to update mongod, so that it carries the following commit[1]. MongoDB have an identical issue, under a different name, which fixes this[2], but there is unfortunately no work around without upgrading mongod so it caries the former commit.
The original investigation around Juju failing to restore a previous dump can be seen here[3]
I've verified with the 3.6.11 and the same restore, worked fine.
[Test Case]
The following test case doesn't require Juju, and can be replicated quite easily inside a lxc container. It may require 2 or 3 goes, depending on timings of the creation of the index and the mongodump, but I've been getting this consistently.
You can also find instructions in one pastebin[4]
// Launch a container
lxc launch ubuntu:18.04 mongo
lxc exec mongo bash
// Install a version of mongo that Juju is using.
sudo apt update
sudo apt install -y mongodb
// Edit /etc/mongodb.conf and add the following line:
replSet = "rs0"
// Restart the service
systemctl restart mongodb.service
// not required, but helps diagnose bsondump
snap install jq
// launch the mongo CLI, running each command
mongo --eval "rs.initiate()"
mongo --eval "rs.status()"
mongo --eval "db.foo.insert({'foo':'bar'})"
mongo --eval "db.foo.createIndex({'bar':1})"
// Get the dump
mongodump --oplog
// check if it's going to fail
// if the output contains a createIndex, but doesn't contain an oid then it'll fail.
bsondump dump/oplog.bson | jq
// Drop the foo collection, so we don't get any duplication errors
mongo --eval "db.foo.drop()"
// Now restore the dump
mongorestore --oplogReplay dump/
// It should fail with
2019-03-22T17:26:28.034+0000 preparing collections to restore from
2019-03-22T17:26:28.035+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T17:26:28.053+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T17:26:28.056+0000 restoring indexes for collection test.foo from metadata
2019-03-22T17:26:28.062+0000 finished restoring test.foo (1 document)
2019-03-22T17:26:28.062+0000 replaying oplog
2019-03-22T17:26:28.064+0000 Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
[Other Info]
0. https://paste.ubuntu.com/p/9r9pYc9g9D/
1. https://github.com/mongodb/mongo/commit/0d0ba866052fd2b9ceaaa66c2b725a02822b102d
2. https://jira.mongodb.org/browse/SERVER-36951
3. https://bugs.launchpad.net/juju/+bug/1818831
4. https://paste.ubuntu.com/p/HJsDf5yNkS/ |
|
| 2019-03-22 19:51:56 |
Andreas Hasenack |
description |
[Impact]
When an operator of Juju attempts to restore a backup, it fails with a mongorestore related error[0]. Under the hood, Juju uses the supplied mongo related command line interfaces (mongodump, mongorestore etc), using these in conjunction with the oplog related flags for the former CLIs, the mongorestore fails with the following error:
> Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
The only work around for Juju at the moment is to update mongod, so that it carries the following commit[1]. MongoDB have an identical issue, under a different name, which fixes this[2], but there is unfortunately no work around without upgrading mongod so it caries the former commit.
The original investigation around Juju failing to restore a previous dump can be seen here[3]
I've verified with the 3.6.11 and the same restore, worked fine.
[Test Case]
The following test case doesn't require Juju, and can be replicated quite easily inside a lxc container. It may require 2 or 3 goes, depending on timings of the creation of the index and the mongodump, but I've been getting this consistently.
You can also find instructions in one pastebin[4]
// Launch a container
lxc launch ubuntu:18.04 mongo
lxc exec mongo bash
// Install a version of mongo that Juju is using.
sudo apt update
sudo apt install -y mongodb
// Edit /etc/mongodb.conf and add the following line:
replSet = "rs0"
// Restart the service
systemctl restart mongodb.service
// not required, but helps diagnose bsondump
snap install jq
// launch the mongo CLI, running each command
mongo --eval "rs.initiate()"
mongo --eval "rs.status()"
mongo --eval "db.foo.insert({'foo':'bar'})"
mongo --eval "db.foo.createIndex({'bar':1})"
// Get the dump
mongodump --oplog
// check if it's going to fail
// if the output contains a createIndex, but doesn't contain an oid then it'll fail.
bsondump dump/oplog.bson | jq
// Drop the foo collection, so we don't get any duplication errors
mongo --eval "db.foo.drop()"
// Now restore the dump
mongorestore --oplogReplay dump/
// It should fail with
2019-03-22T17:26:28.034+0000 preparing collections to restore from
2019-03-22T17:26:28.035+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T17:26:28.053+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T17:26:28.056+0000 restoring indexes for collection test.foo from metadata
2019-03-22T17:26:28.062+0000 finished restoring test.foo (1 document)
2019-03-22T17:26:28.062+0000 replaying oplog
2019-03-22T17:26:28.064+0000 Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
[Other Info]
0. https://paste.ubuntu.com/p/9r9pYc9g9D/
1. https://github.com/mongodb/mongo/commit/0d0ba866052fd2b9ceaaa66c2b725a02822b102d
2. https://jira.mongodb.org/browse/SERVER-36951
3. https://bugs.launchpad.net/juju/+bug/1818831
4. https://paste.ubuntu.com/p/HJsDf5yNkS/ |
[Impact]
When an operator of Juju attempts to restore a backup, it fails with a mongorestore related error[0]. Under the hood, Juju uses the supplied mongo related command line interfaces (mongodump, mongorestore etc), using these in conjunction with the oplog related flags for the former CLIs, the mongorestore fails with the following error:
> Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
The only work around for Juju at the moment is to update mongod, so that it carries the following commit[1]. MongoDB have an identical issue, under a different name, which fixes this[2], but there is unfortunately no work around without upgrading mongod so it caries the former commit.
The original investigation around Juju failing to restore a previous dump can be seen here[3]
I've verified with the 3.6.11 and the same restore, worked fine.
[Test Case]
The following test case doesn't require Juju, and can be replicated quite easily inside a lxc container. It may require 2 or 3 goes, depending on timings of the creation of the index and the mongodump, but I've been getting this consistently.
You can also find instructions in one pastebin[4]
// Launch a container
lxc launch ubuntu:18.04 mongo
lxc exec mongo bash
// Install a version of mongo that Juju is using.
sudo apt update
sudo apt install -y mongodb
// Edit /etc/mongodb.conf and add the following line:
replSet = "rs0"
// Restart the service
systemctl restart mongodb.service
// not required, but helps diagnose bsondump
snap install jq
// launch the mongo CLI, running each command
mongo --eval "rs.initiate()"
mongo --eval "rs.status()"
mongo --eval "db.foo.insert({'foo':'bar'})"
mongo --eval "db.foo.createIndex({'bar':1})"
// Get the dump
mongodump --oplog
// check if it's going to fail
// if the output contains a createIndex, but doesn't contain an oid then it'll fail.
bsondump dump/oplog.bson | jq
// Drop the foo collection, so we don't get any duplication errors
mongo --eval "db.foo.drop()"
// Now restore the dump
mongorestore --oplogReplay dump/
// It should fail with
2019-03-22T17:26:28.034+0000 preparing collections to restore from
2019-03-22T17:26:28.035+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T17:26:28.053+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T17:26:28.056+0000 restoring indexes for collection test.foo from metadata
2019-03-22T17:26:28.062+0000 finished restoring test.foo (1 document)
2019-03-22T17:26:28.062+0000 replaying oplog
2019-03-22T17:26:28.064+0000 Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
After updating the packages, re-run the drop and try to import again, this time it should work:
# mongo --eval "db.foo.drop()"
MongoDB shell version v3.6.3
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.3
true
r
# mongorestore --oplogReplay dump/
2019-03-22T19:49:43.838+0000 preparing collections to restore from
2019-03-22T19:49:43.840+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T19:49:43.856+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T19:49:43.858+0000 restoring indexes for collection test.foo from metadata
2019-03-22T19:49:43.866+0000 finished restoring test.foo (1 document)
2019-03-22T19:49:43.866+0000 replaying oplog
2019-03-22T19:49:43.867+0000 done
[Other Info]
0. https://paste.ubuntu.com/p/9r9pYc9g9D/
1. https://github.com/mongodb/mongo/commit/0d0ba866052fd2b9ceaaa66c2b725a02822b102d
2. https://jira.mongodb.org/browse/SERVER-36951
3. https://bugs.launchpad.net/juju/+bug/1818831
4. https://paste.ubuntu.com/p/HJsDf5yNkS/ |
|
| 2019-03-22 19:58:07 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~ahasenack/ubuntu/+source/mongodb/+git/mongodb/+merge/364985 |
|
| 2019-03-25 08:01:13 |
Christian Ehrhardt |
description |
[Impact]
When an operator of Juju attempts to restore a backup, it fails with a mongorestore related error[0]. Under the hood, Juju uses the supplied mongo related command line interfaces (mongodump, mongorestore etc), using these in conjunction with the oplog related flags for the former CLIs, the mongorestore fails with the following error:
> Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
The only work around for Juju at the moment is to update mongod, so that it carries the following commit[1]. MongoDB have an identical issue, under a different name, which fixes this[2], but there is unfortunately no work around without upgrading mongod so it caries the former commit.
The original investigation around Juju failing to restore a previous dump can be seen here[3]
I've verified with the 3.6.11 and the same restore, worked fine.
[Test Case]
The following test case doesn't require Juju, and can be replicated quite easily inside a lxc container. It may require 2 or 3 goes, depending on timings of the creation of the index and the mongodump, but I've been getting this consistently.
You can also find instructions in one pastebin[4]
// Launch a container
lxc launch ubuntu:18.04 mongo
lxc exec mongo bash
// Install a version of mongo that Juju is using.
sudo apt update
sudo apt install -y mongodb
// Edit /etc/mongodb.conf and add the following line:
replSet = "rs0"
// Restart the service
systemctl restart mongodb.service
// not required, but helps diagnose bsondump
snap install jq
// launch the mongo CLI, running each command
mongo --eval "rs.initiate()"
mongo --eval "rs.status()"
mongo --eval "db.foo.insert({'foo':'bar'})"
mongo --eval "db.foo.createIndex({'bar':1})"
// Get the dump
mongodump --oplog
// check if it's going to fail
// if the output contains a createIndex, but doesn't contain an oid then it'll fail.
bsondump dump/oplog.bson | jq
// Drop the foo collection, so we don't get any duplication errors
mongo --eval "db.foo.drop()"
// Now restore the dump
mongorestore --oplogReplay dump/
// It should fail with
2019-03-22T17:26:28.034+0000 preparing collections to restore from
2019-03-22T17:26:28.035+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T17:26:28.053+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T17:26:28.056+0000 restoring indexes for collection test.foo from metadata
2019-03-22T17:26:28.062+0000 finished restoring test.foo (1 document)
2019-03-22T17:26:28.062+0000 replaying oplog
2019-03-22T17:26:28.064+0000 Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
After updating the packages, re-run the drop and try to import again, this time it should work:
# mongo --eval "db.foo.drop()"
MongoDB shell version v3.6.3
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.3
true
r
# mongorestore --oplogReplay dump/
2019-03-22T19:49:43.838+0000 preparing collections to restore from
2019-03-22T19:49:43.840+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T19:49:43.856+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T19:49:43.858+0000 restoring indexes for collection test.foo from metadata
2019-03-22T19:49:43.866+0000 finished restoring test.foo (1 document)
2019-03-22T19:49:43.866+0000 replaying oplog
2019-03-22T19:49:43.867+0000 done
[Other Info]
0. https://paste.ubuntu.com/p/9r9pYc9g9D/
1. https://github.com/mongodb/mongo/commit/0d0ba866052fd2b9ceaaa66c2b725a02822b102d
2. https://jira.mongodb.org/browse/SERVER-36951
3. https://bugs.launchpad.net/juju/+bug/1818831
4. https://paste.ubuntu.com/p/HJsDf5yNkS/ |
[Impact]
When an operator of Juju attempts to restore a backup, it fails with a mongorestore related error[0]. Under the hood, Juju uses the supplied mongo related command line interfaces (mongodump, mongorestore etc), using these in conjunction with the oplog related flags for the former CLIs, the mongorestore fails with the following error:
> Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
The only work around for Juju at the moment is to update mongod, so that it carries the following commit[1]. MongoDB have an identical issue, under a different name, which fixes this[2], but there is unfortunately no work around without upgrading mongod so it caries the former commit.
The original investigation around Juju failing to restore a previous dump can be seen here[3]
I've verified with the 3.6.11 and the same restore, worked fine.
[Test Case]
The following test case doesn't require Juju, and can be replicated quite easily inside a lxc container. It may require 2 or 3 goes, depending on timings of the creation of the index and the mongodump, but I've been getting this consistently.
Please do mind that timing is important, so putting the commands in a script will help a lot to reliably trigger the issue.
You can also find instructions in one pastebin[4]
// Launch a container
lxc launch ubuntu:18.04 mongo
lxc exec mongo bash
// Install a version of mongo that Juju is using.
sudo apt update
sudo apt install -y mongodb
// Edit /etc/mongodb.conf and add the following line:
replSet = "rs0"
// Restart the service
systemctl restart mongodb.service
// not required, but helps diagnose bsondump
snap install jq
// launch the mongo CLI, running each command
mongo --eval "rs.initiate()"
mongo --eval "rs.status()"
mongo --eval "db.foo.insert({'foo':'bar'})"
mongo --eval "db.foo.createIndex({'bar':1})"
// Get the dump
mongodump --oplog
// check if it's going to fail
// if the output contains a createIndex, but doesn't contain an oid then it'll fail.
bsondump dump/oplog.bson | jq
// Drop the foo collection, so we don't get any duplication errors
mongo --eval "db.foo.drop()"
// Now restore the dump
mongorestore --oplogReplay dump/
// It should fail with
2019-03-22T17:26:28.034+0000 preparing collections to restore from
2019-03-22T17:26:28.035+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T17:26:28.053+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T17:26:28.056+0000 restoring indexes for collection test.foo from metadata
2019-03-22T17:26:28.062+0000 finished restoring test.foo (1 document)
2019-03-22T17:26:28.062+0000 replaying oplog
2019-03-22T17:26:28.064+0000 Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
After updating the packages, re-run the drop and try to import again, this time it should work:
# mongo --eval "db.foo.drop()"
MongoDB shell version v3.6.3
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.3
true
r
# mongorestore --oplogReplay dump/
2019-03-22T19:49:43.838+0000 preparing collections to restore from
2019-03-22T19:49:43.840+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T19:49:43.856+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T19:49:43.858+0000 restoring indexes for collection test.foo from metadata
2019-03-22T19:49:43.866+0000 finished restoring test.foo (1 document)
2019-03-22T19:49:43.866+0000 replaying oplog
2019-03-22T19:49:43.867+0000 done
[Other Info]
0. https://paste.ubuntu.com/p/9r9pYc9g9D/
1. https://github.com/mongodb/mongo/commit/0d0ba866052fd2b9ceaaa66c2b725a02822b102d
2. https://jira.mongodb.org/browse/SERVER-36951
3. https://bugs.launchpad.net/juju/+bug/1818831
4. https://paste.ubuntu.com/p/HJsDf5yNkS/ |
|
| 2019-03-25 08:01:20 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Cosmic |
|
| 2019-03-25 08:01:20 |
Christian Ehrhardt |
bug task added |
|
mongodb (Ubuntu Cosmic) |
|
| 2019-03-25 08:01:20 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Bionic |
|
| 2019-03-25 08:01:20 |
Christian Ehrhardt |
bug task added |
|
mongodb (Ubuntu Bionic) |
|
| 2019-03-25 08:01:25 |
Christian Ehrhardt |
mongodb (Ubuntu): status |
In Progress |
Fix Released |
|
| 2019-03-25 08:01:28 |
Christian Ehrhardt |
mongodb (Ubuntu Bionic): status |
New |
In Progress |
|
| 2019-03-25 08:01:31 |
Christian Ehrhardt |
mongodb (Ubuntu Cosmic): status |
New |
In Progress |
|
| 2019-03-25 08:01:37 |
Christian Ehrhardt |
mongodb (Ubuntu Bionic): assignee |
|
Andreas Hasenack (ahasenack) |
|
| 2019-03-25 08:01:41 |
Christian Ehrhardt |
mongodb (Ubuntu Cosmic): assignee |
|
Andreas Hasenack (ahasenack) |
|
| 2019-03-25 08:01:44 |
Christian Ehrhardt |
mongodb (Ubuntu): assignee |
Andreas Hasenack (ahasenack) |
|
|
| 2019-03-25 08:01:48 |
Christian Ehrhardt |
mongodb (Ubuntu Bionic): importance |
Undecided |
Critical |
|
| 2019-03-25 08:01:49 |
Christian Ehrhardt |
mongodb (Ubuntu Cosmic): importance |
Undecided |
Critical |
|
| 2019-03-25 14:49:16 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2019-03-27 18:04:12 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~ahasenack/ubuntu/+source/mongodb/+git/mongodb/+merge/365188 |
|
| 2019-03-28 17:37:44 |
Andreas Hasenack |
description |
[Impact]
When an operator of Juju attempts to restore a backup, it fails with a mongorestore related error[0]. Under the hood, Juju uses the supplied mongo related command line interfaces (mongodump, mongorestore etc), using these in conjunction with the oplog related flags for the former CLIs, the mongorestore fails with the following error:
> Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
The only work around for Juju at the moment is to update mongod, so that it carries the following commit[1]. MongoDB have an identical issue, under a different name, which fixes this[2], but there is unfortunately no work around without upgrading mongod so it caries the former commit.
The original investigation around Juju failing to restore a previous dump can be seen here[3]
I've verified with the 3.6.11 and the same restore, worked fine.
[Test Case]
The following test case doesn't require Juju, and can be replicated quite easily inside a lxc container. It may require 2 or 3 goes, depending on timings of the creation of the index and the mongodump, but I've been getting this consistently.
Please do mind that timing is important, so putting the commands in a script will help a lot to reliably trigger the issue.
You can also find instructions in one pastebin[4]
// Launch a container
lxc launch ubuntu:18.04 mongo
lxc exec mongo bash
// Install a version of mongo that Juju is using.
sudo apt update
sudo apt install -y mongodb
// Edit /etc/mongodb.conf and add the following line:
replSet = "rs0"
// Restart the service
systemctl restart mongodb.service
// not required, but helps diagnose bsondump
snap install jq
// launch the mongo CLI, running each command
mongo --eval "rs.initiate()"
mongo --eval "rs.status()"
mongo --eval "db.foo.insert({'foo':'bar'})"
mongo --eval "db.foo.createIndex({'bar':1})"
// Get the dump
mongodump --oplog
// check if it's going to fail
// if the output contains a createIndex, but doesn't contain an oid then it'll fail.
bsondump dump/oplog.bson | jq
// Drop the foo collection, so we don't get any duplication errors
mongo --eval "db.foo.drop()"
// Now restore the dump
mongorestore --oplogReplay dump/
// It should fail with
2019-03-22T17:26:28.034+0000 preparing collections to restore from
2019-03-22T17:26:28.035+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T17:26:28.053+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T17:26:28.056+0000 restoring indexes for collection test.foo from metadata
2019-03-22T17:26:28.062+0000 finished restoring test.foo (1 document)
2019-03-22T17:26:28.062+0000 replaying oplog
2019-03-22T17:26:28.064+0000 Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
After updating the packages, re-run the drop and try to import again, this time it should work:
# mongo --eval "db.foo.drop()"
MongoDB shell version v3.6.3
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.3
true
r
# mongorestore --oplogReplay dump/
2019-03-22T19:49:43.838+0000 preparing collections to restore from
2019-03-22T19:49:43.840+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T19:49:43.856+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T19:49:43.858+0000 restoring indexes for collection test.foo from metadata
2019-03-22T19:49:43.866+0000 finished restoring test.foo (1 document)
2019-03-22T19:49:43.866+0000 replaying oplog
2019-03-22T19:49:43.867+0000 done
[Other Info]
0. https://paste.ubuntu.com/p/9r9pYc9g9D/
1. https://github.com/mongodb/mongo/commit/0d0ba866052fd2b9ceaaa66c2b725a02822b102d
2. https://jira.mongodb.org/browse/SERVER-36951
3. https://bugs.launchpad.net/juju/+bug/1818831
4. https://paste.ubuntu.com/p/HJsDf5yNkS/ |
[Impact]
When an operator of Juju attempts to restore a backup, it fails with a mongorestore related error[0]. Under the hood, Juju uses the supplied mongo related command line interfaces (mongodump, mongorestore etc), using these in conjunction with the oplog related flags for the former CLIs, the mongorestore fails with the following error:
> Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
The only work around for Juju at the moment is to update mongod, so that it carries the following commit[1]. MongoDB have an identical issue, under a different name, which fixes this[2], but there is unfortunately no work around without upgrading mongod so it caries the former commit.
The original investigation around Juju failing to restore a previous dump can be seen here[3]
I've verified with the 3.6.11 and the same restore, worked fine.
[Test Case]
The following test case doesn't require Juju, and can be replicated quite easily inside a lxc container. It may require 2 or 3 goes, depending on timings of the creation of the index and the mongodump, but I've been getting this consistently.
Please do mind that timing is important, so putting the commands in a script will help a lot to reliably trigger the issue.
You can also find instructions in one pastebin[4]
// Launch a container
lxc launch ubuntu:18.04 mongo
lxc exec mongo bash
// Install a version of mongo that Juju is using.
sudo apt update
sudo apt install -y mongodb
// Edit /etc/mongodb.conf and add the following line:
replSet = "rs0"
// Restart the service
sudo systemctl restart mongodb.service
// not required, but helps diagnose bsondump
sudo snap install jq
// launch the mongo CLI, running each command
mongo --eval "rs.initiate()"
mongo --eval "rs.status()"
mongo --eval "db.foo.insert({'foo':'bar'})"
mongo --eval "db.foo.createIndex({'bar':1})"
// Get the dump
mongodump --oplog
// check if it's going to fail
// if the output contains a createIndex, but doesn't contain an oid then it'll fail.
bsondump dump/oplog.bson | jq
// Drop the foo collection, so we don't get any duplication errors
mongo --eval "db.foo.drop()"
// Now restore the dump
mongorestore --oplogReplay dump/
// It should fail with
2019-03-22T17:26:28.034+0000 preparing collections to restore from
2019-03-22T17:26:28.035+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T17:26:28.053+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T17:26:28.056+0000 restoring indexes for collection test.foo from metadata
2019-03-22T17:26:28.062+0000 finished restoring test.foo (1 document)
2019-03-22T17:26:28.062+0000 replaying oplog
2019-03-22T17:26:28.064+0000 Failed: restore error: error applying oplog: applyOps: applyOps had one or more errors applying ops
After updating the packages, re-run the drop and try to import again, this time it should work:
# mongo --eval "db.foo.drop()"
MongoDB shell version v3.6.3
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.3
true
r
# mongorestore --oplogReplay dump/
2019-03-22T19:49:43.838+0000 preparing collections to restore from
2019-03-22T19:49:43.840+0000 reading metadata for test.foo from dump/test/foo.metadata.json
2019-03-22T19:49:43.856+0000 restoring test.foo from dump/test/foo.bson
2019-03-22T19:49:43.858+0000 restoring indexes for collection test.foo from metadata
2019-03-22T19:49:43.866+0000 finished restoring test.foo (1 document)
2019-03-22T19:49:43.866+0000 replaying oplog
2019-03-22T19:49:43.867+0000 done
[Other Info]
0. https://paste.ubuntu.com/p/9r9pYc9g9D/
1. https://github.com/mongodb/mongo/commit/0d0ba866052fd2b9ceaaa66c2b725a02822b102d
2. https://jira.mongodb.org/browse/SERVER-36951
3. https://bugs.launchpad.net/juju/+bug/1818831
4. https://paste.ubuntu.com/p/HJsDf5yNkS/ |
|
| 2019-03-29 10:27:32 |
Timo Aaltonen |
mongodb (Ubuntu Cosmic): status |
In Progress |
Fix Committed |
|
| 2019-03-29 10:27:33 |
Timo Aaltonen |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-03-29 10:27:35 |
Timo Aaltonen |
bug |
|
|
added subscriber SRU Verification |
| 2019-03-29 10:27:39 |
Timo Aaltonen |
tags |
|
verification-needed verification-needed-cosmic |
|
| 2019-03-29 10:30:01 |
Timo Aaltonen |
mongodb (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2019-03-29 10:30:09 |
Timo Aaltonen |
tags |
verification-needed verification-needed-cosmic |
verification-needed verification-needed-bionic verification-needed-cosmic |
|
| 2019-03-29 18:38:42 |
Andreas Hasenack |
tags |
verification-needed verification-needed-bionic verification-needed-cosmic |
verification-done-bionic verification-needed verification-needed-cosmic |
|
| 2019-03-29 18:46:29 |
Andreas Hasenack |
tags |
verification-done-bionic verification-needed verification-needed-cosmic |
verification-done-bionic verification-done-cosmic verification-needed |
|
| 2019-04-09 18:38:06 |
Launchpad Janitor |
mongodb (Ubuntu Cosmic): status |
Fix Committed |
Fix Released |
|
| 2019-04-09 18:38:21 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2019-04-09 18:50:19 |
Launchpad Janitor |
mongodb (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
