Could not evalute MongoDB shell command: printjson(rs.conf())

[Derek Higgins]

Tripleo CI has been failing since about 2000 UTC on tuesday

Looks like this is the relevant error in the overcloud controller logs

Jan 20 01:28:00 overcloud-controller-0 os-collect-config[3014]: Error: Could not prefetch mongodb_replset provider 'mongo': Could not evalute MongoDB shell command: printjson(rs.conf())
Jan 20 01:28:00 overcloud-controller-0 os-collect-config[3014]: Warning: Can't connect to replicaset member 192.0.2.8:27017.
Jan 20 01:28:00 overcloud-controller-0 os-collect-config[3014]: Error: /Stage[main]/Main/Mongodb_replset[tripleo]: Could not evaluate: Can't connect to any member of replicaset tripleo.

[Derek Higgins (derekh)]

Running puppet with --trace --debug --verbose shows this output

2016-01-20T12:21:58.192+0000 file [] doesn't exist
failed to load:' Retry: '2'
Debug: Executing '/bin/mongo admin --quiet --host 192.0.2.22:27017 --eval printjson(rs.status())'
Debug: Request failed: 'Execution of '/bin/mongo admin --quiet --host 192.0.2.22:27017 --eval printjson(rs.status())' returned 253: {
        "set" : "tripleo",
        "date" : ISODate("2016-01-20T12:22:01Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "name" : "192.0.2.22:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 70984,
                        "optime" : Timestamp(1453292495, 60),
                        "optimeDate" : ISODate("2016-01-20T12:21:35Z"),
                        "electionTime" : Timestamp(1453221546, 2),
                        "electionDate" : ISODate("2016-01-19T16:39:06Z"),
                        "self" : true
                }
        ],
        "ok" : 1
}
2016-01-20T12:22:01.260+0000 file [] doesn't exist
failed to load:' Retry: '3'
Debug: Got an exception: Could not evalute MongoDB shell command: printjson(rs.status())
Warning: Can't connect to replicaset member 192.0.2.22:27017.
Debug: Alive members: []
Debug: Dead members: ["192.0.2.22:27017"]
Error: /Stage[main]/Main/Mongodb_replset[tripleo]: Could not evaluate: Can't connect to any member of replicaset tripleo.

It appears that the josn being output can't be parsed due to the ISODate and Timestamp types.

[Derek Higgins (derekh)]

Some local testing suggest the failure started when this patch merged

(MODULES-2983) Enable IPv6 in mongodb provider
https://github.com/puppetlabs/puppetlabs-mongodb/commit/a77d1a191419a3f1f468707aefa69abd689c3e4b

[Derek Higgins (derekh)]

The problem is this part of the patch

+ has_ipv6 = ipv6_is_enabled
 + if has_ipv6
 + ipv6 = '--ipv6'
 + else
 + ipv6 = ''
 + end
      out = nil
      retry_count.times do |n|
        begin
          if host
 - out = mongo([db, '--quiet', '--host', host, '--eval', cmd])
 + out = mongo([db, '--quiet', ipv6, '--host', host, '--eval', cmd])

if ipv6 == "", the mongo command is being run with a blank parameter, its interpreting this as the name of a file which it can't open

[root@overcloud-controller-0 mongodb]# /bin/mongo admin --quiet "" --host 192.0.2.22:27017 --eval "printjson(rs.conf())"
{
        "_id" : "tripleo",
        "version" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "192.0.2.22:27017"
                }
        ]
}
2016-01-20T12:42:02.127+0000 file [] doesn't exist
failed to load:
[root@overcloud-controller-0 mongodb]# echo $?
253

[Emilien Macchi (emilienm)]

Will be fixed by https://github.com/puppetlabs/puppetlabs-mongodb/pull/257

[Derek Higgins (derekh)]

The fix for this has been merged into puppetlabs-mongodb