mongodb hook failed: "mongos-relation-changed"

[Matt Bruzek]

I am working on the trusty demo on IBM power and we tried to deploy a mongo bundle in the local environment.

I got an error on the mongos/0 unit :

  mongos:
    charm: local:trusty/mongodb-31
    exposed: false
    relations:
      mongos:
      - shard1
      - shard2
      - shard3
      mongos-cfg:
      - configsvr
      replica-set:
      - mongos
    units:
      mongos/0:
        agent-state: error
        agent-state-info: 'hook failed: "mongos-relation-changed"'
        agent-version: 1.17.6.1
        machine: "2"
        open-ports:
        - 27017/tcp
        - 27019/tcp
        - 27021/tcp
        - 28017/tcp
        public-address: 10.0.3.2

The unit-mongos-0.log does not contain very much information:

2014-03-28 19:16:07 INFO juju-log mongos:5: relation_list: relation_id: None
2014-03-28 19:16:07 INFO juju-log mongos:5: relation_id None returns: [u'shard3/0']
2014-03-28 19:16:07 INFO juju-log mongos:5: relation_get: scope: hostname, unit_name: shard3/0, relation_id: None
2014-03-28 19:16:07 INFO juju-log mongos:5: relation_get returns: None
2014-03-28 19:16:07 INFO juju-log mongos:5: relation_get: scope: port, unit_name: shard3/0, relation_id: None
2014-03-28 19:16:07 INFO juju-log mongos:5: relation_get returns: None
2014-03-28 19:16:08 INFO juju-log mongos:5: relation_get: scope: type, unit_name: shard3/0, relation_id: None
2014-03-28 19:16:08 INFO juju-log mongos:5: relation_get returns: None
2014-03-28 19:16:08 INFO juju-log mongos:5: mongos_relation_changed: relation data not ready.
2014-03-28 19:16:08 INFO juju-log mongos:5: mongos_relation_changed returns: False
2014-03-28 19:16:08 ERROR juju.worker.uniter uniter.go:475 hook failed: exit status 1

Please let me know if you need more information.

[Matt Bruzek (mbruzek)]

Also attaching the mongodb_bundle.yaml file that was attempted.

[Tudor Rogoz (rogoz)]

Is this issue scheduled for fixing? Any available workaround? I suppose it's a blocker for deploying mongo database clusters.

Thanks a lot,
Tudor

[Tugdual de Kerviler (dekervit)]

Yes there is a bug here http://bazaar.launchpad.net/~charmers/charms/precise/mongodb/trunk/view/head:/hooks/hooks.py#L1210
When relation data is not ready `mongos_relation_changed` should return true.

It's one of the bugs I fixed here https://code.launchpad.net/~dekervit/charms/precise/mongodb/trunk
but did not ask for a merge as I introduced too many breaking changes (like using the mongodb-org package).

[Juan L. Negron (negronjl)]

I'm working on it ...

-Juan

[Juan L. Negron (negronjl)]

FYI: Although I agree that this has a race condition, a workaround is just to juju resolved --retry mongos/0.

I'm working on the race condition ... just to be clear

-Juan

[Juan L. Negron (negronjl)]

Slippery little bug ... I'm testing some "fixes" ... still working on it.

-Juan

[Juan L. Negron (negronjl)]

Ha! I found it. Testing the fix now.

-Juan

[Juan L. Negron (negronjl)]

I think this is fixed now ... I tested 6 different times using the attached bundle.

Calling it fixed.

[Tudor Rogoz (tudor-x)]

Thanks for the fix Juan, I'll check on my side if is still reproducible.

Tudor