Build 2695: [contrail-alarm-gen]: Error: Consumer Failure LeaderNotAvailableError occured

[Ankit Jain]

Seeing the below error msg in contrail-alarm-gen.log:

01/11/2016 12:12:30 PM [contrail-alarm-gen]: Error: Consumer Failure LeaderNotAvailableError occured. Arguments:
(TopicMetadata(topic='-uve-4', error=5, partitions=[]),) trace Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/opserver/partition_handler.py", line 565, in _run
    consumer = SimpleConsumer(self._kfk, self._group, self._topic, buffer_size = 4096*4, max_buffer_size=4096*32)
  File "/usr/lib/python2.7/dist-packages/kafka/consumer/simple.py", line 115, in __init__
    auto_commit_every_t=auto_commit_every_t)
  File "/usr/lib/python2.7/dist-packages/kafka/consumer/base.py", line 48, in __init__
    self.client.load_metadata_for_topics(topic)
  File "/usr/lib/python2.7/dist-packages/kafka/client.py", line 325, in load_metadata_for_topics
    kafka.common.check_error(topic_metadata)
  File "/usr/lib/python2.7/dist-packages/kafka/common.py", line 215, in check_error
    raise error_class(response)
LeaderNotAvailableError: TopicMetadata(topic='-uve-4', error=5, partitions=[])

Logs copied @ /cs-shared/bugs/<bug-id> on any blr shell server (ex nodeb6)

[Anish Mehta (amehta00)]

Is this reproducible?
Did the system recover itself eventually?
Were you able to take manual steps to recover it?

[Anish Mehta (amehta00)]

From the logs, I cannot tell if the system eventually recovered or not.
Do let me know if you can reproduce it and leave the system in this state.

In any case, next time you see this:
- please save the contents of /var/log/kafka on all DB nodes.
- please run this command on any of the DB nodes:
    /usr/share/kafka/bin# ./kafka-topics.sh --zookeeper 127.0.0.1:2181 --describe

[OpenContrail Admin (ci-admin-f)]

bug update...

[Ato (amonge)]

FYI I hit a similar issue and it was configuration-related:

[root@server-1]# grep advertised /usr/share/kafka/config/server.properties
advertised.host.name=192.168.100.2 <<<--- This address was wrong

After configuring the right value and restarting kafka, everything was OK.

[Raj Reddy (rajreddy)]

no additional information to proceed further.. we can reopen or open new PR if it re-occurs