svc-mon keeps crashing on startup with keyerror on instance_ip_address on a setup

[Vedamurthy Joshi]

R2.20 Build 40 Ubuntu 14.04 Juno multi-node setup

In this setup, somehow, there was an instance ip with no instance_ip_address key.

After upgrade to 2.20 Build 40, it was seen svc-mon keeps crashing due to this. This is even after i deleted any router objects and having no service instance objects.

Logs on the node running svc-mon will be in http://10.204.216.50/Docs/bugs/#

root@nodec1:/var/log/contrail# neutron port-list
+--------------------------------------+--------------------------------------+-------------------+---------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+--------------------------------------+-------------------+---------------------------------------------------------------------------------+
| 20cf6b21-2f3d-4b8f-b2b4-ed543f46c9e6 | 20cf6b21-2f3d-4b8f-b2b4-ed543f46c9e6 | 02:20:cf:6b:21:2f | {"subnet_id": "58d6bfd8-fd12-4d7b-8693-e6e178e0eca6", "ip_address": "10.1.1.3"} |
| 79f59dd7-9567-4891-98c2-d01cec2f49bc | 79f59dd7-9567-4891-98c2-d01cec2f49bc | 02:79:f5:9d:d7:95 | {"ip_address": null} <<<<< |
| 8e7825d1-8e79-4fd1-adc8-5db4487744e5 | 8e7825d1-8e79-4fd1-adc8-5db4487744e5 | 9a:35:6d:9b:e1:1e | {"subnet_id": "58d6bfd8-fd12-4d7b-8693-e6e178e0eca6", "ip_address": "10.1.1.4"} |
| e87b7e13-b4d0-42eb-9a6e-c761468366c5 | e87b7e13-b4d0-42eb-9a6e-c761468366c5 | 02:e8:7b:7e:13:b4 | {"ip_address": null} |
+--------------------------------------+--------------------------------------+-------------------+---------------------------------------------------------------------------------+
root@nodec1:/var/log/contrail#

instance_ip_address can be deleted on the instance-ip object when we update the instance-ip object(i.e. set_instance_ip_address() method) with address None

Traceback (most recent call last):
  File "/usr/bin/contrail-svc-monitor", line 9, in <module>
    load_entry_point('svc-monitor==0.1dev', 'console_scripts', 'contrail-svc-monitor')()
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 1042, in server_main
    main()
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 1037, in main
    run_svc_monitor, args)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/zkclient.py", line 281, in master_election
    self._election.run(self._zk_election_callback, func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/kazoo/recipe/election.py", line 48, in run
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/zkclient.py", line 273, in _zk_election_callback
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 1018, in run_svc_monitor
    monitor.post_init(vnc_api, args)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 331, in post_init
    self.sync_sm()
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 544, in sync_sm
    InstanceIpSM.locate(uuid)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_db.py", line 55, in locate
    cls._dict[key] = cls(key, *args)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/config_db.py", line 497, in __init__
    self.update(obj_dict)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/config_db.py", line 505, in update
    self.address = obj['instance_ip_address']
KeyError: 'instance_ip_address'

env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6, host7],
    'cfgm': [host1,host2,host3],
    'openstack': [host1,host2,host3],
    'control': [host1,host2,host3],
    'compute': [host4,host5, host6, host7],
    'collector': [host1,host2,host3],
    'webui': [host1],
    'database': [host1,host2,host3],
    'toragent': [host6, host7],
    'tsn': [host6, host7],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodec1', 'nodec2', 'nodec3', 'nodek1', 'nodek2', 'nodek3', 'nodeg11']
}

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11379
Submitter: Rudra Rugge (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11380
Submitter: Rudra Rugge (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11380
Committed: http://github.org/Juniper/contrail-controller/commit/c88769730db3a9772952c1ca6bafaa261d008644
Submitter: Zuul
Branch: R2.20

commit c88769730db3a9772952c1ca6bafaa261d008644
Author: Rudra Rugge <email address hidden>
Date: Mon Jun 8 11:12:54 2015 -0700

Add check for address in instance-ip

Ensure that address exists in instance-ip object. This is to
overcome upgrade issues where we hit an old bug if missing address
in an instance-ip.

Change-Id: Ic578895175dbbc5e0b31584c92267ac4eed3a295
Closes-Bug: #1462666

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11379
Committed: http://github.org/Juniper/contrail-controller/commit/2acd5d72f45f2524e66ac06d37c0263f7e17bf44
Submitter: Zuul
Branch: master

commit 2acd5d72f45f2524e66ac06d37c0263f7e17bf44
Author: Rudra Rugge <email address hidden>
Date: Mon Jun 8 11:12:54 2015 -0700

Add check for address in instance-ip

Ensure that address exists in instance-ip object. This is to
overcome upgrade issues where we hit an old bug if missing address
in an instance-ip.

Change-Id: Ic578895175dbbc5e0b31584c92267ac4eed3a295
Closes-Bug: #1462666

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11434
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11434
Committed: http://github.org/Juniper/contrail-packages/commit/f79cae08d9878aeb16f26c6ea2a68f08d0ef6bef
Submitter: Zuul
Branch: master

commit f79cae08d9878aeb16f26c6ea2a68f08d0ef6bef
Author: Ananth Suryanarayana <email address hidden>
Date: Tue Jun 9 15:53:34 2015 -0700

Closes-Bug: #1462666 Add control-user:control-user-passwd information to default basicauthusers.properties

By default, ifmap authentication information has all necessary contrail users except control-node

Change-Id: I7406bdd50a5d95398fa114684c15fbf610a844fd