contrail-control in "initializing (No BGP configuration for self)" status after upgrade from R2.x

[wenqing liang]

After upgrade from R2.22.1-8 to R3.0.2.0-32 juno, contrail-control was stuck in "initializing (No BGP configuration for self)" status. The same is also seen after successful upgrades from R2.20, R2.21, R2.21.1, R2.22.

upgrade logs are uploaded to 10.84.5.112:/cs-shared/bugs/1574962.

root@cmbu-vse2100-1:/opt/contrail/utils# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 3.0.2.0-32 32
contrail-config 3.0.2.0-32 32
contrail-config-openstack 3.0.2.0-32 32
contrail-control 3.0.2.0-32 32
contrail-dns 3.0.2.0-32 32
contrail-docs 3.0.2.0-32 32
contrail-f5 3.0.2.0-32 32
contrail-fabric-utils 3.0.2.0-32 32
contrail-heat 3.0.2.0-32 32
contrail-install-packages 3.0.2.0-32~juno 32
contrail-lib 3.0.2.0-32 32
contrail-nodemgr 3.0.2.0-32 32
contrail-nova-networkapi 3.0.2.0-32 32
contrail-openstack 3.0.2.0-32 32
contrail-openstack-analytics 3.0.2.0-32 32
contrail-openstack-config 3.0.2.0-32 32
contrail-openstack-control 3.0.2.0-32 32
contrail-openstack-dashboard 3.0.2.0-32 32
contrail-openstack-database 3.0.2.0-32 32
contrail-openstack-webui 3.0.2.0-32 32
contrail-setup 3.0.2.0-32 32
contrail-utils 3.0.2.0-32 32
contrail-web-controller 3.0.2.0-32 32
contrail-web-core 3.0.2.0-32 32
ifmap-python-client 0.1-2 32
ifmap-server 0.3.2-1contrail1 32
neutron-plugin-contrail 3.0.2.0-32 32
nova-api 1:2014.2.3-0ubuntu1~cloud0.3contrail32
nova-common 1:2014.2.3-0ubuntu1~cloud0.3contrail32
nova-conductor 1:2014.2.3-0ubuntu1~cloud0.3contrail32
nova-console 1:2014.2.3-0ubuntu1~cloud0.3contrail32
nova-consoleauth 1:2014.2.3-0ubuntu1~cloud0.3contrail32
nova-novncproxy 1:2014.2.3-0ubuntu1~cloud0.3contrail32
nova-objectstore 1:2014.2.3-0ubuntu1~cloud0.3contrail32
nova-scheduler 1:2014.2.3-0ubuntu1~cloud0.3contrail32
python-contrail 3.0.2.0-32 32
python-neutronclient 1:2.3.8-0ubuntu1~cloud0.2contrail32
python-nova 1:2014.2.3-0ubuntu1~cloud0.3contrail32
root@cmbu-vse2100-1:/opt/contrail/utils# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control initializing (No BGP configuration for self)
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager active
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor active
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: active
supervisor-database: active
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

== Contrail Storage ==
contrail-storage-stats: active
root@cmbu-vse2100-1:/opt/contrail/utils#

Cluster nodes:

root@cmbu-vse2100-10:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing (No control-nodes configured)
contrail-vrouter-nodemgr active

== Contrail Storage ==
contrail-storage-stats: active
root@cmbu-vse2100-10:~#

root@cmbu-vse2100-11:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing (No control-nodes configured)
contrail-vrouter-nodemgr active

== Contrail Storage ==
contrail-storage-stats: active
root@cmbu-vse2100-11:~#

[Nischal Sheth (nsheth)]

@wliang

Can you provide access to a setup in this state?

[wenqing liang (wliang)]

@nsheth

Recreated. See cluster 10.87.129.173-175, root/n1keenA.

[Nischal Sheth (nsheth)]

Debugged with Sachin.
There's 2 problems:

1. There's no 'router_type' in router_params.

-> if not router_params['router_type']:
(Pdb) n
KeyError: ('router_type',)
> /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py(1491)update_bgp_router_type()
-> if not router_params['router_type']:
(Pdb) p router_params
{u'vendor': u'mx', u'port': 179, u'local_autonomous_system': None, u'auth_data': None, u'address': u'10.87.140.185', u'identifier': u'10.87.140.185', u'hold_time': 90, u'autonomous_system': 64512, u'address_families': {u'family': [u'route-target', u'inet-vpn', u'e-vpn', u'inet6-vpn']}}

2. set_context has not been called when we try to log an error for the above problem

*** AttributeError: 'greenlet.greenlet' object has no attribute 'api_context'
(Pdb) bt
  /usr/bin/contrail-api(9)<module>()
-> load_entry_point('vnc-cfg-api-server==0.1dev', 'console_scripts', 'contrail-api')()
  /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py(3194)server_main()
-> main()
  /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py(3153)main()
-> vnc_api_server = VncApiServer(args_str)
  /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py(1405)__init__()
-> self._db_init_entries()
  /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py(2498)_db_init_entries()
-> self._db_conn.db_resync()
  /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py(1354)db_resync()
-> self._cassandra_db.walk(self._dbe_resync)
  /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py(922)walk()
-> result = fn(obj_type, uuid_list)
  /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py(1534)_dbe_resync()
-> 'dbe_resync:cassandra_read', str(e))
  /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py(2000)config_object_error()
-> id, fq_name_str, obj_type, operation, err_str)
  /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py(2731)config_object_error()
-> self._set_api_audit_info(apiConfig)
  /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py(2743)_set_api_audit_info()
-> apiConfig.url = get_request().url
> /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/context.py(76)get_request()
-> def get_request():
(Pdb)

[Nischal Sheth (nsheth)]

There was no bgp-router object in CN's IFMap database.
Hence the inactive status in o/p of contrail-status.

API server showed 2 bgp-router objects but none in
ifmap-view output.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19823
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19824
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19823
Committed: http://github.org/Juniper/contrail-controller/commit/ed67eb2b3cddbd6ce64557bd4ced02658bc61c06
Submitter: Zuul
Branch: master

commit ed67eb2b3cddbd6ce64557bd4ced02658bc61c06
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon May 2 22:56:40 2016 +0000

Using config_log to log errors during dbe_resync as the api_context will not be
set in the greenlet at this point.

using get method to check for router_type in the router_params dictionary to
handle KeyError.

Change-Id: I1261065c401275df9812638ebb2a6c22e2c58513
Closes-Bug: 1574962

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19824
Committed: http://github.org/Juniper/contrail-controller/commit/c4ee3213a260d6664f6b182e7fae33af429d4f13
Submitter: Zuul
Branch: R3.0

commit c4ee3213a260d6664f6b182e7fae33af429d4f13
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon May 2 22:56:40 2016 +0000

Using config_log to log errors during dbe_resync as the api_context will not be
set in the greenlet at this point.

using get method to check for router_type in the router_params dictionary to
handle KeyError.

Change-Id: I1261065c401275df9812638ebb2a6c22e2c58513
Closes-Bug: 1574962