Intermittent sm provision failure on api down due to cassandra issue
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R4.0 |
Incomplete
|
High
|
wenqing liang | |||
R4.1 |
Incomplete
|
High
|
wenqing liang | |||
Trunk |
Incomplete
|
High
|
wenqing liang |
Bug Description
Seen in CB R16.04.2-107 newton, but not in following builds 109 and 111.
== Contrail Config ==
contrail-api: initializing (Collector, Database:
contrail-schema: initializing (ApiServer:
contrail-
contrail-
contrail-
== Contrail Config ==
contrail-api: initializing
contrail-schema: initializing (ApiServer:
contrail-
contrail-
contrail-
== Contrail Config Database==
contrail-database: active
In the debug.log we see:
"2017-11-08 21:14:42,
File "/usr/share/
main()
File "/usr/share/
AnalyticsNo
File "/usr/share/
fq_
File "/usr/lib/
return func(self, *args, **kwargs)
File "/usr/lib/
res_type, fq_name, fq_name_str, id, ifmap_id)
File "/usr/lib/
return (True, self.fq_
File "/usr/lib/
return func(self, *args, **kwargs)
File "/usr/lib/
raise he
When I run contrail-api in foreground It crashes with error:
Traceback (most recent call last):
File "/usr/bin/
load_
File "/usr/lib/
main(args_str, VncApiServer(
File "/usr/lib/
self.
File "/usr/lib/
config_
File "/usr/lib/
cass_uuid = self._db_
File "/usr/lib/
return func(*args, **kwargs)
File "/usr/lib/
raise VncError('Multi match %s for %s' % (fq_name_str, obj_type))
VncError: Multi match default-
root@servermana
+------
| id | status | ip_address | mac_address |
+------
| server2 | provision_completed | 10.0.0.5 | 02:A1:6D:72:65:B6 |
| server4 | provision_failed | 10.0.0.7 | 02:6D:CD:36:01:9A |
| server1 | provision_completed | 10.0.0.4 | 02:6F:55:2B:06:A4 |
| server6 | provision_failed | 10.0.0.9 | 02:62:63:CD:E1:51 |
| server9 | provision_failed | 10.0.0.12 | 02:B4:9F:B0:99:90 |
| server8 | provision_failed | 10.0.0.11 | 02:22:CA:66:15:86 |
| server10 | provision_failed | 10.0.0.13 | 02:B1:84:1B:0F:2F |
| server5 | provision_failed | 10.0.0.8 | 02:71:B2:B6:09:1B |
| server7 | provision_failed | 10.0.0.10 | 02:34:37:29:74:63 |
| server3 | provision_completed | 10.0.0.6 | 02:A9:9B:11:49:15 |
+------
root@servermana
/var/log/contrail/ and /var/log/cassandra/ from all three nodes are being uploaded to /cs-shared/bugs/.
Need the collect the /var/log/contrail/ and /var/log/cassandra/ from all three controller containers.
Please share the setup/above logs when we hit this issue again.