Juniper Openstack

R3.2-build-12-setup-all failed also casandra failed to come up if cfgm and database role are different.

Bug #1608683 reported by shajuvk on 2016-08-01

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R3.1	Fix Committed	Critical	Ignatious Johnson Christopher	Juniper Openstack r3.1.0.0-fcs
Trunk	Fix Committed	Critical	Ignatious Johnson Christopher	Juniper Openstack r3.2.0.0-fcs "r3.2.0.0"

Bug Description

if the cfgm and database roles are not the same node, database not comimg up. also seeing below exception part of setup_all.
===
mail chat:
=========

Ok, I have the check for it https://github.com/Juniper/contrail-fabric-utils/blob/master/fabfile/utils/host.py#L396,

I am checking for same nodes, in this testbed topology, config is subset of database nodes.

I will submit a patch now.

Shaju, to unblock now, you can set same set of nodes for config and database as below,

‘cfgm’ : [host1, host2, host3],
'database': [host1, host2, host3],

Thanks,
Ignatious

On 8/1/16, 10:24 AM, "Shaju V.K" <email address hidden> wrote:

Hi Ignatious,

Below is the roledef and logs: Please let me know if it is supported ?

env.roledefs = {
    'all': [host1, host2, host3, host4, contrail_vm1, contrail_vm2],
    'cfgm': [host1, host2],
    'openstack': [host1],
    'control': [host1, host3],
    'compute': [host4, contrail_vm1, contrail_vm2],
    'vcenter_compute':[host3],
    'collector': [host1],
    'webui': [host1],
    'database': [host1, host2, host3],
    'build': [host_build],
}

DEBUG [SSTableBatchOpen:1] 2016-08-01 01:25:55,928 SSTableReader.java:459 - Opening /var/lib/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/la-5-big (5770 bytes)
INFO [main] 2016-08-01 01:25:55,934 ColumnFamilyStore.java:382 - Initializing system.available_ranges
ERROR [main] 2016-08-01 01:25:56,103 CassandraDaemon.java:638 - Fatal exception during initialization
org.apache.cassandra.exceptions.ConfigurationException: Saved cluster name ContrailConfigDB != configured name Contrail
        at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:736) ~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.service.StartupChecks$9.execute(StartupChecks.java:306) ~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:107) ~[apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:162) [apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [apache-cassandra-2.2.5.jar:2.2.5]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625) [apache-cassandra-2.2.5.jar:2.2.5]

Thanks,
Shaju

On 8/1/16, 10:23 AM, "Megh Bhatt" <email address hidden> wrote:

Hi Ignatious,
Shaju had a testbed.py where 2 nodes had database and config and one node had only database.

The provisioning fails since the configured cluster name Contrail is not same as saved cluster name ContrailConfigDB.

We should explicitly add check in fabric to not support such a configuration/testbed.py

Thanks

Megh

=========
2016-08-01 13:41:28:677775: *args, **kwargs
2016-08-01 13:41:28:680160: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 375, in execute
2016-08-01 13:41:28:680189: multiprocessing
2016-08-01 13:41:28:680381: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 271, in _execute
2016-08-01 13:41:28:680402: return task.run(*args, **kwargs)
2016-08-01 13:41:28:680493: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2016-08-01 13:41:28:680542: return self.wrapped(*args, **kwargs)
2016-08-01 13:41:28:680619: File "/usr/local/lib/python2.7/dist-packages/fabric/decorators.py", line 53, in inner_decorator
2016-08-01 13:41:28:680648: return func(*args, **kwargs)
2016-08-01 13:41:28:680782: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 168, in __call__
2016-08-01 13:41:28:680802: return self.run(*args, **kwargs)
2016-08-01 13:41:28:680851: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2016-08-01 13:41:28:680867: return self.wrapped(*args, **kwargs)
2016-08-01 13:41:28:680914: File "/opt/contrail/utils/fabfile/tasks/provision.py", line 2479, in setup_all
2016-08-01 13:41:28:680930: execute('setup_cfgm')
2016-08-01 13:41:28:681887: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 375, in execute
2016-08-01 13:41:28:681910: multiprocessing
2016-08-01 13:41:28:681985: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 271, in _execute
2016-08-01 13:41:28:682001: return task.run(*args, **kwargs)
2016-08-01 13:41:28:682058: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2016-08-01 13:41:28:682074: return self.wrapped(*args, **kwargs)
2016-08-01 13:41:28:682121: File "/usr/local/lib/python2.7/dist-packages/fabric/decorators.py", line 53, in inner_decorator
2016-08-01 13:41:28:682136: return func(*args, **kwargs)
2016-08-01 13:41:28:682171: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 168, in __call__
2016-08-01 13:41:28:682187: return self.run(*args, **kwargs)
2016-08-01 13:41:28:682232: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2016-08-01 13:41:28:682247: return self.wrapped(*args, **kwargs)
2016-08-01 13:41:28:682292: File "/opt/contrail/utils/fabfile/tasks/provision.py", line 55, in setup_cfgm
2016-08-01 13:41:28:682308: execute("setup_cfgm_node", env.host_string)
2016-08-01 13:41:28:682343: File "/usr/local/lib/python2.7/dist-packages/fabric/t
2016-08-01 13:41:28:677534: Disconnecting from 10.84.13.32... done.
2016-08-01 13:41:28:796945: Disconnecting from 10.84.13.33... done.
2016-08-01 13:41:28:910904: Disconnecting from 10.84.13.38... done.
2016-08-01 13:41:28:942763: asks.py", line 415, in execute
2016-08-01 13:41:28:682359: results['<local-only>'] = task.run(*args, **new_kwargs)
2016-08-01 13:41:28:682455: File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 171, in run
2016-08-01 13:41:28:682472: return self.wrapped(*args, **kwargs)
2016-08-01 13:41:28:682517: File "/opt/contrail/utils/fabfile/tasks/provision.py", line 601, in setup_cfgm_node
2016-08-01 13:41:28:682533: cmd = frame_vnc_config_cmd(host_string)
2016-08-01 13:41:28:682625: File "/opt/contrail/utils/fabfile/utils/commandline.py", line 174, in frame_vnc_config_cmd
2016-08-01 13:41:28:682642: cmd += " --seed_list %s" % ','.join(database_ip_list[:2])
2016-08-01 13:41:28:682918: NameError: global name 'database_ip_list' is not defined

See original description

Tags:

shajuvk (shajuvk) on 2016-08-01

summary:	- R3.2-build-12-setup-all failed also casandra failed to come up with - diffent cfgm and database role + R3.2-build-12-setup-all failed also casandra failed to come up if cfgm + and database role are different.
description:	updated

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-01: [Review update] master

Review in progress for https://review.opencontrail.org/22732
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-01: [Review update] R3.1

Review in progress for https://review.opencontrail.org/22736
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-01: A change has been merged

Reviewed: https://review.opencontrail.org/22736
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/f0d87ac48d4fc5632340d0286a2197c8d35427b8
Submitter: Zuul
Branch: R3.1

commit f0d87ac48d4fc5632340d0286a2197c8d35427b8
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Aug 1 14:50:24 2016 -0700

Check for cfgm/database nodes and make sure cfgm is not subset of
database nodes to decide config_db, also passing the correct ip
address list for the seed list.

Change-Id: I76834166d956cf0839d3f8335e731f3ee9ca8658
Closes-Bug: 1608683

shajuvk (shajuvk) on 2016-08-01

information type:

Proprietary → Public

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-02: [Review update] R3.1

Review in progress for https://review.opencontrail.org/22746
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-02: [Review update] master

Review in progress for https://review.opencontrail.org/22732
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-02: A change has been merged

#10

Reviewed: https://review.opencontrail.org/22746
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/6bcfb57b504f306adc0465bed38cffe6c14dffaf
Submitter: Zuul
Branch: R3.1

commit 6bcfb57b504f306adc0465bed38cffe6c14dffaf
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Aug 1 17:29:21 2016 -0700

Check for cfgm/database nodes and make sure cfgm is not subset of
database nodes to decide config_db, also passing the correct ip
address list for the seed list.

Closes-Bug: 1608683

Change-Id: Iff83b9eb185882a0bac171733cc041a39d7e1d34

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-08-03:

#11

Reviewed: https://review.opencontrail.org/22732
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/c6d822cbe40cb2b71572ebcb7ebeeca5e5a699da
Submitter: Zuul
Branch: master

commit c6d822cbe40cb2b71572ebcb7ebeeca5e5a699da
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Aug 1 14:50:24 2016 -0700

Check for cfgm/database nodes and make sure cfgm is not subset of
database nodes to decide config_db, also passing the correct ip
address list for the seed list.

Change-Id: I76834166d956cf0839d3f8335e731f3ee9ca8658
Closes-Bug: 1608683

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1608113

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.