Cassandra int-test Failing for Trove

Bug #1314980 reported by Aniket Jeewankar on 2014-05-01
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Low
Petr Malik

Bug Description

Please follow below steps to reproduce the issue.

1. Fetched code from the git location as below
git clone https://github.com/openstack/trove.git
git clone https://github.com/openstack/trove-integration.git

2. Ran following commands to install datastore with Trove
./redstack install
./redstack kick-start cassandra
sed -i 's/10.0.0.1/10.1.0.1/g' /opt/stack/trove/etc/trove/trove-guestagent.conf.sample

3. Ran int-tests with below command
./redstack int-tests --group=cassandra

The int-tests are failing in trove-create test case. The detailed logs are attached for the reference.

Investigating on the shut down machine, logged in with ssh

/tmp/logfile.txt (as guest agent was running on the box) logged the following error:

from (pid=1304) execute /home/ubuntu/trove/trove/openstack/common/processutils.py:162
2014-05-01 13:18:44.131 ERROR trove.guestagent.datastore.cassandra.service [req-694f6da0-7493-4d2a-9568-ddd9e8bb43d0 0080251f7580483792684c62bcfffe87 51bf377e5995402cbaf1aad813fa24b0] Pr
ocess execution Unexpected error while running command.
Command: echo "use system;" > /tmp/check; cqlsh -f /tmp/check
Exit code: 1
Stdout: ''
Stderr: 'Connection error: Could not connect to localhost:9160\n'

description: updated
Changed in trove:
importance: Undecided → Medium
milestone: none → ongoing
Denis M. (dmakogon) wrote :

It seems that it's not a bug, because cassandra service requires pretty much time to start.
Also, we cannot talk about cassandra datastore without showing cassandra log file

Denis M. (dmakogon) wrote :

Other thing. Given https://bugs.launchpad.net/trove/+bug/1314980/+attachment/4102294/+files/cassandra_inttest_log.txt shows that int-test are pretty tight to mysql datastore.

Denis M. (dmakogon) wrote :

Even more. Int-tests expecting to have datastore in BUILDING, ACTIVE state for successful assertion, but for cassandra it doesn't work, because for some short period of time instance stays in SHUTDOWN state (that is being described by bug-report).

troveclient.compat.client: DEBUG: RESP:{'date': 'Tue, 10 Jun 2014 15:21:02 GMT', 'status': '200', 'content-length': '769', 'content-type': 'application/json', 'content-location': u'http://172.18.196.62:8779/v1.0/531983aa109042a59a1b7e550186b9e2/instances/0f856f88-a6d2-403a-a916-090c822b7c6a'} {"instance": {"status": "SHUTDOWN", "updated": "2014-06-10T15:06:24", "name": "TEST_2014-06-10 08:06:19.421363", "links": [{"href": "https://172.18.196.62:8779/v1.0/531983aa109042a59a1b7e550186b9e2/instances/0f856f88-a6d2-403a-a916-090c822b7c6a", "rel": "self"}, {"href": "https://172.18.196.62:8779/instances/0f856f88-a6d2-403a-a916-090c822b7c6a", "rel": "bookmark"}], "created": "2014-06-10T15:06:19", "ip": ["10.0.0.2"], "id": "0f856f88-a6d2-403a-a916-090c822b7c6a", "volume": {"used": 0.03, "size": 1}, "flavor": {"id": "7", "links": [{"href": "https://172.18.196.62:8779/v1.0/531983aa109042a59a1b7e550186b9e2/flavors/7", "rel": "self"}, {"href": "https://172.18.196.62:8779/flavors/7", "rel": "bookmark"}]}, "datastore": {"version": "2.0.6", "type": "cassandra"}}}

trove.common.utils: ERROR: in looping call
Traceback (most recent call last):
  File "/opt/stack/trove/trove/common/utils.py", line 213, in _inner
    self.f(*self.args, **self.kw)
  File "/opt/stack/trove/trove/common/utils.py", line 250, in poll_and_check
    obj = retriever()
  File "/opt/stack/trove/trove/tests/api/instances.py", line 753, in result_is_active
    assert_equal("BUILD", instance.status)
  File "/usr/local/lib/python2.7/dist-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: BUILD != SHUTDOWN
--------------------- >> end captured logging << ---------------------

As the first iteration, i forsee that int tests should be fixed in next way:

at https://github.com/openstack/trove/blob/master/trove/tests/api/instances.py#L742-L767

instead of:
assert_equal("BUILD", instance.status)
use:
assert_true(instance.status not in ["FAILED", "ERROR"])

Changed in trove:
assignee: nobody → Denis M. (dmakogon)
status: New → In Progress
Denis M. (dmakogon) wrote :

After re-iterating given issue was made decision to use:
assert_true(instance.status not in ["BUILD", "SHUTDOWN"])

Denis M. (dmakogon) wrote :

assert_true(instance.status in ["BUILD", "SHUTDOWN"])

Denis M. (dmakogon) wrote :

It would be useful to use log changes as the way to define if C* is starting or not

ubuntu@cassandra-7:~$ date; md5sum /var/log/cassandra/system.log
Tue Jul 8 12:53:46 UTC 2014
88a5658e2c4ac3c120c625b905a6c86b /var/log/cassandra/system.log
ubuntu@cassandra-7:~$ date; md5sum /var/log/cassandra/system.log
Tue Jul 8 12:53:48 UTC 2014
0d6c3d04cfae99705542564ab609f5f2 /var/log/cassandra/system.log
ubuntu@cassandra-7:~$ date; md5sum /var/log/cassandra/system.log
Tue Jul 8 12:53:49 UTC 2014
c8ee21f7bd50eea9263f87a1b5890ad8 /var/log/cassandra/system.log
ubuntu@cassandra-7:~$ date; md5sum /var/log/cassandra/system.log
Tue Jul 8 12:53:49 UTC 2014
2ad85a5e6b75bed59233a03958cbae95 /var/log/cassandra/system.log
ubuntu@cassandra-7:~$ date; md5sum /var/log/cassandra/system.log
Tue Jul 8 12:53:50 UTC 2014
cac5ff441da4ae73fdbd44aafc99c755 /var/log/cassandra/system.log
ubuntu@cassandra-7:~$ date; md5sum /var/log/cassandra/system.log
Tue Jul 8 12:53:51 UTC 2014
cac5ff441da4ae73fdbd44aafc99c755 /var/log/cassandra/system.log
ubuntu@cassandra-7:~$ date; md5sum /var/log/cassandra/system.log
Tue Jul 8 12:53:52 UTC 2014
cac5ff441da4ae73fdbd44aafc99c755 /var/log/cassandra/system.log
ubuntu@cassandra-7:~$ date; md5sum /var/log/cassandra/system.log
Tue Jul 8 12:53:53 UTC 2014
c2649b0af13a2bcec9e8ecc5496cd522 /var/log/cassandra/system.log

Fix proposed to branch: master
Review: https://review.openstack.org/107100

Change abandoned by Denis M. (<email address hidden>) on branch: master
Review: https://review.openstack.org/99424

Amrith Kumar (amrith) wrote :

Updating importance to "Low" (bugscrub, 2014-10-23)

Changed in trove:
importance: Medium → Low
Denis M. (dmakogon) wrote :

I was able to fix this problem, here's log.

Change abandoned by Denis M. (<email address hidden>) on branch: master
Review: https://review.openstack.org/107100

Doug Shelley (0-doug) on 2015-03-03
Changed in trove:
assignee: Denis M. (dmakogon) → Petr Malik (pmalik)
Petr Malik (pmalik) wrote :

The issue no longer exists in the new scenario-tests infrastructure.

Changed in trove:
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers