hook failed: "database-admin-relation-joined" when installing cqlsh on the same machine as Cassandra host

Bug #1872534 reported by Vladimir Grevtsev on 2020-04-13
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cassandra Juju Charm

Bug Description

$ juju status -m cassandra
Model Controller Cloud/Region Version SLA Timestamp
cassandra orangebox-cloud-RegionOne orangebox-cloud/RegionOne 2.6.10 unsupported 19:38:16Z

App Version Status Scale Charm Store Rev OS Notes
cassandra 3.11.6 error 1 cassandra jujucharms 54 ubuntu
cqlsh 3.11.6 waiting 1 cqlsh jujucharms 2 ubuntu

Unit Workload Agent Machine Public address Ports Message
cassandra/1* error idle 1 9042/tcp,9160/tcp hook failed: "database-admin-relation-joined"
cqlsh/0* waiting idle 1 Waiting for Cassandra

Machine State DNS Inst id Series AZ Message
1 started f27460d9-71c2-4fc9-8911-3366de6821fc bionic nova ACTIVE

unit-cassandra-1: 17:44:34 INFO unit.cassandra/1.juju-log database-admin:4: Publishing credentials for cqlsh (database-admin:4)
unit-cassandra-1: 17:44:34 DEBUG worker.uniter.jujuc running hook tool "leader-get"
unit-cassandra-1: 17:44:34 DEBUG worker.uniter.jujuc running hook tool "leader-get"
unit-cassandra-1: 17:45:35 DEBUG worker.uniter.jujuc running hook tool "juju-log"
unit-cassandra-1: 17:45:35 ERROR unit.cassandra/1.juju-log database-admin:4: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cassandra-1/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main
  File "/var/lib/juju/agents/unit-cassandra-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
  File "/var/lib/juju/agents/unit-cassandra-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
  File "/var/lib/juju/agents/unit-cassandra-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
  File "/var/lib/juju/agents/unit-cassandra-1/charm/reactive/cassandra/client.py", line 55, in publish
    publish_credentials(rel, True)
  File "/var/lib/juju/agents/unit-cassandra-1/charm/reactive/cassandra/client.py", line 88, in publish_credentials
    with cassandra.connect() as session:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "lib/charms/layer/cassandra.py", line 698, in connect
    session = cluster.connect()
  File "/var/lib/juju/agents/unit-cassandra-1/.venv/lib/python3.6/site-packages/cassandra/cluster.py", line 1335, in connect
  File "/var/lib/juju/agents/unit-cassandra-1/.venv/lib/python3.6/site-packages/cassandra/cluster.py", line 2863, in connect
  File "/var/lib/juju/agents/unit-cassandra-1/.venv/lib/python3.6/site-packages/cassandra/cluster.py", line 2906, in _reconnect_internal
    raise NoHostAvailable("Unable to connect to any servers", errors)
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'': ConnectionRefusedError(111, "Tried connecting to [('', 9042)]. Last error: Connection refused")})

Before cqlsh charm deployment, Cassandra unit was in "Live seed" idle/active status.
full unit log: http://paste.ubuntu.com/p/rjhBRtMCKq/

Revision history for this message
Stuart Bishop (stub) wrote :

It isn't supported and rarely recommended to install multiple charms into the same machine (hulk smash, smoosh). In this case, I think the cqlsh charms destroyed the credentials stored in the cassandra user's home directory. I think you need to install cqlsh into an lxd container if you need to share the same hardware.

Changed in cassandra-charm:
status: New → Won't Fix
Revision history for this message
Vladimir Grevtsev (vlgrevtsev) wrote :

Hi Stuart,

I believe, the statement "It isn't supported and rarely recommended to install multiple charms into the same machine" is un-obvious from the operator point of view, so that might lead to the an unexpected consequences like in this issue. What do you think about having a cqlsh charm as a subordinate, to improve user experience and avoid dedicated machine allocation (either lxd or cloud VM) to host a single CLI tool?

Revision history for this message
Stuart Bishop (stub) wrote :

The cassandra charm already ends up with cqlsh installed and available, so a subordinate does not seem necessary. The cs:~cassandra-charmers/cqlshrc charm was primarily written to test the cassandra charm, and available as it might be useful to provide CLI access to a user who should not have access to the cassandra machine itself.

A suboprdinate would also need to be careful to not conflict, eg. both attempting to own the cqlshrc files in ~root. But if you have a use case I can't see why not. A subordinate would only need to handle the relation and write out authentication credentials, as the cassandra tools would already be installed. A subordinate would only get connection details to the local node due to the different relation scope, but that isn't really a problem with Cassandra as the other nodes would be discovered by the client (assuming the local Cassandra server is running).

$ juju ssh cassandra/9
Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-96-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

  System information as of Tue Apr 28 10:57:37 UTC 2020

  System load: 0.01 Processes: 89
  Usage of /: 20.7% of 9.52GB Users logged in: 0
  Memory usage: 77% IP address for ens2:
  Swap usage: 0%

0 packages can be updated.
0 updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@juju-e60653-default-29:~$ cqlsh
Connected to juju at
[cqlsh 5.0.1 | Cassandra 3.11.6 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers