MySQL InnoDB Cluster Charm

[master] update-status can error out due to attribute error in error handling code

Bug #2015368 reported by Alex Kavanagh on 2023-04-05

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	MySQL InnoDB Cluster Charm	Fix Committed	Undecided	Alex Kavanagh

Bug Description

Essentially, the error handling code during the update-status handling breaks if it doesn't get stderr output:

github.com/juju/juju/rpc.init:14: connection is shut down
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api.(*state).APICall:1252:
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 Traceback (most recent call last):
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 1202, in get_cluster_status
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 self.wait_until_cluster_available()
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 339, in wrapped_f
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 return self(f, *args, **kw)
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 430, in __call__
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 do = self.iter(retry_state=retry_state)
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 378, in iter
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 raise retry_exc.reraise()
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 206, in reraise
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 raise self.last_attempt.result()
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 return self.__get_result()
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 raise self._exception
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 433, in __call__
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 result = fn(*args, **kwargs)
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 1752, in wait_until_cluster_available
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 self.cluster_user, self.cluster_password, self.cluster_address,
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 334, in cluster_address
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 return self.options.cluster_address
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 84, in cluster_address
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 return ch_net_ip.get_relation_ip("cluster")
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/network/ip.py", line 571, in get_relation_ip
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 address = network_get_primary_address(interface)
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 1180, in inner_translate_exc2
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 return f(*args, **kwargs)
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 1374, in network_get_primary_address
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 response = subprocess.check_output(
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/usr/lib/python3.10/subprocess.py", line 524, in run
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 raise CalledProcessError(retcode, process.args,
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 subprocess.CalledProcessError: Command '['network-get', '--primary-address', 'cluster']' returned non-zero exit status 1.
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 During handling of the above exception, another exception occurred:
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 Traceback (most recent call last):
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/hooks/update-status", line 22, in <module>
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 main()
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 84, in main
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 hookenv._run_atexit()
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 1357, in _run_atexit
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 callback(*args, **kwargs)
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 1378, in atexit_assess_status
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 self._assess_status()
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 1633, in _assess_status
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 _cluster_status = self.get_cluster_status_summary(nocache=True)
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 1266, in get_cluster_status_summary
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 _status = self.get_cluster_status(nocache=nocache)
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 File "/var/lib/juju/agents/unit-mysql-innodb-cluster-0/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 1206, in get_cluster_status
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 .format(e.stderr.decode("UTF-8")), "ERROR")
2023-04-05 02:20:30 WARNING unit.mysql-innodb-cluster/0.update-status logger.go:60 AttributeError: 'NoneType' object has no attribute 'decode'
2023-04-05 02:20:30 ERROR juju.worker.uniter.operation runhook.go:140 hook "update-status" (via explicit, bespoke hook script) failed: exit status 1

Fix is to handle the issue properly.

Tags:

Alex Kavanagh (ajkavanagh) on 2023-04-05

tags:

added: good-first-bug

Alex Kavanagh (ajkavanagh) on 2023-05-22

Changed in charm-mysql-innodb-cluster:
assignee:	nobody → Alex Kavanagh (ajkavanagh)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-05-22: Fix proposed to charm-mysql-innodb-cluster (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/883839

Changed in charm-mysql-innodb-cluster:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-05-26: Fix merged to charm-mysql-innodb-cluster (master)

Reviewed: https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/883839
Committed: https://opendev.org/openstack/charm-mysql-innodb-cluster/commit/4a71dcbaaca786d168bf849b03832339e64a717d
Submitter: "Zuul (22348)"
Branch: master

commit 4a71dcbaaca786d168bf849b03832339e64a717d
Author: Alex Kavanagh <email address hidden>
Date: Mon May 22 14:10:13 2023 +0100

Improve handling of errors in update status hook

The update status hook can error during logging on cluster status. This
just prevents it from going into error during update-status.

Change-Id: Ife6235bab82bb828c943ba1215bf938238d4b257
Closes-Bug: #2015368

Changed in charm-mysql-innodb-cluster:
status:	In Progress → Fix Committed

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1897628

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.