Intermittent Traceback during upgrade-charm hook

Bug #1988617 reported by Frode Nordahl
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
charm-ovn-central
New
Undecided
Unassigned

Bug Description

2022-09-01 07:29:06 INFO juju.worker.uniter.charm bundles.go:78 downloading ch:amd64/jammy/ovn-central-37 from API server
2022-09-01 07:29:07 INFO juju.worker.uniter resolver.go:149 found queued "upgrade-charm" hook
2022-09-01 07:30:27 INFO unit.ovn-central/0.juju-log server.go:316 Reactive main running for hook upgrade-charm
2022-09-01 07:30:27 INFO unit.ovn-central/0.juju-log server.go:316 Initializing Snap Layer
2022-09-01 07:30:27 INFO unit.ovn-central/0.juju-log server.go:316 Initializing Leadership Layer (is leader)
2022-09-01 07:30:27 INFO unit.ovn-central/0.juju-log server.go:316 Invoking reactive handler: reactive/snap.py:117:upgrade_charm
2022-09-01 07:30:27 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 All snaps up to date.
2022-09-01 07:30:27 INFO unit.ovn-central/0.juju-log server.go:316 Invoking reactive handler: reactive/layer_openstack.py:46:default_upgrade_charm
2022-09-01 07:30:27 INFO unit.ovn-central/0.juju-log server.go:316 Invoking reactive handler: reactive/layer_openstack.py:34:default_config_changed
2022-09-01 07:30:27 INFO unit.ovn-central/0.juju-log server.go:316 Making dir /etc/ovn root:root 555
2022-09-01 07:30:28 INFO unit.ovn-central/0.juju-log server.go:316 Invoking reactive handler: reactive/layer_openstack.py:56:run_default_upgrade_charm
2022-09-01 07:30:33 INFO unit.ovn-central/0.juju-log server.go:316 Invoking reactive handler: reactive/layer_openstack.py:126:default_request_certificates
2022-09-01 07:30:36 WARNING unit.ovn-central/0.juju-log server.go:316 No network binding for internal
2022-09-01 07:30:37 WARNING unit.ovn-central/0.juju-log server.go:316 No network binding for admin
2022-09-01 07:30:37 WARNING unit.ovn-central/0.juju-log server.go:316 No network binding for public
2022-09-01 07:30:37 WARNING unit.ovn-central/0.juju-log server.go:316 No network binding for internal
2022-09-01 07:30:37 WARNING unit.ovn-central/0.juju-log server.go:316 No network binding for admin
2022-09-01 07:30:37 WARNING unit.ovn-central/0.juju-log server.go:316 No network binding for public
2022-09-01 07:30:37 WARNING unit.ovn-central/0.juju-log server.go:316 Skipping request for certificate for ip in internal space, no local address found
2022-09-01 07:30:37 WARNING unit.ovn-central/0.juju-log server.go:316 Skipping request for certificate for ip in admin space, no local address found
2022-09-01 07:30:37 WARNING unit.ovn-central/0.juju-log server.go:316 Skipping request for certificate for ip in public space, no local address found
2022-09-01 07:30:37 INFO unit.ovn-central/0.juju-log server.go:316 Invoking reactive handler: reactive/ovn_central_handlers.py:112:configure_firewall
2022-09-01 07:30:37 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping adding existing rule\nSkipping adding existing rule (v6)\n'
2022-09-01 07:30:37 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping adding existing rule\nSkipping adding existing rule (v6)\n'
2022-09-01 07:30:38 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping adding existing rule\nSkipping adding existing rule (v6)\n'
2022-09-01 07:30:38 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping adding existing rule\nSkipping adding existing rule (v6)\n'
2022-09-01 07:30:38 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping inserting existing rule\n'
2022-09-01 07:30:38 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping inserting existing rule\n'
2022-09-01 07:30:38 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping inserting existing rule\n'
2022-09-01 07:30:39 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping inserting existing rule\n'
2022-09-01 07:30:39 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping inserting existing rule\n'
2022-09-01 07:30:39 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping inserting existing rule\n'
2022-09-01 07:30:39 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping inserting existing rule\n'
2022-09-01 07:30:39 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping inserting existing rule\n'
2022-09-01 07:30:40 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping inserting existing rule\n'
2022-09-01 07:30:40 INFO unit.ovn-central/0.juju-log server.go:316 b'Skipping inserting existing rule\n'
2022-09-01 07:30:40 INFO unit.ovn-central/0.juju-log server.go:316 Invoking reactive handler: reactive/ovn_central_handlers.py:132:publish_addr_to_clients
2022-09-01 07:30:40 INFO unit.ovn-central/0.juju-log server.go:316 Invoking reactive handler: reactive/ovn_central_handlers.py:158:render
2022-09-01 07:30:40 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 Removed /etc/systemd/system/ovn-central.service.
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 Removed /etc/systemd/system/ovn-ovsdb-server-nb.service.
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 Removed /etc/systemd/system/ovn-ovsdb-server-sb.service.
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 2022-09-01T07:30:41Z|00001|jsonrpc|WARN|unix:/var/run/ovn/ovnsb_db.sock: receive error: Connection reset by peer
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 2022-09-01T07:30:41Z|00002|reconnect|WARN|unix:/var/run/ovn/ovnsb_db.sock: connection dropped (Connection reset by peer)
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 ovn-sbctl: unix:/var/run/ovn/ovnsb_db.sock: database connection failed (Connection reset by peer)
2022-09-01 07:30:41 ERROR unit.ovn-central/0.juju-log server.go:316 Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-ovn-central-0/charm/reactive/ovn_central_handlers.py", line 196, in render
    ovn_charm.configure_ovn(
  File "/var/lib/juju/agents/unit-ovn-central-0/charm/lib/charm/openstack/ovn_central.py", line 608, in configure_ovn
    self.configure_ovn_listener(
  File "/var/lib/juju/agents/unit-ovn-central-0/charm/lib/charm/openstack/ovn_central.py", line 468, in configure_ovn_listener
    for connection in connections.find(
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/network/ovs/ovsdb.py", line 230, in _find_tbl
    output = utils._run(*cmd)
  File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/network/ovs/utils.py", line 26, in _run
    return subprocess.check_output(args, universal_newlines=True)
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('ovn-sbctl', '-f', 'json', 'find', 'connection', 'target="pssl:16642"')' returned non-zero exit status 1.

2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 Traceback (most recent call last):
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 File "/var/lib/juju/agents/unit-ovn-central-0/charm/hooks/upgrade-charm", line 22, in <module>
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 main()
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 bus.dispatch(restricted=restricted_mode)
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 _invoke(other_handlers)
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 handler.invoke()
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 self._action(*args)
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 File "/var/lib/juju/agents/unit-ovn-central-0/charm/reactive/ovn_central_handlers.py", line 196, in render
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 ovn_charm.configure_ovn(
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 File "/var/lib/juju/agents/unit-ovn-central-0/charm/lib/charm/openstack/ovn_central.py", line 608, in configure_ovn
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 self.configure_ovn_listener(
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 File "/var/lib/juju/agents/unit-ovn-central-0/charm/lib/charm/openstack/ovn_central.py", line 468, in configure_ovn_listener
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 for connection in connections.find(
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/network/ovs/ovsdb.py", line 230, in _find_tbl
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 output = utils._run(*cmd)
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 File "/var/lib/juju/agents/unit-ovn-central-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/network/ovs/utils.py", line 26, in _run
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 return subprocess.check_output(args, universal_newlines=True)
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 File "/usr/lib/python3.10/subprocess.py", line 524, in run
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 raise CalledProcessError(retcode, process.args,
2022-09-01 07:30:41 WARNING unit.ovn-central/0.upgrade-charm logger.go:60 subprocess.CalledProcessError: Command '('ovn-sbctl', '-f', 'json', 'find', 'connection', 'target="pssl:16642"')' returned non-zero exit status 1.
2022-09-01 07:30:42 ERROR juju.worker.uniter.operation runhook.go:140 hook "upgrade-charm" (via explicit, bespoke hook script) failed: exit status 1

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.