openstack-hypervisor snap cannot determine gateway

[Marian Gasparovic]

Similar to https://bugs.launchpad.net/snap-openstack/+bug/2025403

During sunbeam bootstrap openstack-hypervisor fails installing

2023-08-07 11:37:54 INFO unit.openstack-hypervisor/0.juju-log server.go:325 Running legacy hooks/start.
2023-08-07 11:37:55 INFO juju.worker.uniter.operation runhook.go:186 ran "start" hook (via hook dispatching script: dispatch)
2023-08-07 11:37:56 INFO juju.worker.uniter.operation runhook.go:186 ran "amqp-relation-joined" hook (via hook dispatching script: dispatch)
2023-08-07 11:37:58 INFO juju.worker.uniter.operation runhook.go:186 ran "amqp-relation-changed" hook (via hook dispatching script: dispatch)
2023-08-07 11:37:59 INFO unit.openstack-hypervisor/0.juju-log server.go:325 amqp:2: Entering guarded section: 'Bootstrapping'
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 error: cannot perform the following tasks:
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 - Run install hook of "openstack-hypervisor" snap if present (run hook "install":
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 -----
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 Traceback (most recent call last):
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 File "/snap/openstack-hypervisor/94/snap/hooks/install", line 9, in <module>
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 sys.exit(install(Snap()))
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 File "/snap/openstack-hypervisor/94/lib/python3.10/site-packages/openstack_hypervisor/hooks.py", line 214, in install
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 _update_default_config(snap)
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 File "/snap/openstack-hypervisor/94/lib/python3.10/site-packages/openstack_hypervisor/hooks.py", line 333, in _update_default_config
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 default = default()
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 File "/snap/openstack-hypervisor/94/lib/python3.10/site-packages/openstack_hypervisor/hooks.py", line 139, in _get_local_ip_by_default_rou
te
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 interface = gateways()["default"][AF_INET][1]
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 KeyError: 2
2023-08-07 11:38:20 WARNING unit.openstack-hypervisor/0.amqp-relation-changed logger.go:60 -----)
2023-08-07 11:38:20 ERROR unit.openstack-hypervisor/0.juju-log server.go:325 amqp:2: An exception occurred when installing charmcraft. Reason: Snap: 'openstack-hypervisor'; command ['snap', 'install', 'openstack-hypervisor', '--cha
nnel="2023.1/stable"'] failed with output = ''
2023-08-07 11:38:20 ERROR unit.openstack-hypervisor/0.juju-log server.go:325 amqp:2: Exception raised in section 'Bootstrapping': 2
2023-08-07 11:38:20 ERROR unit.openstack-hypervisor/0.juju-log server.go:325 amqp:2: Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-openstack-hypervisor-0/charm/venv/ops_sunbeam/guard.py", line 91, in guard
    yield
  File "/var/lib/juju/agents/unit-openstack-hypervisor-0/charm/venv/ops_sunbeam/charm.py", line 264, in configure_charm
    self.configure_unit(event)
  File "/var/lib/juju/agents/unit-openstack-hypervisor-0/charm/./src/charm.py", line 182, in configure_unit
    local_ip = _get_local_ip_by_default_route()
  File "/var/lib/juju/agents/unit-openstack-hypervisor-0/charm/./src/charm.py", line 52, in _get_local_ip_by_default_route
    interface = gateways()["default"][AF_INET][1]
KeyError: 2

2023-08-07 11:38:20 INFO juju.worker.uniter.operation runhook.go:186 ran "amqp-relation-changed" hook (via hook dispatching script: dispatch)
2023-08-07 11:39:25 INFO juju.worker.uniter.operation runhook.go:186 ran "ovsdb-cms-relation-joined" hook (via hook dispatching script: dispatch)
2023-08-07 11:39:26 INFO unit.openstack-hypervisor/0.juju-log server.go:325 ovsdb-cms:3: Entering guarded section: 'Bootstrapping'

ubuntu@cractus:~$ ip r
default via 10.245.128.1 dev enp134s0f0 proto static
10.1.10.0/23 dev enp96s0f0 proto kernel scope link src 10.1.11.150
blackhole 10.1.144.192/26 proto 80

[Hemanth Nakkina (hemanth-n)]

Fix - https://review.opendev.org/c/openstack/charm-openstack-hypervisor/+/890711

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-openstack-hypervisor/+/890711
Committed: https://opendev.org/openstack/charm-openstack-hypervisor/commit/fdf7fae8d5c6a0e36ee3833fc1d91c3ec27fdbbd
Submitter: "Zuul (22348)"
Branch: main

commit fdf7fae8d5c6a0e36ee3833fc1d91c3ec27fdbbd
Author: Hemanth Nakkina <email address hidden>
Date: Tue Aug 8 11:17:12 2023 +0530

    Use fallback for determining interface for local ip

    In some cases, the netifaces does not properly detect the default
    gateway for the node and returns an empty dict. However, there is
    a default gateway actually set. This adds a fallback path to parse
    the contents of the /proc/net/route table and determine which
    interface to use based on which flags are set on the routes.

    Closes-Bug: #2030497
    Change-Id: I1cf358ad42f0cec4733e3afc9b60f5ddfade2bfe

[Billy Olsen (billy-olsen)]

Patches proposed

https://github.com/openstack-snaps/snap-openstack-hypervisor/pull/30

and backport

https://github.com/openstack-snaps/snap-openstack-hypervisor/pull/31