This is on Juju 2.9.44
Panics in the instance-poller and the firewaller, but no stack trace is visible. The end result is that the workers are restarting and trying to carry on, only to be bounced again. The problem does eventually fix itself, but we shouldn't be panicing.
Logs from around the panic.
---- instance-poller
2024-05-08 00:22:35 WARNING juju.provider.openstack networking.go:391 failed to get list of networks
caused by: Resource at https://neutron.ps5.canonical.com:9696/networks?router%3Aexternal=false not found
caused by: request (https://neutron.ps5.canonical.com:9696/networks?router%3Aexternal=false) returned unexpected status: 404; error info: {"code": 404, "title": "Not Found", "description": null}
To resolve this error, set a value for "network" in model-config or model-defaults;
or supply it via --config when creating a new model
2024-05-08 00:22:35 ERROR juju.worker.dependency engine.go:695 "instance-poller" manifold worker returned unexpected error: panic resulted in: runtime error: invalid memory address or nil pointer dereference
2024-05-08 00:22:35 WARNING juju.core.raftlease client.go:134 response timeout waiting for Command(ver: 1, op: extend, ns: application-leadership, model: c060b6, lease: autocert-lpqateam, holder: autocert-lpqateam/0) to be processed
2024-05-08 00:22:35 WARNING juju.core.raftlease client.go:134 response timeout waiting for Command(ver: 1, op: extend, ns: application-leadership, model: 84b28a, lease: assets-apache2, holder: assets-apache2/0) to be processed
2024-05-08 00:22:35 WARNING juju.core.raftlease client.go:134 response timeout waiting for Command(ver: 1, op: extend, ns: application-leadership, model: f81502, lease: ubuntu-syncproxy-ports, holder: ubuntu-syncproxy-ports/1) to be processed
2024-05-08 00:22:35 WARNING juju.worker.httpserver log.go:194 http: TLS handshake error from 10.132.54.215:46750: EOF
----- firewaller
2024-05-07 17:40:59 ERROR juju.worker.dependency engine.go:695 "firewaller" manifold worker returned unexpected error: cannot respond to units changes for "machine-3", "f8c78212-28d2-4176-8693-0d5ed0c4ed95": failed to list security groups
caused by: Resource at https://neutron.ps5.canonical.com:9696/security-groups not found
caused by: request (https://neutron.ps5.canonical.com:9696/security-groups) returned unexpected status: 404; error info: {"code": 404, "title": "Not Found", "description": null}
2024-05-07 17:40:59 WARNING juju.provider.openstack networking.go:391 failed to get list of networks
caused by: Resource at https://neutron.ps5.canonical.com:9696/networks?router%3Aexternal=false not found
caused by: request (https://neutron.ps5.canonical.com:9696/networks?router%3Aexternal=false) returned unexpected status: 404; error info: {"code": 404, "title": "Not Found", "description": null}
To resolve this error, set a value for "network" in model-config or model-defaults;
or supply it via --config when creating a new model
2024-05-07 17:40:59 ERROR juju.worker.dependency engine.go:695 "instance-poller" manifold worker returned unexpected error: enumerating network interface list for instances: listing subnets: failed to retrieve subnets: failed to get list of subnets
caused by: Resource at https://neutron.ps5.canonical.com:9696/subnets not found
caused by: request (https://neutron.ps5.canonical.com:9696/subnets) returned unexpected status: 404; error info: {"code": 404, "title": "Not Found", "description": null}
2024-05-07 17:41:00 ERROR juju.worker.dependency engine.go:695 "firewaller" manifold worker returned unexpected error: failed to list security groups
caused by: Resource at https://neutron.ps5.canonical.com:9696/security-groups not found
caused by: request (https://neutron.ps5.canonical.com:9696/security-groups) returned unexpected status: 404; error info: {"code": 404, "title": "Not Found", "description": null}
2024-05-07 17:41:01 INFO juju.api apiclient.go:687 connection established to "wss://10.131.3.230:17070/model/823b1bbf-ce75-42a9-8f5a-94cd96768b73/api"
2024-05-07 17:41:04 WARNING juju.worker.httpserver log.go:194 http: TLS handshake error from 127.0.0.1:44868: EOF
2024-05-07 17:41:09 WARNING juju.provider.openstack networking.go:391 failed to get list of networks
caused by: Resource at https://neutron.ps5.canonical.com:9696/networks?router%3Aexternal=false not found
caused by: request (https://neutron.ps5.canonical.com:9696/networks?router%3Aexternal=false) returned unexpected status: 404; error info: {"code": 404, "title": "Not Found", "description": null}
To resolve this error, set a value for "network" in model-config or model-defaults;
or supply it via --config when creating a new model
2024-05-07 17:41:09 ERROR juju.worker.dependency engine.go:695 "instance-poller" manifold worker returned unexpected error: panic resulted in: runtime error: invalid memory address or nil pointer dereference
2024-05-07 17:41:10 ERROR juju.worker.dependency engine.go:695 "instance-poller" manifold worker returned unexpected error: panic resulted in: runtime error: invalid memory address or nil pointer dereference
2024-05-07 17:41:12 WARNING juju.worker.httpserver log.go:194 http: TLS handshake error from 127.0.0.1:53152: EOF
2024-05-07 17:41:16 INFO juju.api apiclient.go:687 connection established to "wss://10.131.3.230:17070/model/823b1bbf-ce75-42a9-8f5a-94cd96768b73/api"
2024-05-07 17:41:19 ERROR juju.worker.dependency engine.go:695 "firewaller" manifold worker returned unexpected error: panic resulted in: runtime error: invalid memory address or nil pointer dereference
2024-05-07 17:41:19 ERROR juju.worker.dependency engine.go:695 "firewaller" manifold worker returned unexpected error: cannot respond to units changes for "machine-4", "8506a73b-5de4-4c58-8a00-8878dbc0aa96": failed to list security groups
caused by: Resource at https://neutron.ps5.canonical.com:9696/security-groups not found
caused by: request (https://neutron.ps5.canonical.com:9696/security-groups) returned unexpected status: 404; error info: {"code": 404, "title": "Not Found", "description": null}
2024-05-07 17:41:22 WARNING juju.worker.httpserver log.go:194 http: TLS handshake error from 127.0.0.1:60318: EOF
2024-05-07 17:41:24 ERROR juju.worker.dependency engine.go:695 "instance-poller" manifold worker returned unexpected error: panic resulted in: runtime error: invalid memory address or nil pointer dereference
2024-05-07 17:41:25 ERROR juju.worker.dependency engine.go:695 "firewaller" manifold worker returned unexpected error: cannot respond to units changes for "machine-0", "b45c47c9-2dc1-41ac-8e15-2707ddf81ee8": failed to list security groups
caused by: Resource at https://neutron.ps5.canonical.com:9696/security-groups not found
caused by: request (https://neutron.ps5.canonical.com:9696/security-groups) returned unexpected status: 404; error info: {"code": 404, "title": "Not Found", "description": null}
2024-05-07 17:41:30 WARNING juju.worker.httpserver log.go:194 http: TLS handshake error from 127.0.0.1:60334: EOF
2024-05-07 17:41:31 INFO juju.api apiclient.go:687 connection established to "wss://10.131.3.171:17070/model/823b1bbf-ce75-42a9-8f5a-94cd96768b73/api"
2024-05-07 17:41:39 WARNING juju.provider.openstack networking.go:391 failed to get list of networks
caused by: Resource at https://neutron.ps5.canonical.com:9696/networks?router%3Aexternal=false not found
caused by: request (https://neutron.ps5.canonical.com:9696/networks?router%3Aexternal=false) returned unexpected status: 404; error info: {"code": 404, "title": "Not Found", "description": null}
To resolve this error, set a value for "network" in model-config or model-defaults;
or supply it via --config when creating a new model
2024-05-07 17:41:39 ERROR juju.worker.dependency engine.go:695 "instance-poller" manifold worker returned unexpected error: panic resulted in: runtime error: invalid memory address or nil pointer dereference
2024-05-07 17:41:40 WARNING juju.provider.openstack networking.go:391 failed to get list of networks
caused by: Resource at https://neutron.ps5.canonical.com:9696/networks?router%3Aexternal=false not found
caused by: request (https://neutron.ps5.canonical.com:9696/networks?router%3Aexternal=false) returned unexpected status: 404; error info: {"code": 404, "title": "Not Found", "description": null}
To resolve this error, set a value for "network" in model-config or model-defaults;
or supply it via --config when creating a new model
2024-05-07 17:41:40 ERROR juju.worker.dependency engine.go:695 "instance-poller" manifold worker returned unexpected error: enumerating network interface list for instances: listing subnets: failed to retrieve subnets: failed to get list of subnets
caused by: Resource at https://neutron.ps5.canonical.com:9696/subnets not found
caused by: request (https://neutron.ps5.canonical.com:9696/subnets) returned unexpected status: 404; error info: {"code": 404, "title": "Not Found", "description": null}
2024-05-07 17:41:40 ERROR juju.worker.dependency engine.go:695 "firewaller" manifold worker returned unexpected error: cannot respond to units changes for "machine-0", "83f493a7-4ade-4c4a-833c-4e240170c7a1": failed to list security groups
caused by: Resource at https://neutron.ps5.canonical.com:9696/security-groups not found
caused by: request (https://neutron.ps5.canonical.com:9696/security-groups) returned unexpected status: 404; error info: {"code": 404, "title": "Not Found", "description": null}
2024-05-07 17:41:42 INFO juju.apiserver.connection request_notifier.go:96 agent login: unit-assets-ubuntu-com-0 for b5e53ea2-8df4-4137-8dce-9b8e033d3c1a
2024-05-07 17:41:43 WARNING juju.worker.httpserver log.go:194 http: TLS handshake error from 127.0.0.1:45396: EOF
I spent an hour trying to understand where the panic could come from. Although I found some issues, I can't through just reading the code find where the common path between the instance-poller and firewaller is dereferencing a nil pointer.