horizon unit blocked on healthcheck failure

[Hemanth Nakkina]

Deployed sunbeam edge channel and horizon unit is in blocked state

juju status shows:

horizon waiting 1 horizon-k8s 2023.1/edge 46 10.152.183.110 no installing agent

horizon/0* blocked idle 10.1.173.141 (container:horizon) healthcheck failed: online

Looking into the logs:

1. sudo microk8s.kubectl -n openstack logs horizon-0 -c horizon

2023-06-27T12:18:03.528Z [wsgi-horizon] 127.0.0.1 - - [27/Jun/2023:12:17:32 +0000] "GET /openstack-horizon/auth/login/ HTTP/1.1" 200 3728 "-" "Go-http-client/1.1"
2023-06-27T12:18:10.458Z [pebble] Check "online" failure 5 (threshold 3): Get "http://localhost:80/openstack-horizon/auth/login/": context deadline exceeded
2023-06-27T12:18:20.487Z [pebble] Check "online" failure 6 (threshold 3): Get "http://localhost:80/openstack-horizon/auth/login/": context deadline exceeded
2023-06-27T12:18:24.431Z [wsgi-horizon] 127.0.0.1 - - [27/Jun/2023:12:17:42 +0000] "GET /openstack-horizon/auth/login/ HTTP/1.1" 200 3738 "-" "Go-http-client/1.1"
2023-06-27T12:18:28.692Z [wsgi-horizon] 127.0.0.1 - - [27/Jun/2023:12:18:02 +0000] "GET /openstack-horizon/auth/login/ HTTP/1.1" 200 3729 "-" "Go-http-client/1.1"
2023-06-27T12:18:30.453Z [pebble] Check "online" failure 7 (threshold 3): Get "http://localhost:80/openstack-horizon/auth/login/": context deadline exceeded
2023-06-27T12:18:33.468Z [wsgi-horizon] 127.0.0.1 - - [27/Jun/2023:12:17:52 +0000] "GET /openstack-horizon/auth/login/ HTTP/1.1" 200 3731 "-" "Go-http-client/1.1"
2023-06-27T12:18:36.431Z [wsgi-horizon] 127.0.0.1 - - [27/Jun/2023:12:18:12 +0000] "GET /openstack-horizon/auth/login/ HTTP/1.1" 200 3729 "-" "Go-http-client/1.1"
2023-06-27T12:18:40.453Z [pebble] Check "online" failure 8 (threshold 3): Get "http://localhost:80/openstack-horizon/auth/login/": context deadline exceeded

2. root@horizon-0:/# charm/bin/pebble checks
Check Level Status Failures
online ready down 4743/3
up alive up 0/3

Seems like 8s timeout for pebble ready check is not sufficient enough (takes around 9s for http call on my VM - 16GB/4vcpu ).

time curl http://localhost:80/openstack-horizon/auth/login/
...
real 0m8.995s
user 0m0.004s
sys 0m0.015s

[Hemanth Nakkina (hemanth-n)]

PRs for review:
https://review.opendev.org/c/openstack/charm-ops-sunbeam/+/887129
https://review.opendev.org/c/openstack/charm-horizon-k8s/+/887130