503 Service Unavailable after a postgresql failover, Landscape still trying to connect to an old postgres even after rendering a new config file
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Landscape Charm |
New
|
Undecided
|
Unassigned |
Bug Description
How to reproduce:
1. deploy the stable bundle
$ juju deploy landscape-scalable
Located bundle "landscape-
2. scale out postgresql to have 3 units for HA
$ juju add-unit postgresql -n 2
3. take down the primary postgresql unit by forcibly powering it off
Then, Landscape stops functioning unless I manually run `lsctl restart` to forcibly reload the config to the Landscape server processes.
$ curl -sLkv 192.168.151.106
...
< HTTP/1.1 503 Service Unavailable
< content-length: 107
< cache-control: no-cache
< content-type: text/html
<
{ [107 bytes data]
* Connection #1 to host 192.168.151.106 left intact
Aug 6 02:57:15 pingserver-2 CRIT Unhandled error in Deferred:
Aug 6 02:57:15 pingserver-2 CRIT #012Traceback (most recent call last):#012 File "/usr/lib/
[initial juju status]
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
landscape maas-controller maas/default 3.5.3 unsupported 02:44:44Z
App Version Status Scale Charm Channel Rev Exposed Message
haproxy active 1 haproxy latest/stable 75 yes Unit is ready
landscape-server active 1 landscape-server latest/stable 111 no Unit is ready
postgresql 14.11 active 3 postgresql 14/stable 429 no
rabbitmq-server 3.9.13 active 1 rabbitmq-server 3.9/stable 188 no Unit is ready
Unit Workload Agent Machine Public address Ports Message
haproxy/0* active idle 0 192.168.151.106 80,443/tcp Unit is ready
landscape-server/0* active idle 1 192.168.151.109 Unit is ready
postgresql/0 active idle 2 192.168.151.107 5432/tcp
postgresql/1 active idle 4 192.168.151.110 5432/tcp
postgresql/2* active idle 5 192.168.151.108 5432/tcp Primary
rabbitmq-server/0* active idle 3 192.168.151.111 5672,15672/tcp Unit is ready
Machine State Address Inst id Base AZ Message
0 started 192.168.151.106 machine-2 ubuntu@22.04 default Deployed
1 started 192.168.151.109 machine-3 ubuntu@22.04 default Deployed
2 started 192.168.151.107 machine-4 ubuntu@22.04 default Deployed
3 started 192.168.151.111 machine-5 ubuntu@22.04 default Deployed
4 started 192.168.151.110 machine-7 ubuntu@22.04 default Deployed
5 started 192.168.151.108 machine-6 ubuntu@22.04 default Deployed
[initial landscape config]
$ juju exec -u landscape-server/0 -- head -n 12 /etc/landscape/
[stores]
user = landscape
password = 7YyGp99aOAKBdkCG
host = 192.168.
main = landscape-
account-1 = landscape-
resource-1 = landscape-
package = landscape-
session = landscape-
session-autocommit = landscape-
knowledge = landscape-
^^^ 192.168.151.108 is written as the current primary postgresql host
[initial curl status]
$ curl -sLkv -o/dev/null 192.168.151.106
...
* Issue another request to this URL: 'https:/
* Found bundle for host 192.168.151.106: 0x63ba18e1a050 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#1) with host 192.168.151.106
* Connected to 192.168.151.106 (192.168.151.106) port 443 (#1)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
> GET /new-standalone
> Host: 192.168.151.106
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 Ok
< server: TwistedWeb/22.1.0
-> 200
[juju status after a postgresql failover]
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
landscape maas-controller maas/default 3.5.3 unsupported 02:52:45Z
App Version Status Scale Charm Channel Rev Exposed Message
haproxy active 1 haproxy latest/stable 75 yes Unit is ready
landscape-server active 1 landscape-server latest/stable 111 no Unit is ready
postgresql 14.11 active 2/3 postgresql 14/stable 429 no
rabbitmq-server 3.9.13 active 1 rabbitmq-server 3.9/stable 188 no Unit is ready
Unit Workload Agent Machine Public address Ports Message
haproxy/0* active idle 0 192.168.151.106 80,443/tcp Unit is ready
landscape-server/0* active idle 1 192.168.151.109 Unit is ready
postgresql/0* active idle 2 192.168.151.107 5432/tcp Primary
postgresql/1 active idle 4 192.168.151.110 5432/tcp
postgresql/2 unknown lost 5 192.168.151.108 5432/tcp agent lost, see 'juju show-status-log postgresql/2'
rabbitmq-server/0* active idle 3 192.168.151.111 5672,15672/tcp Unit is ready
Machine State Address Inst id Base AZ Message
0 started 192.168.151.106 machine-2 ubuntu@22.04 default Deployed
1 started 192.168.151.109 machine-3 ubuntu@22.04 default Deployed
2 started 192.168.151.107 machine-4 ubuntu@22.04 default Deployed
3 started 192.168.151.111 machine-5 ubuntu@22.04 default Deployed
4 started 192.168.151.110 machine-7 ubuntu@22.04 default Deployed
5 down 192.168.151.108 machine-6 ubuntu@22.04 default Deployed
[landscape config after the failover]
$ juju exec -u landscape-server/0 -- head -n 12 /etc/landscape/
[stores]
user = landscape
password = 7YyGp99aOAKBdkCG
host = 192.168.
main = landscape-
account-1 = landscape-
resource-1 = landscape-
package = landscape-
session = landscape-
session-autocommit = landscape-
knowledge = landscape-
^^^ a new host as 192.168.151.107 instead of the previous primary node is written by the charm (probably by the db-relation-changed hook)
But still Landscape server processes try to connect to the previous primary node.
> storm.database.
Subscribing ~field-high. HA of Landscape deployment doesn't work after a failover without human intervention.