haproxy failed: unable to find required use_backend

[Andreas Hasenack]

I deployed two units of landscape-server (lp:landscape-charm). I then removed the first one:

juju destroy-unit landscape-server/0

haproxy failed:
2015-05-25 19:50:49 INFO unit.haproxy/0.juju-log cmd.go:247 reverseproxy:33: Unit 'landscape-server/1' overrides 'services', skipping further processing.
2015-05-25 19:50:49 INFO unit.haproxy/0.juju-log cmd.go:247 reverseproxy:33: Service: landscape-https
2015-05-25 19:50:49 INFO unit.haproxy/0.juju-log cmd.go:247 reverseproxy:33: Service: landscape-http
2015-05-25 19:50:49 INFO unit.haproxy/0.juju-log cmd.go:247 reverseproxy:33: Using self-signed certificate
2015-05-25 19:50:49 INFO unit.haproxy/0.juju-log cmd.go:247 reverseproxy:33: Required config not found or invalid (metrics_target, metrics_sample_interval), disabling metrics
2015-05-25 19:50:49 INFO unit.haproxy/0.reverseproxy-relation-departed logger.go:40 [ALERT] 144/195049 (3041) : Proxy 'haproxy-0-443': unable to find required use_backend: 'landscape-package-upload'.
2015-05-25 19:50:49 INFO unit.haproxy/0.reverseproxy-relation-departed logger.go:40 [ALERT] 144/195049 (3041) : Fatal errors found in configuration.
2015-05-25 19:50:49 INFO unit.haproxy/0.juju-log cmd.go:247 reverseproxy:33: HAProxy configuration check failed, exiting.
2015-05-25 19:50:49 ERROR juju.worker.uniter.operation runhook.go:89 hook "reverseproxy-relation-departed" failed: exit status 1

In juju status:
    units:
      haproxy/0:
        agent-state: error
        agent-state-info: 'hook failed: "reverseproxy-relation-departed" for landscape-server:website'

haproxy.cfg does mention landscape-package-upload in its frontend section, but there is indeed no such backend. That server was in the landscape-server/0 unit which was destroyed. It should have been moved to landscape-server/1 (new "leader"), or else should have been removed from the frontend section too:

frontend haproxy-0-443
    bind 0.0.0.0:443 ssl crt /var/lib/haproxy/default.pem
    default_backend landscape-https
    mode http
    http-request set-header X-Forwarded-Proto https
    acl message path_beg -i /message-system
    acl api path_beg -i /api
    acl package-upload path_beg -i /upload
    use_backend landscape-message if message
    use_backend landscape-api if api
    use_backend landscape-package-upload if package-upload

but no such backend indeed:
# grep package-upload /etc/haproxy/haproxy.cfg
    acl package-upload path_beg -i /upload
    use_backend landscape-package-upload if package-upload

[Free Ekanayaka (free.ekanayaka)]

At first sight it seems indeed a leader change problem. I think we should take the chance to move to juju builtin leader election, that will offer the primitives and hooks (leader-changed) to handle this properly.