setting a floating address on the state server prevents new agent connections

[Aaron Bentley]

I have a juju env where adding a subordinate charm relation doesn't work. It's stuck at:
        subordinates:
          jenkins-packages/0:
            agent-state: pending

Investigating the agent's log shows:
2013-11-06 18:25:39 INFO juju runner.go:253 worker: start "api"
2013-11-06 18:25:39 INFO juju apiclient.go:111 state/api: dialing "wss://162.213.34.53:17070/"
2013-11-06 18:26:42 ERROR juju apiclient.go:116 state/api: websocket.Dial wss://162.213.34.53:17070/: dial tcp 162.213.34.53:17070: connection timed out
2013-11-06 18:26:42 ERROR juju runner.go:211 worker: exited "api": websocket.Dial wss://162.213.34.53:17070/: dial tcp 162.213.34.53:17070: connection timed out
2013-11-06 18:26:42 INFO juju runner.go:245 worker: restarting "api" in 3s

The problem here is the agent is trying to use the stateserver's floating IP. If it used the fixed IP, it could connect just fine. Another agent (remember this is a subordinate charm) is happily using the fixed IP.

2013-11-01 18:52:26 INFO juju runner.go:253 worker: start "api"
2013-11-01 18:52:26 INFO juju apiclient.go:111 state/api: dialing "wss://10.55.60.105:17070/"
2013-11-01 18:52:26 INFO juju apiclient.go:121 state/api: connection established

This agent was started before the floating IP was assigned to the stateserver, so it continues to use it.

Because the agent cannot contact the state server, the service cannot be destroyed, similar to the problems described in bug #1089291 and Bug #1219902.

Full status:
$ juju status
environment: orange2
machines:
  "0":
    agent-state: started
    agent-version: 1.16.2
    dns-name: 162.213.34.53
    instance-id: 8e882718-4a09-4ae6-a771-ea7c0a83cb96
    instance-state: ACTIVE
    series: precise
    hardware: arch=amd64 cpu-cores=1 mem=2048M root-disk=10240M
  "1":
    agent-state: started
    agent-version: 1.16.2
    dns-name: 10.55.60.162
    instance-id: ab3f0c63-ff6b-4ca2-a91f-9a79275aebeb
    instance-state: ACTIVE
    series: precise
    hardware: arch=amd64 cpu-cores=1 mem=8192M root-disk=10240M
services:
  apache2:
    charm: cs:precise/apache2-12
    exposed: true
    relations:
      reverseproxy:
      - jenkins
    units:
      apache2/0:
        agent-state: started
        agent-version: 1.16.2
        machine: "0"
        open-ports:
        - 80/tcp
        - 443/tcp
        public-address: 162.213.34.53
  jenkins:
    charm: cs:precise/jenkins-8
    exposed: false
    relations:
      juju-info:
      - jenkins-packages
      website:
      - apache2
    units:
      jenkins/0:
        agent-state: started
        agent-version: 1.16.2
        machine: "1"
        open-ports:
        - 8080/tcp
        public-address: 10.55.60.162
        subordinates:
          jenkins-packages/0:
            agent-state: pending
  jenkins-packages:
    charm: cs:~abentley/precise/packages-3
    exposed: false
    relations:
      extension:
      - jenkins
    subordinate-to:
    - jenkins

[John A Meinel (jameinel)]

I wonder if this is fixed or made worse by the changes for bug #1205371

Juju now notices when IP addresses change, though it still should be trying to give a "Private" address for something like a charm.

[Curtis Hovey (sinzui)]

This bug is somewhat like the inverse of bug 928624 where the agent ip address changes.

[Anastasia (anastasia-macmood)]

@Aaron,
If you can reproduce this with Juju 2, could you please file a new bug against "juju" with reproducible scenario and newer logs?
We will not be fixing this for Juju 1.x series.