juju should use ip monitor or netlink to detect ip address changes instead of polling

[Kapil Thangavelu]

Currently juju does a 15m poll for address changes, this is undocumented, unintuitive and arbitrary to an end user, and will compromise our ability to do active network management in the future.

As an alternative using ip monitor or directly querying the kernel via netlink can give precise signaling when addresses changes. The ip monitor output is a bit volumnius, and needs verification, but as a signaling mechanism its pretty good.

$ sudo ip monitor │
[sudo] password for kapil: │
Deleted fe80::/64 dev wlan0 proto kernel metric 256 │
Deleted ff00::/8 dev wlan0 table local metric 256 │
delete ff02::2 dev wlan0 lladdr 33:33:00:00:00:02 NOARP │
delete ff02::16 dev wlan0 lladdr 33:33:00:00:00:16 NOARP │
delete ff02::1:ff7f:5cd0 dev wlan0 lladdr 33:33:ff:7f:5c:d0 NOARP │
delete ff02::fb dev wlan0 lladdr 33:33:00:00:00:fb NOARP │
Deleted 3: wlan0 inet6 fe80::a288:b4ff:fe7f:5cd0/64 scope link │
       valid_lft forever preferred_lft forever │
Deleted local fe80::a288:b4ff:fe7f:5cd0 dev lo table local proto none metric 0 │
ff00::/8 dev wlan0 table local metric 256 │
fe80::/64 dev wlan0 proto kernel metric 256 │
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 │
    link/ether a0:88:b4:7f:5c:d0 │
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DORMANT │
    link/ether a0:88:b4:7f:5c:d0 brd ff:ff:ff:ff:ff:ff │
Deleted default via 192.168.1.1 dev wlan0 proto static │
Deleted 192.168.1.0/24 dev wlan0 proto kernel scope link src 192.168.1.10 metric 9 │
Deleted broadcast 192.168.1.0 dev wlan0 table local proto kernel scope link src 192.168.1.10 │
Deleted local 192.168.1.10 dev wlan0 table local proto kernel scope host src 192.168.1.10 │
Deleted broadcast 192.168.1.255 dev wlan0 table local proto kernel scope link src 192.168.1.10 │
Deleted 3: wlan0 inet 192.168.1.10/24 brd 192.168.1.255 scope global wlan0 │
       valid_lft forever preferred_lft forever │
delete 255.255.255.255 dev wlan0 lladdr ff:ff:ff:ff:ff:ff NOARP │
delete 224.0.0.22 dev wlan0 lladdr 01:00:5e:00:00:16 NOARP │
delete 224.0.0.251 dev wlan0 lladdr 01:00:5e:00:00:fb NOARP │
delete 192.168.1.255 dev wlan0 lladdr ff:ff:ff:ff:ff:ff NOARP │
delete 192.168.1.1 dev wlan0 lladdr 00:26:62:8a:c9:59 REACHABLE │
3: wlan0: <BROADCAST,MULTICAST,UP> │
    link/ether │
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN │
    link/ether a0:88:b4:7f:5c:d0 brd ff:ff:ff:ff:ff:ff │
3: wlan0 inet6 fe80::a288:b4ff:fe7f:5cd0/64 scope link │
       valid_lft forever preferred_lft forever │
local fe80::a288:b4ff:fe7f:5cd0 dev lo table local proto none metric 0 │
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> │
    link/ether │
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> │
    link/ether

[Kapil Thangavelu (hazmat)]

considering we have several users hitting this (and the polling period is
entirely undocumented) i'd suggest bumping the priority a bit, else they
just conclude juju is broken.

On Mon, Dec 2, 2013 at 5:59 AM, Curtis Hovey <email address hidden> wrote:

> ** Changed in: juju-core
> Status: New => Triaged
>
> ** Changed in: juju-core
> Importance: Undecided => Low
>
> ** Tags added: improvement state-server
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1256053
>
> Title:
> juju should use ip monitor or netlink to detect ip address changes
> instead of polling
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1256053/+subscriptions
>

[Simon Davy (bloodearnest)]

Docker has a go wrapper of netlink api that might be useful:

https://github.com/dotcloud/docker/tree/master/pkg/netlink

[Thiago Bellini (bellini666)]

avoine pointed this bug to me on #juju when I was looking there for some help:

We have a production machine here using juju and lxc. The ip of the mysql and postgresql units changed, but the units related to them were not "informed" about that change. I mean, I forced some local charms to update and I could see in the log that they did a "relation-get" for the ip address, but it's still returning the old one.

Also, juju knows about the ip change since "juju status" returns me the correct public-address.

I could workaround that on some charms that doesn't override the "db config file" on every event. For some others, I had to fork the charm to "hardcode" the new ip (I don't need to say that this is unacceptable right?).

Like hazmat said above, juju is broken atm for me and many others. There's no workaround documented for this, nobody at #juju knows what to do (other than saying that "the fix is to destroy and recreate your environment" after I stated that I'm on a production machine). So, please, give this a high priority!

[Kapil Thangavelu (hazmat)]

Hi Thiago, the situation your describing is captured in bug: 1215579 (http://pad.lv/1215579) Namely the lack of address changes propagating to relations. For now the only real solution is to update the relations.. i'll see if i can work up a script for this and post it to both the bugs.

[Kapil Thangavelu (hazmat)]

Workaround posted to https://gist.github.com/kapilt/a61efcb4eaef9e685397