network-get returns the vip as ingress address

[Ponnuvel Palaniyappan]

In a user deployment, we see that network-get returns the vip instead of the container's local address. In the environment, there's a ceilometer application with vip's set up and vip is no longer associated with the unit.

This is reported already in the nrpe charm: https://bugs.launchpad.net/charm-nrpe/+bug/1894175

But I believe this is in juju as the charm simpl gets the address from juju.

After some config changes, the nrpe charm runs network-get to get the container unit's primary address:

$ juju run --unit nrpe-container/28 "network-get monitors --format yaml"
bind-addresses:
- macaddress: 00:46:50:0e:b0:e1
  interfacename: eth0
  addresses:
  - hostname: ""
    address: 10.224.211.64
    cidr: 10.224.211.0/24
  - hostname: ""
    address: 10.224.211.241
    cidr: 10.224.211.0/24
egress-subnets:
- 10.224.211.241/32
ingress-addresses:
- 10.224.211.241
- 10.224.211.64

Here 10.224.211.241 is the vip address. 10.224.211.64 is the primary ip address. The vip isn't with the unit anymore The nrpe charm takes the first address listed under 'ingress-addresses' and thus picks up the wrong ip. Is juju not removing the vip from its db even after it's gone?

P.S: 'unit-get' returns the correct address:
$ juju run --unit nrpe-container/28 "unit-get private-address"
10.224.211.64

Please let me know if you need more information.

[Ponnuvel Palaniyappan (pponnuvel)]

Attached the nrpe unit's (lxc container) log.

[Joseph Phillips (manadart)]

1) What is the version of Juju running here?

2) What is the output from: juju run --unit nrpe-container/28 "ip a"

[Ponnuvel Palaniyappan (pponnuvel)]

Hi @Joseph,

1) Juju version 2.7.7

2) Output ip -o addr (from sosreport)

1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
1: lo inet6 ::1/128 scope host \ valid_lft forever preferred_lft forever
134: eth0 inet 10.224.211.64/24 brd 10.224.211.255 scope global eth0\ valid_lft forever preferred_lft forever
134: eth0 inet6 fe80::216:3eff:fec9:b23a/64 scope link \ valid_lft forever preferred_lft forever
136: eth1 inet 10.226.211.31/24 brd 10.226.211.255 scope global eth1\ valid_lft forever preferred_lft forever
136: eth1 inet6 fe80::216:3eff:fe72:969/64 scope link \ valid_lft forever preferred_lft forever
138: eth2 inet 10.227.211.40/24 brd 10.227.211.255 scope global eth2\ valid_lft forever preferred_lft forever
138: eth2 inet6 fe80::216:3eff:fe98:dcd7/64 scope link \ valid_lft forever preferred_lft forever

'ip a' isn't available in sosreport - if you'd like to have it, I can ask.

[Joseph Phillips (manadart)]

This patch is available in 2.8.3, which will remove the address from Juju:
https://github.com/juju/juju/pull/11853

We are also aware of https://bugs.launchpad.net/juju/+bug/1863916. A resolution for this remains undetermined, as the Go standard library does not recognise the flag in question.

[Ponnuvel Palaniyappan (pponnuvel)]

@Joseph, we have received another report from a different customer with this problem (vip reported as ingress) using Juju 2.8.3. I've asked for additional details and will update once I receive them.

[Ponnuvel Palaniyappan (pponnuvel)]

Further to my previous comment (#5): the problem I've encountered is in a deployment using network spaces [0] and that's not due to this bug but due to [1]. So disregard #5.

[0] https://juju.is/docs/spaces
[1] https://bugs.launchpad.net/juju/+bug/1863916

[Xav Paice (xavpaice)]

subscribed field-medium as this issue is affecting multiple production environments

[Zachary Zehring (zzehring)]

@Joseph, does this patch fix the issue for LXDs as well adding/removing interfaces? I experienced the same issue as stated in original description. Upgraded Juju from 2.7.8 to 2.8.7. However, I found that the VIP IPs still appear in network-interfaces using "juju show-machine" even on 2.8.7. This was after also restarting jujud-machine agents on both the host and the lxd.

[Joseph Phillips (manadart)]

The patch addresses the observed symptoms in the original report by removing IP addresses that are no longer observed on machines.

VIPs that still exist will be recorded against their machines when they are detected by the machine agent.

Can you confirm whether the address in your case is actually associated with the machine, or has it been removed and is being reported in error?

There is also an outstanding bug regarding secondary addresses:
https://bugs.launchpad.net/juju/+bug/1863916

[Joseph Phillips (manadart)]

This should be addressed by https://github.com/juju/juju/pull/13184, which ensures that the primary address is preferred for bind-addresses, ingress-addresses and as the basis for egress-subnets.