network-get returning fan address

[Francesco Banconi]

I am seeing this using 2.4-beta1-xenial-amd64.
Deployed postgres, related another app (blues-browser) to postgres, the app gets the fan address from the db relation, and therefore it's not able to communicate with the database.

Note that the blues-browser unit does not have a fan address and therefore is not able to ping the postgres one, maybe because its machine is trusty.

% juju run --unit postgresql/0 -- network-get db
bind-addresses:
- macaddress: 0e:78:23:52:0b:a5
  interfacename: fan-252
  addresses:
  - address: 252.0.64.1
    cidr: 252.0.0.0/8
egress-subnets:
- 252.0.64.1/32
ingress-addresses:
- 252.0.64.1

% juju run --unit postgresql/0 -- ip -4 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
    inet 10.132.0.4/32 brd 10.132.0.4 scope global ens4
       valid_lft forever preferred_lft forever
4: fan-252: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1410 qdisc noqueue state UP group default qlen 1000
    inet 252.0.64.1/8 scope global fan-252
       valid_lft forever preferred_lft forever

% juju run --unit blues-browser/0 -- ip -4 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
    inet 10.132.0.3/32 brd 10.132.0.3 scope global eth0
       valid_lft forever preferred_lft forever

% juju run --unit blues-browser/0 -- relation-get -r database:2 - postgresql/0
allowed-subnets: 10.132.0.3/32
allowed-units: blues-browser/0
database: blues-browser
egress-subnets: 252.0.64.1/32
host: 252.0.64.1
ingress-address: 252.0.64.1
master: dbname=blues-browser host=252.0.64.1 password=p9hHXqFYfM2mdcKXgXRS9LPxKrgrMgGhGJcz
  port=5432 user=juju_blues-browser
password: p9hHXqFYfM2mdcKXgXRS9LPxKrgrMgGhGJcz
port: "5432"
private-address: 252.0.64.1
schema_password: p9hHXqFYfM2mdcKXgXRS9LPxKrgrMgGhGJcz
schema_user: juju_blues-browser
state: standalone
user: juju_blues-browser
version: "9.5"

% juju status
Model Controller Cloud/Region Version SLA
default google google/europe-west1 2.4-beta1.1 unsupported

App Version Status Scale Charm Store Rev OS Notes
blues-browser unknown 1 blues-browser local 1 ubuntu
postgresql 9.5.12 active 1 postgresql jujucharms 174 ubuntu

Unit Workload Agent Machine Public address Ports Message
blues-browser/0* unknown idle 0 192.158.28.30
postgresql/0* active idle 1 35.195.79.188 5432/tcp Live master (9.5.12)

Machine State DNS Inst id Series AZ Message
0 started 192.158.28.30 juju-0c45ed-0 trusty europe-west1-b RUNNING
1 started 35.195.79.188 juju-0c45ed-1 xenial europe-west1-c RUNNING

Relation provider Requirer Interface Type Message
postgresql:coordinator postgresql:coordinator coordinator peer
postgresql:db blues-browser:database pgsql regular
postgresql:replication postgresql:replication pgpeer peer

[Anastasia (anastasia-macmood)]

Marking as Won't Fix for 2.3 series since we are not planning to make any further releases in this series at this stage.

[Przemyslaw Hausman (phausman)]

I'm experiencing `network-get` returning fan network IP address as well on vSphere substrate. I'd expect network-get always to return non-fan address.

$ juju --version
2.9-beta1-bionic-amd64

I'm deploying vault and mysql. Both in other vmware VMs. When I run `network-get shared-db` on mysql unit (deployed on VM1), it returns IP from primary network. But when I run `network-get shared-db` on vault unit (deployed on VM2), I get the IP address from the fan network.

As a result, vault unit stays blocked and reports "Error initializing storage of type mysql: failed to check mysql schema exist: Error 1130: Host '10.38.0.251' is not allowed to connect to this MySQL server". I checked the mysql.user table on mysql unit and in fact the value for 'Host' is incorrect. It holds the IP address from the fan network range, instead of the IP address of the "physical" NIC.

ubuntu@juju-7f9018-15:~$ journalctl -u vault
[...]
Sep 15 11:18:43 juju-7f9018-15 systemd[1]: Started HashiCorp Vault.
Sep 15 11:18:44 juju-7f9018-15 vault[8855]: Error initializing storage of type mysql: failed to check mysql schema exist: Error 1130: Host '10.38.0.251' is not allowed to connect to this MySQL server

ubuntu@infra-1:~/deploy$ juju run -m lma -u vault/0 'network-get shared-db --primary-address'
10.1.15.177

ubuntu@infra-1:~/deploy$ juju ssh -m lma vault/0 ip -o -4 a
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
2: ens192 inet 10.38.0.251/24 brd 10.38.0.255 scope global dynamic ens192\ valid_lft 464sec preferred_lft 464sec
2: ens192 inet 10.38.0.17/24 brd 10.38.0.255 scope global secondary ens192\ valid_lft forever preferred_lft forever
3: fan-10-1 inet 10.1.15.177/20 scope global fan-10-1\ valid_lft forever preferred_lft forever
5: lxdbr0 inet 10.112.216.1/24 scope global lxdbr0\ valid_lft forever preferred_lft forever
Connection to 10.38.0.251 closed.

Interestingly, on mysql unit, `network-get` returns correct IP address:

ubuntu@infra-1:~/deploy$ juju run -m lma -u mysql/0 'network-get shared-db --primary-address'
10.38.0.250

ubuntu@infra-1:~/deploy$ juju run -m lma -u mysql/0 'ip -o -4 a'
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
2: ens192 inet 10.38.0.250/24 brd 10.38.0.255 scope global dynamic ens192\ valid_lft 513sec preferred_lft 513sec
2: ens192 inet 10.38.0.16/24 brd 10.38.0.255 scope global secondary ens192\ valid_lft forever preferred_lft forever
3: fan-10-1 inet 10.1.15.161/20 scope global fan-10-1\ valid_lft forever preferred_lft forever

juju ssh -m lma mysql/0

ubuntu@juju-7f9018-21:~$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2715
Server version: 5.7.20-18-18-log Percona XtraDB Cluster (GPL), Release rel18, Revision e19a6b7, WSREP version 29.24, wsrep_29.24

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

mysql> select Host,User FROM mysql.user;
+---------------+...

[Przemyslaw Hausman (phausman)]

Subscribing field-critical as this blocks the customer deployment.

[John A Meinel (jameinel)]

For sites that make use of fan networking, you need to return the fan address for containers to be able to interoperate with the other machines. My memory is a little bit hazy on the exact details, but for 2 application to interoperate where one is in a container and one is on the host machine, they both should use fan addresses in order to communicate.

We should certainly be consistent here, and it is unclear why one machine might return a fan address but not the other.

You should be able to disable fan networking entirely with model config
  container-networking-method: local
  fan-config: ""

With that setting, you won't be able to put applications into containers and have them accessible to the rest of the model, but you shouldn't get fan addresses anywhere.

[Przemyslaw Hausman (phausman)]

Thank you, @jameinel. Let me remove fan networking altogether.
Unsubscribing field-critical as we have a workaround.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.