network-get returns wrong ip for cluster_address when rendering the etcd.conf template

Bug #1947034 reported by Andrew Scribner
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Etcd Charm
New
Undecided
Unassigned

Bug Description

In some cases where a node has a mix of ipv4 and ipv6 addresses, etcd may pull the wrong one. This shows in `juju status` as:

```
etcd 3.4.5 active 1 etcd charmstore stable 607 ubuntu Errored with 0 known peers
```

with:
```
$ juju debug-log -i etcd/0 --replay
unit-etcd-0: 14:34:16 WARNING unit.etcd/0.update-status Error: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: connect: connection refused
unit-etcd-0: 14:34:16 WARNING unit.etcd/0.update-status ; error #1: dial tcp 127.0.0.1:2379: connect: connection refused
unit-etcd-0: 14:34:16 WARNING unit.etcd/0.update-status
unit-etcd-0: 14:34:16 WARNING unit.etcd/0.update-status error #0: dial tcp 127.0.0.1:4001: connect: connection refused
unit-etcd-0: 14:34:16 WARNING unit.etcd/0.update-status error #1: dial tcp 127.0.0.1:2379: connect: connection refused
unit-etcd-0: 14:34:16 WARNING unit.etcd/0.update-status
unit-etcd-0: 14:34:16 ERROR unit.etcd/0.juju-log ['/snap/bin/etcd.etcdctl', 'cluster-health']
unit-etcd-0: 14:34:16 ERROR unit.etcd/0.juju-log {'ETCDCTL_API': '2', 'ETCDCTL_CA_FILE': '/var/snap/etcd/common/ca.crt', 'ETCDCTL_CERT_FILE': '/var/snap/etcd/common/server.crt', 'ETCDCTL_KEY_FILE': '/var/snap/etcd/common/server.key'}
unit-etcd-0: 14:34:16 ERROR unit.etcd/0.juju-log b'cluster may be unhealthy: failed to list members\n'
unit-etcd-0: 14:34:16 ERROR unit.etcd/0.juju-log None
unit-etcd-0: 14:34:16 WARNING unit.etcd/0.juju-log Notice: Unit failed cluster-health check
unit-etcd-0: 14:34:17 WARNING unit.etcd/0.update-status Error: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: connect: connection refused
unit-etcd-0: 14:34:17 WARNING unit.etcd/0.update-status ; error #1: dial tcp 127.0.0.1:4001: connect: connection refused
unit-etcd-0: 14:34:17 WARNING unit.etcd/0.update-status
unit-etcd-0: 14:34:17 WARNING unit.etcd/0.update-status error #0: dial tcp 127.0.0.1:2379: connect: connection refused
unit-etcd-0: 14:34:17 WARNING unit.etcd/0.update-status error #1: dial tcp 127.0.0.1:4001: connect: connection refused
unit-etcd-0: 14:34:17 WARNING unit.etcd/0.update-status
unit-etcd-0: 14:34:17 ERROR unit.etcd/0.juju-log ['/snap/bin/etcd.etcdctl', 'member', 'list']
unit-etcd-0: 14:34:17 ERROR unit.etcd/0.juju-log {'ETCDCTL_API': '2', 'ETCDCTL_CA_FILE': '/var/snap/etcd/common/ca.crt', 'ETCDCTL_CERT_FILE': '/var/snap/etcd/common/server.crt', 'ETCDCTL_KEY_FILE': '/var/snap/etcd/common/server.key'}
unit-etcd-0: 14:34:17 ERROR unit.etcd/0.juju-log b''
unit-etcd-0: 14:34:17 ERROR unit.etcd/0.juju-log None
```

and (from the unit)

```
$ journalctl -u snap.etcd.etcd.service
Oct 13 18:39:42 frank-moose systemd[1]: snap.etcd.etcd.service: Failed with result 'exit-code'.
Oct 13 18:39:52 frank-moose systemd[1]: snap.etcd.etcd.service: Scheduled restart job, restart counter is at 1023.
Oct 13 18:39:52 frank-moose systemd[1]: Stopped Service for snap application etcd.etcd.
Oct 13 18:39:52 frank-moose systemd[1]: Started Service for snap application etcd.etcd.
Oct 13 18:39:52 frank-moose etcd.etcd[93173]: Running as system with data in /var/snap/etcd/230
Oct 13 18:39:52 frank-moose etcd.etcd[93173]: Configuration from /var/snap/etcd/common/etcd.conf.yml
Oct 13 18:39:52 frank-moose etcd.etcd[93173]: unexpected error setting up initial-advertise-peer-urls: URL address does not have the form "host:port": https://2607:fea8:c3a1:8f00:216:3eff>
Oct 13 18:39:52 frank-moose systemd[1]: snap.etcd.etcd.service: Main process exited, code=exited, status=1/FAILURE
Oct 13 18:39:52 frank-moose systemd[1]: snap.etcd.etcd.service: Failed with result 'exit-code'.
```

where we see an ipv6 address being used. This is pulled from the top of ingress-addresses
```
$ juju run --unit etcd/0 'network-get cluster'
bind-addresses:
- mac-address: 00:16:3e:02:69:96
  interface-name: eth2
  addresses:
  - hostname: ""
    address: 2607:fea8:c3a1:8f00:216:3eff:fe02:6996
    cidr: 2607:fea8:c3a1:8f00::/64
  - hostname: ""
    address: 2607:fea8:c3a1:8f00::fc1d
    cidr: 2607:fea8:c3a1:8f00::fc1d/128
  - hostname: ""
    address: 10.0.0.42
    cidr: 10.0.0.0/24
  macaddress: 00:16:3e:02:69:96
  interfacename: eth2
egress-subnets:
- 2607:fea8:c3a1:8f00:216:3eff:fe02:6996/128
ingress-addresses:
- 2607:fea8:c3a1:8f00:216:3eff:fe02:6996
- 2607:fea8:c3a1:8f00::fc1d
- 10.0.0.42
```

where [this](https://github.com/charmed-kubernetes/layer-etcd/blob/master/lib/etcd_lib.py#L21) line pulls all addresses and [this](https://github.com/charmed-kubernetes/layer-etcd/blob/master/lib/etcd_lib.py#L29) line uses only the first one (which in this case was ipv6).

Revision history for this message
Andrew Scribner (ca-scribner) wrote :

As a quick fix, changing all entries of the ipv6 addresses in /var/snap/etcd/common/etcd.conf.yml to the correct ipv4 address for etcd up and running with a healthy peer.

Revision history for this message
George Kraft (cynerva) wrote :

Thanks. Marking this as a duplicate of https://bugs.launchpad.net/charm-etcd/+bug/1913348

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.