[UBUNTU 18.04.4] Charmed Kubernetes local deployment failed for s390x because of etcd error

Bug #1887539 reported by bugproxy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Etcd Snaps
Fix Released
Undecided
Unassigned
Ubuntu on IBM z Systems
Fix Released
Undecided
Skipper Bug Screeners

Bug Description

Steps to reproduce:

Clean Z LPAR machine. Followed official documentation https://ubuntu.com/kubernetes/docs/install-local.

Actual results:
Cluster did not start and hang in the follow state:
# juju status
Model Controller Cloud/Region Version SLA Timestamp
k8s lxd-controller localhost/localhost 2.8.1 unsupported 16:11:10+02:00

App Version Status Scale Charm Store Rev OS Notes
containerd 1.3.3 maintenance 5 containerd jujucharms 80 ubuntu
easyrsa 3.0.1 active 1 easyrsa jujucharms 318 ubuntu
etcd 3.3.15 waiting 3 etcd jujucharms 521 ubuntu
flannel 0.11.0 blocked 5 flannel jujucharms 492 ubuntu
kubeapi-load-balancer 1.14.0 active 1 kubeapi-load-balancer jujucharms 730 ubuntu exposed
kubernetes-master 1.18.5 waiting 2 kubernetes-master jujucharms 850 ubuntu
kubernetes-worker 1.18.5 blocked 3 kubernetes-worker jujucharms 682 ubuntu exposed

Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 10.63.81.207 Certificate Authority connected.
etcd/0* active idle 1 10.63.81.209 2379/tcp Errored with 0 known peers
etcd/1 waiting idle 2 10.63.81.36 Waiting to retry etcd registration
etcd/2 waiting idle 3 10.63.81.218 Waiting to retry etcd registration
kubeapi-load-balancer/0* active idle 4 10.63.81.145 NGINX is ready
kubernetes-master/0* waiting executing 5 10.63.81.239 Waiting for kube-api-endpoint relation
  containerd/4 maintenance executing 10.63.81.239 (install) installing charm software
  flannel/4 blocked executing 10.63.81.239 (install) Waiting for etcd relation.
kubernetes-master/1 waiting executing 6 10.63.81.213 Waiting for kube-api-endpoint relation
  containerd/3 active executing 10.63.81.213 (leader-settings-changed) Container runtime available
  flannel/3 blocked executing 10.63.81.213 (leader-settings-changed) Waiting for etcd relation.
kubernetes-worker/0 waiting executing 7 10.63.81.155 Waiting for cluster DNS.
  containerd/2 active executing 10.63.81.155 (start) Container runtime available
  flannel/2 maintenance executing 10.63.81.155 Negotiating flannel network subnet.
kubernetes-worker/1 blocked executing 8 10.63.81.53 (config-changed) Connect a container runtime.
  containerd/1 active executing 10.63.81.53 (start) Container runtime available
  flannel/1 maintenance executing 10.63.81.53 Negotiating flannel network subnet.
kubernetes-worker/2* blocked executing 9 10.63.81.219 Connect a container runtime.
  containerd/0* active idle 10.63.81.219 Container runtime available
  flannel/0* maintenance executing 10.63.81.219 Negotiating flannel network subnet.

Machine State DNS Inst id Series AZ Message
0 started 10.63.81.207 juju-6e9ea7-0 bionic Running
1 started 10.63.81.209 juju-6e9ea7-1 bionic Running
2 started 10.63.81.36 juju-6e9ea7-2 bionic Running
3 started 10.63.81.218 juju-6e9ea7-3 bionic Running
4 started 10.63.81.145 juju-6e9ea7-4 bionic Running
5 started 10.63.81.239 juju-6e9ea7-5 bionic Running
6 started 10.63.81.213 juju-6e9ea7-6 bionic Running
7 started 10.63.81.155 juju-6e9ea7-7 bionic Running
8 started 10.63.81.53 juju-6e9ea7-8 bionic Running
9 started 10.63.81.219 juju-6e9ea7-9 bionic Running

After ssh to etcd/1 I can see error logs at /var/log/syslog:
Jul 14 14:11:46 juju-6e9ea7-2 etcd.etcd[27280]: 2020-07-14 14:11:46.612925 E | etcdmain: etcd on unsupported platform without ETCD_UNSUPPORTED_ARCH=s390x set.
Jul 14 14:11:46 juju-6e9ea7-2 systemd[1]: snap.etcd.etcd.service: Main process exited, code=exited, status=1/FAILURE
Jul 14 14:11:46 juju-6e9ea7-2 systemd[1]: snap.etcd.etcd.service: Failed with result 'exit-code'.

bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-186835 severity-high targetmilestone-inin18045
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Frank Heimes (fheimes)
affects: linux (Ubuntu) → etcd (Ubuntu)
Changed in ubuntu-z-systems:
assignee: nobody → Kubernetes Charmers (kubernetes)
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
assignee: Kubernetes Charmers (kubernetes) → Skipper Bug Screeners (skipper-screen-team)
Changed in etcd (Ubuntu):
assignee: Skipper Bug Screeners (skipper-screen-team) → kubernetes (k8s)
Changed in ubuntu-z-systems:
status: New → Triaged
Frank Heimes (fheimes)
Changed in etcd (Ubuntu):
assignee: kubernetes (k8s) → Kubernetes Charmers (kubernetes)
George Kraft (cynerva)
no longer affects: cdk8s
Changed in etcd (Ubuntu):
status: New → Invalid
Revision history for this message
George Kraft (cynerva) wrote :

We're aware of this issue and committed a fix back in March, but it looks like we never released it.

See https://bugs.launchpad.net/etcd-snaps/+bug/1869232

no longer affects: charm-etcd
Changed in etcd-snaps:
status: New → Fix Committed
Frank Heimes (fheimes)
no longer affects: etcd (Ubuntu)
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-07-16 04:43 EDT-------
Is there a planned target date available, when this problem will be fixed?
Many thx in advance

Revision history for this message
George Kraft (cynerva) wrote :

We don't currently have a target date. My boss is out this week, but when he gets in on Monday, I will bring it to his attention. I'm hopeful we can get a release out soon, maybe sometime next week, but no promises.

Revision history for this message
Tim Van Steenburgh (tvansteenburgh) wrote :

Fix is in these snaps:

  3.4/stable: 3.4.5 2020-04-30 (230) 22MB -
  3.4/candidate: 3.4.5 2020-04-30 (230) 22MB -
  3.4/beta: 3.4.5 2020-04-30 (230) 22MB -
  3.4/edge: 3.4.5 2020-04-30 (230) 22MB -
  3.3/stable: 3.3.19 2020-07-21 (229) 21MB -
  3.3/candidate: 3.3.19 2020-07-21 (229) 21MB -
  3.3/beta: 3.3.19 2020-07-21 (229) 21MB -
  3.3/edge: 3.3.19 2020-04-07 (229) 21MB -

Changed in etcd-snaps:
status: Fix Committed → Fix Released
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Triaged → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-08-26 09:09 EDT-------
@Canonical. any update available here?

Revision history for this message
Frank Heimes (fheimes) wrote :

This ticket is already marked as Fix Released and therefore closed.
With that a fix was provided and updates were rolled-out already.
See also comment #4.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-09-04 03:02 EDT-------
IBM Bugzilla closed, Fix Released by Canonical

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.