Bug #1740896 “Kubernetes Dashboard is not working.” : Bugs : Magnum

Syed Armani (dce3062) on 2018-01-02

description:	updated
description:	updated

Revision history for this message

Syed Armani (dce3062) wrote on 2018-01-08:

#1

Update: After removing the livenessProbe as suggested by ykarel on irc, the dashboard pod has stopped crashing. This is how it looks now: http://paste.openstack.org/show/640751/

When trying to access the K8 dashboard; the browser throws this error: http://paste.openstack.org/show/639201/

# kubectl --namespace=kube-system get ep kubernetes-dashboard -- http://paste.openstack.org/show/640754/

The systemd daemon for dashboard refuse to start perhaps because of its type i.e. oneshot. The "kube-dash.service" and "kube-system-namespace.service" are in dead and inactive state.

http://paste.openstack.org/show/640752/

I also observed that at L106[1], there is no kube-dashboard deployment, this should be changed to "kubernetes-dashboard" in /usr/local/bin/kube-dash

[1] https://github.com/openstack/magnum/blob/stable/pike/magnum/drivers/common/templates/kubernetes/fragments/kube-dashboard-service.sh#L106

Revision history for this message

Syed Armani (dce3062) wrote on 2018-01-11:

#2

Download full text (10.3 KiB)

The following output is from a devstack Pike deployment:

[root@kubernetes-iwuczy6he3eq-master-0 log]# systemctl -a | grep kube | grep dead
kube-dash.service loaded inactive dead kube-dash.service
kube-system-namespace.service loaded inactive dead kube-system-namespace.service

[root@kubernetes-iwuczy6he3eq-master-0 log]# systemctl -a | grep kube | grep failed
● kube-controller-manager.service loaded failed failed kubernetes-controller-manager

[root@kubernetes-iwuczy6he3eq-master-0 log]# tail -n200 cloud-init-output.log
Cloud-init v. 0.7.9 running 'init-local' at Thu, 11 Jan 2018 13:33:52 +0000. Up 25.70 seconds.
Cloud-init v. 0.7.9 running 'init' at Thu, 11 Jan 2018 13:33:57 +0000. Up 31.43 seconds.
ci-info: +++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++
ci-info: +--------+------+-----------+---------------+-------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+-----------+---------------+-------+-------------------+
ci-info: | eth0: | True | 10.0.0.3 | 255.255.255.0 | . | fa:16:3e:e3:e7:76 |
ci-info: | eth0: | True | . | . | d | fa:16:3e:e3:e7:76 |
ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . |
ci-info: | lo: | True | . | . | d | . |
ci-info: +--------+------+-----------+---------------+-------+-------------------+
ci-info: ++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 10.0.0.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: | 2 | 169.254.169.254 | 10.0.0.1 | 255.255.255.255 | eth0 | UGH |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
Cloud-init v. 0.7.9 running 'modules:config' at Thu, 11 Jan 2018 13:34:06 +0000. Up 39.97 seconds.
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 1417 100 1417 0 0 1211 0 0:00:01 0:00:01 --:--:-- 1216
Generating RSA private key, 4096 bit long modulus
..........................................................................................................................................++
...................................................................................................................................++
unable to write 'random state'
e is 65537 (0x010001)
  % Total % Received % Xferd Average Speed ...

The following output is from a devstack Pike deployment:

[root@kubernetes-iwuczy6he3eq-master-0 log]# systemctl -a | grep kube | grep dead
  kube-dash.service                                                                                              loaded    inactive dead      kube-dash.service
  kube-system-namespace.service                                                                                  loaded    inactive dead      kube-system-namespace.service

[root@kubernetes-iwuczy6he3eq-master-0 log]# systemctl -a | grep kube | grep failed
● kube-controller-manager.service                                                                                loaded    failed     failed          kubernetes-controller-manager

[root@kubernetes-iwuczy6he3eq-master-0 log]# tail -n200 cloud-init-output.log
Cloud-init v. 0.7.9 running 'init-local' at Thu, 11 Jan 2018 13:33:52 +0000. Up 25.70 seconds.
Cloud-init v. 0.7.9 running 'init' at Thu, 11 Jan 2018 13:33:57 +0000. Up 31.43 seconds.
ci-info: +++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++
ci-info: +--------+------+-----------+---------------+-------+-------------------+
ci-info: | Device |  Up  |  Address  |      Mask     | Scope |     Hw-Address    |
ci-info: +--------+------+-----------+---------------+-------+-------------------+
ci-info: | eth0:  | True |  10.0.0.3 | 255.255.255.0 |   .   | fa:16:3e:e3:e7:76 |
ci-info: | eth0:  | True |     .     |       .       |   d   | fa:16:3e:e3:e7:76 |
ci-info: |  lo:   | True | 127.0.0.1 |   255.0.0.0   |   .   |         .         |
ci-info: |  lo:   | True |     .     |       .       |   d   |         .         |
ci-info: +--------+------+-----------+---------------+-------+-------------------+
ci-info: ++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
ci-info: | Route |   Destination   | Gateway  |     Genmask     | Interface | Flags |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
ci-info: |   0   |     0.0.0.0     | 10.0.0.1 |     0.0.0.0     |    eth0   |   UG  |
ci-info: |   1   |     10.0.0.0    | 0.0.0.0  |  255.255.255.0  |    eth0   |   U   |
ci-info: |   2   | 169.254.169.254 | 10.0.0.1 | 255.255.255.255 |    eth0   |  UGH  |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
Cloud-init v. 0.7.9 running 'modules:config' at Thu, 11 Jan 2018 13:34:06 +0000. Up 39.97 seconds.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1417  100  1417    0     0   1211      0  0:00:01  0:00:01 --:--:--  1216
Generating RSA private key, 4096 bit long modulus
..........................................................................................................................................++
...................................................................................................................................++
unable to write 'random state'
e is 65537 (0x010001)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5787  100  3844  100  1943   3246   1641  0:00:01  0:00:01 --:--:--  3249
  Logical volume "docker-root-lv" successfully removed
  Size of logical volume atomicos/root changed from 2.93 GiB (750 extents) to 5.00 GiB (1280 extents).
  Logical volume atomicos/root successfully resized.
meta-data=/dev/mapper/atomicos-root isize=512    agcount=4, agsize=192000 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1 spinodes=0 rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=768000, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 768000 to 1310720
  Using default stripesize 64.00 KiB.
  Rounding up size to full physical extent 24.00 MiB
  Logical volume "docker-pool" created.
  Logical volume atomicos/docker-pool changed.
configuring kubernetes (master)
Pulling layer eb64f1fb7ee0716f652a39381e4b18393cb9900910472235ddc3f9345261d574
Pulling layer 8d941e01c5e903b561e1911629de5465990d66ea6e0c82ef37bec6a3fed0c7d1
Pulling layer 0df692b4291a04a352d4dcba308d54bd7a9d66cafbad67a114e444932c6eba10
Pulling layer b9769b82ce121190e240a532ea9b4d6f6fb852cd37b1696b077f4a2619fbba9c
Pulling layer 13377f22e91034305c3f717dfeea74a72109288dab1a884fd19af5fc022158dc
Pulling layer e64209a9fce88e72ac7e1fd07ef66c2f32e509a54397162b5d8d80f79cdbd25f
Pulling layer d6f88f4a8cf117af666c4e7d4d8e4f47babc9220987e6218ac8d273f4be201b6
Extracting to /var/lib/containers/atomic/kubelet.0
File /etc/kubernetes/config already exists.
File /etc/kubernetes/kubelet already exists.
systemctl daemon-reload
systemctl enable kubelet
Pulling layer f0c510d733c9606b91aa96d95a4c2f7ac5c02c50057269ecd28dcadac96536de
Pulling layer 22ddfee1625bdd1d13c7292a138521c7f6cc5f91bbb700da32aefd0a18a17529
Pulling layer f508804388fb7966cb4c521a4b44437de8a83c797df6e24cbb1121bcd495fd78
Pulling layer 328489b8a82fe8df8c6afe676ff976b4c5d35828660fe31a0914a87f6355d812
Extracting to /var/lib/containers/atomic/kube-proxy.0
File /etc/kubernetes/config already exists.
File /etc/kubernetes/proxy already exists.
systemctl daemon-reload
systemctl enable kube-proxy
Pulling layer 30d390a776555d74cde2560c8940aa1f1e8162d4090e87fc7232febc764f651c
Pulling layer f2b925ff232c0235807d73159b937636f2d4194d3c4c192e43bb76a470dad208
Pulling layer 431d515173ff031aa328304f4bf032246ae2257d48b413b700cc6711c3e8adf5
Pulling layer 2683e1b75e077bb91f0e20b43bd7bf8c3cd355fc4418310a2d034c93353c6f22
Pulling layer 60dd089151d6d60cdba6612077dd5705ffc671c72a1e66d2f3afd41e6c1b67a0
Pulling layer 6269798af3926f4a085e52dc099511bf3f5e5f8d873fd726d288872a35835691
Pulling layer f920532e2d37839913da276fb2dbadab1adb1eb1c9d041ae6c2ce8fbedf7e672
Pulling layer aa08a998035e930437171f6de5ff3052b232582dd2ac5ac748b916d0002a354e
Extracting to /var/lib/containers/atomic/kube-apiserver.0
File /etc/kubernetes/apiserver already exists.
File /etc/kubernetes/config already exists.
systemctl daemon-reload
systemctl enable kube-apiserver
Pulling layer 39bd7bae433268b7fc42e0211cfaa5a4b284bd9a5e6c29b5cddcc73d621aa5d6
Pulling layer 8c496ae5d983608658b3d1dd564e3100383bd2c2b5abc54362bf51ec80170e83
Pulling layer a2eda224f51c8cd0a91d4d6a452b917bd560b86b2dccae42ecb1dba05db008fe
Extracting to /var/lib/containers/atomic/kube-controller-manager.0
File /etc/kubernetes/config already exists.
File /etc/kubernetes/controller-manager already exists.
systemctl daemon-reload
systemctl enable kube-controller-manager
Pulling layer 567e3e894f0f10ee0c7b36a56fc0b977722b835c4e0d1ab78b40cd43d6b136d2
Pulling layer f2e309f5568b717edef381d82ac43354dda0941ebd3672fc9ae8fc244d3d493d
Pulling layer 329afdf1da750b7307dc1e6cc7be7ddff36c4610d7d4ce1808b29ffa796cfc73
Extracting to /var/lib/containers/atomic/kube-scheduler.0
File /etc/kubernetes/config already exists.
File /etc/kubernetes/scheduler already exists.
systemctl daemon-reload
systemctl enable kube-scheduler
starting services
activating service etcd
Created symlink /etc/systemd/system/multi-user.target.wants/etcd.service → /usr/lib/systemd/system/etcd.service.
activating service docker
activating service kube-apiserver
activating service kube-controller-manager
activating service kube-scheduler
activating service kubelet
activating service kube-proxy
creating /usr/local/bin/flannel-config
Created symlink /etc/systemd/system/multi-user.target.wants/flannel-config.service → /etc/systemd/system/flannel-config.service.
activating service flanneld
Created symlink /etc/systemd/system/multi-user.target.wants/flanneld.service → /usr/lib/systemd/system/flanneld.service.
Created symlink /etc/systemd/system/docker.service.requires/flanneld.service → /usr/lib/systemd/system/flanneld.service.
Writing File: /srv/kubernetes/kube-system-namespace.json
Writing File: /usr/local/bin/kube-system-namespace
Writing File: /etc/systemd/system/kube-system-namespace.service
Created symlink /etc/systemd/system/multi-user.target.wants/kube-system-namespace.service → /etc/systemd/system/kube-system-namespace.service.
Writing File: /etc/kubernetes/manifests/kube-coredns.yaml
Waiting for Kubernetes API...
{
  "major": "1",
  "minor": "7",
  "gitVersion": "v1.7.4",
  "gitCommit": "793658f2d7ca7f064d2bdf606519f9fe1229c381",
  "gitTreeState": "clean",
  "buildDate": "2017-08-18T14:17:16Z",
  "goVersion": "go1.9beta2",
  "compiler": "gc",
  "platform": "linux/amd64"
}2018-01-11 13:41:44.516244 I | proto: duplicate proto type registered: google.protobuf.Any
2018-01-11 13:41:44.541569 I | proto: duplicate proto type registered: google.protobuf.Duration
2018-01-11 13:41:44.542057 I | proto: duplicate proto type registered: google.protobuf.Timestamp
configmap "coredns" created
service "kube-dns" created
error: unable to recognize "/etc/kubernetes/manifests/kube-coredns.yaml": no matches for extensions/, Kind=Deployment
Cloud-init v. 0.7.9 running 'modules:final' at Thu, 11 Jan 2018 13:34:10 +0000. Up 43.53 seconds.
2018-01-11 13:41:45,912 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-017 [1]
Writing File: /srv/kubernetes/manifests/kube-dash-deploy.yaml
Writing File: /srv/kubernetes/manifests/kube-dash-svc.yaml
Writing File: /usr/local/bin/kube-dash
Writing File: /etc/systemd/system/kube-dash.service
Created symlink /etc/systemd/system/multi-user.target.wants/kube-dash.service → /etc/systemd/system/kube-dash.service.
Created symlink /etc/systemd/system/multi-user.target.wants/wc-notify.service → /etc/systemd/system/wc-notify.service.
2018-01-11 13:41:46,945 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2018-01-11 13:41:46,951 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3.6/site-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 0.7.9 finished at Thu, 11 Jan 2018 13:41:47 +0000. Datasource DataSourceOpenStack [net,ver=2].  Up 501.41 seconds

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-13: Related fix proposed to magnum (stable/pike)

#3

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/533346

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-15:

#4

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/533529

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-15: Change abandoned on magnum (stable/pike)

#5

Change abandoned by yatin (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/533346
Reason: Fixed here: https://review.openstack.org/#/c/533529/

Revision history for this message

yatin (yatinkarel) wrote on 2018-01-15:

#6

<<< Update: After removing the livenessProbe as suggested by ykarel on irc, the dashboard pod has stopped crashing
>>> This was just to check how kubectl apply behaves without liveliness.

<<< The systemd daemon for dashboard refuse to start perhaps because of its type i.e. oneshot. The "kube-dash.service" and "kube-system-namespace.service" are in dead and inactive state.

kube-dash systemd service is just to create service and deployment so it's dead state is expected.

<<< kubectl --namespace=kube-system get ep kubernetes-dashboard -- http://paste.openstack.org/show/640754/
>>> Here endpoint is missing. This is the issue that needs to be fixed.

<<< I also observed that at L106[1], there is no kube-dashboard deployment, this should be changed to "kubernetes-dashboard" in /usr/local/bin/kube-dash

[1] https://github.com/openstack/magnum/blob/stable/pike/magnum/drivers/common/templates/kubernetes/fragments/kube-dashboard-service.sh#L106

>> Yes, it's wrong

Was not able to reproduce it on devstack stable/pike. Trying to setup a osa setup to look into it.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-15: Related fix merged to magnum (master)

#7

Reviewed: https://review.openstack.org/531690
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=2a60e56d412c07317fbf3b82de691574d8e7e77b
Submitter: Zuul
Branch: master

commit 2a60e56d412c07317fbf3b82de691574d8e7e77b
Author: Syed Armani <email address hidden>
Date: Mon Jan 8 14:13:58 2018 +0530

Change the name of kubernetes-dashboard deployment.

Related-Bug: #1740896

Change-Id: I467ec56cfeac34021a761dd2ebf0e7611052a352

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-15: Related fix merged to magnum (stable/pike)

#8

Reviewed: https://review.openstack.org/533529
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=798180361fb317ef478fa603a0bd310c1583b47b
Submitter: Zuul
Branch: stable/pike

commit 798180361fb317ef478fa603a0bd310c1583b47b
Author: Syed Armani <email address hidden>
Date: Mon Jan 8 14:13:58 2018 +0530

Change the name of kubernetes-dashboard deployment in Pike.

Related-Bug: #1740896

Change-Id: I467ec56cfeac34021a761dd2ebf0e7611052a352
(cherry picked from commit 2a60e56d412c07317fbf3b82de691574d8e7e77b)

tags:

added: in-stable-pike

Magnum

Kubernetes Dashboard is not working.

Bug Description

Other bug subscribers

Remote bug watches