non cdk apps stuck at "waiting for /root/.kube/config" when filebeat in cdk env
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Charmed Kubernetes Bundles |
Fix Released
|
Low
|
Kevin W Monroe | ||
Filebeat Charm |
Fix Released
|
Medium
|
Kevin W Monroe |
Bug Description
In a charmed kubernetes environment, adding the LMA stack (or vault), is seeing that non CDK items are having filebeat stuck in a situation where it says 'Waiting for: /root/.kube/config'
Here's a juju status, notice that all non k8s related items have this message:
---
$ juju status filebeat
Model Controller Cloud/Region Version SLA Timestamp
kubernetes foundations-maas maas_cloud 2.6.10 unsupported 17:32:40Z
App Version Status Scale Charm Store Rev OS Notes
canal 0.10.0/3.10.1 active 0 canal jujucharms 688 ubuntu
ceph-osd 12.2.12 active 3 ceph-osd jujucharms 294 ubuntu
containerd active 0 containerd local 0 ubuntu
filebeat 6.8.6 maintenance 18 filebeat jujucharms 25 ubuntu
hacluster-
hacluster-vault active 0 hacluster jujucharms 63 ubuntu
kubernetes-master 1.17.0 active 3 kubernetes-master jujucharms 788 ubuntu
kubernetes-worker 1.17.0 active 3 kubernetes-worker jujucharms 623 ubuntu exposed
landscape-client maintenance 0 landscape-client jujucharms 32 ubuntu
landscape-haproxy active 1 haproxy jujucharms 55 ubuntu exposed
landscape-
landscape-
landscape-server active 3 landscape-server jujucharms 35 ubuntu
nrpe-container active 0 nrpe jujucharms 60 ubuntu
nrpe-host active 0 nrpe jujucharms 60 ubuntu
telegraf active 0 telegraf jujucharms 29 ubuntu
vault 1.1.1 active 3 vault jujucharms 32 ubuntu
Unit Workload Agent Machine Public address Ports Message
ceph-osd/0* active idle 22 10.109.12.5 Unit is ready (8 OSD)
ceph-osd/1 active idle 23 10.109.12.6 Unit is ready (8 OSD)
ceph-osd/2 active idle 24 10.109.12.7 Unit is ready (8 OSD)
kubernetes-master/0 active idle 19/lxd/2 10.109.15.112 6443/tcp Kubernetes master running.
canal/5 active idle 10.109.15.112 Flannel subnet 10.1.9.1/24
containerd/5 active idle 10.109.15.112 Container runtime available
filebeat/13 active idle 10.109.15.112 Filebeat ready.
hacluster-
landscape-
nrpe-container/12 active idle 10.109.15.112 icmp,5666/tcp ready
telegraf/16 active idle 10.109.15.112 9103/tcp Monitoring kubernetes-master/0
kubernetes-master/1 active idle 20/lxd/2 10.109.15.116 6443/tcp Kubernetes master running.
canal/4 active idle 10.109.15.116 Flannel subnet 10.1.80.1/24
containerd/4 active idle 10.109.15.116 Container runtime available
filebeat/12 active idle 10.109.15.116 Filebeat ready.
hacluster-
landscape-
nrpe-container/11 active idle 10.109.15.116 icmp,5666/tcp ready
telegraf/15 active idle 10.109.15.116 9103/tcp Monitoring kubernetes-master/1
kubernetes-
canal/3 active idle 10.109.15.110 Flannel subnet 10.1.16.1/24
containerd/3 active idle 10.109.15.110 Container runtime available
filebeat/11 active idle 10.109.15.110 Filebeat ready.
hacluster-
landscape-
nrpe-container/10 active idle 10.109.15.110 icmp,5666/tcp ready
telegraf/14 active idle 10.109.15.110 9103/tcp Monitoring kubernetes-master/2
kubernetes-
canal/0* active idle 10.109.12.5 Flannel subnet 10.1.58.1/24
containerd/0* active idle 10.109.12.5 Container runtime available
filebeat/8 active idle 10.109.12.5 Filebeat ready.
landscape-
nrpe-host/12 active idle 10.109.12.5 icmp,5666/tcp ready
telegraf/11 active idle 10.109.12.5 9103/tcp Monitoring kubernetes-worker/0
kubernetes-worker/1 active idle 23 10.109.12.6 80/tcp,443/tcp Kubernetes worker running.
canal/1 active idle 10.109.12.6 Flannel subnet 10.1.19.1/24
containerd/1 active idle 10.109.12.6 Container runtime available
filebeat/9 active idle 10.109.12.6 Filebeat ready.
landscape-
nrpe-host/13 active idle 10.109.12.6 icmp,5666/tcp ready
telegraf/12 active idle 10.109.12.6 9103/tcp Monitoring kubernetes-worker/1
kubernetes-worker/2 active idle 24 10.109.12.7 80/tcp,443/tcp Kubernetes worker running.
canal/2 active idle 10.109.12.7 Flannel subnet 10.1.73.1/24
containerd/2 active idle 10.109.12.7 Container runtime available
filebeat/10 active idle 10.109.12.7 Filebeat ready.
landscape-
nrpe-host/14 active idle 10.109.12.7 icmp,5666/tcp ready
telegraf/13 active idle 10.109.12.7 9103/tcp Monitoring kubernetes-worker/2
landscape-
filebeat/0 maintenance idle 10.109.15.86 Waiting for: /root/.kube/config
landscape-
nrpe-host/1 active idle 10.109.15.86 icmp,5666/tcp ready
telegraf/0* active idle 10.109.15.86 9103/tcp Monitoring landscape-haproxy/0
landscape-
filebeat/15 maintenance idle 10.109.15.90 Waiting for: /root/.kube/config
landscape-
nrpe-host/18 active idle 10.109.15.90 icmp,5666/tcp ready
telegraf/18 active idle 10.109.15.90 9103/tcp Monitoring landscape-
landscape-
filebeat/6 maintenance idle 10.109.15.93 Waiting for: /root/.kube/config
landscape-
nrpe-host/10 active idle 10.109.15.93 icmp,5666/tcp ready
telegraf/6 active idle 10.109.15.93 9103/tcp Monitoring landscape-
landscape-
filebeat/7* maintenance idle 10.109.15.97 Waiting for: /root/.kube/config
landscape-
nrpe-host/11 active idle 10.109.15.97 icmp,5666/tcp ready
telegraf/7 active idle 10.109.15.97 9103/tcp Monitoring landscape-
landscape-
filebeat/17 maintenance idle 10.109.15.92 Waiting for: /root/.kube/config
landscape-
nrpe-host/20 active idle 10.109.15.92 icmp,5666/tcp ready
telegraf/20 active idle 10.109.15.92 9103/tcp Monitoring landscape-
landscape-
filebeat/5 maintenance idle 10.109.15.83 Waiting for: /root/.kube/config
landscape-
nrpe-host/8 active idle 10.109.15.83 icmp,5666/tcp ready
telegraf/5 active idle 10.109.15.83 9103/tcp Monitoring landscape-
landscape-server/0 active idle 15 10.109.15.88
filebeat/1 maintenance idle 10.109.15.88 Waiting for: /root/.kube/config
landscape-
nrpe-host/4 active idle 10.109.15.88 icmp,5666/tcp ready
telegraf/1 active idle 10.109.15.88 9103/tcp Monitoring landscape-server/0
landscape-server/1 active idle 16 10.109.15.95
filebeat/16 maintenance idle 10.109.15.95 Waiting for: /root/.kube/config
landscape-
nrpe-host/19 active idle 10.109.15.95 icmp,5666/tcp ready
telegraf/19 active idle 10.109.15.95 9103/tcp Monitoring landscape-server/1
landscape-server/2* active idle 17 10.109.15.87
filebeat/2 maintenance idle 10.109.15.87 Waiting for: /root/.kube/config
landscape-
nrpe-host/5 active idle 10.109.15.87 icmp,5666/tcp ready
telegraf/2 active idle 10.109.15.87 9103/tcp Monitoring landscape-server/2
vault/0 active idle 6 10.109.15.84 8200/tcp Unit is ready (active: true, mlock: enabled)
filebeat/14 maintenance idle 10.109.15.84 Waiting for: /root/.kube/config
hacluster-vault/2 active idle 10.109.15.84 Unit is ready and clustered
landscape-
nrpe-host/17 active idle 10.109.15.84 icmp,5666/tcp ready
telegraf/17 active idle 10.109.15.84 9103/tcp Monitoring vault/0
vault/1 active idle 7 10.109.15.94 8200/tcp Unit is ready (active: false, mlock: enabled)
filebeat/4 maintenance idle 10.109.15.94 Waiting for: /root/.kube/config
hacluster-
landscape-
nrpe-host/7 active idle 10.109.15.94 icmp,5666/tcp ready
telegraf/4 active idle 10.109.15.94 9103/tcp Monitoring vault/1
vault/2* active idle 8 10.109.15.89 8200/tcp Unit is ready (active: false, mlock: enabled)
filebeat/3 maintenance idle 10.109.15.89 Waiting for: /root/.kube/config
hacluster-vault/0 active idle 10.109.15.89 Unit is ready and clustered
landscape-
nrpe-host/6 active idle 10.109.15.89 icmp,5666/tcp ready
telegraf/3 active idle 10.109.15.89 9103/tcp Monitoring vault/2
---
Here's the relations from that same output
---
Relation provider Requirer Interface Type Message
apache2:juju-info landscape-
apache2:
ceph-mon:admin kubernetes-
ceph-mon:client kubernetes-
ceph-mon:client landscape-
ceph-mon:mon ceph-mon:mon ceph peer
ceph-mon:
ceph-mon:osd ceph-osd:mon ceph-osd regular
easyrsa:client etcd:certificates tls-certificates regular
easyrsa:client kubernetes-
easyrsa:client kubernetes-
easyrsa:juju-info landscape-
easyrsa:juju-info nrpe-host:
elasticsearch:
elasticsearch:
elasticsearch:
elasticsearch:peer elasticsearch:peer http peer
etcd:cluster etcd:cluster etcd peer
etcd:db canal:etcd etcd regular
etcd:db kubernetes-
etcd:db vault:etcd etcd regular
etcd:juju-info landscape-
etcd:nrpe-
grafana:juju-info landscape-
grafana:
graylog:beats filebeat:logstash elastic-beats regular
graylog:juju-info landscape-
graylog:
graylog:website apache2:
hacluster-
hacluster-
hacluster-mysql:ha mysql:ha hacluster subordinate
hacluster-
hacluster-vault:ha vault:ha hacluster subordinate
hacluster-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
kubernetes-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
landscape-
mongodb:database graylog:mongodb mongodb regular
mongodb:juju-info landscape-
mongodb:
mongodb:replica-set mongodb:replica-set mongodb-replica-set peer
mysql:cluster mysql:cluster percona-cluster peer
mysql:juju-info landscape-
mysql:juju-info telegraf:juju-info juju-info subordinate
mysql:nrpe-
mysql:shared-db vault:shared-db mysql-shared regular
nrpe-container:
nrpe-host:monitors nagios:monitors monitors regular
prometheus:
prometheus:
prometheus:
telegraf:
vault:cluster vault:cluster vault-ha peer
vault:juju-info filebeat:beats-host juju-info subordinate
vault:juju-info landscape-
vault:juju-info telegraf:juju-info juju-info subordinate
vault:nrpe-
vault:secrets ceph-osd:
summary: |
- non cdk apps stuck at "waiting for /root/.cdk/config" when filebeat in + non cdk apps stuck at "waiting for /root/.kube/config" when filebeat in cdk env |
description: | updated |
description: | updated |
description: | updated |
Changed in filebeat-charm: | |
assignee: | nobody → Kevin W Monroe (kwmonroe) |
importance: | Undecided → Medium |
Changed in filebeat-charm: | |
status: | In Progress → Fix Committed |
Changed in charmed-kubernetes-bundles: | |
assignee: | nobody → Kevin W Monroe (kwmonroe) |
milestone: | none → 1.18 |
status: | New → In Progress |
importance: | Undecided → Low |
Changed in charmed-kubernetes-bundles: | |
status: | Fix Committed → Fix Released |
Changed in filebeat-charm: | |
status: | Fix Committed → Fix Released |
Duplicate issue with additional details: https:/ /bugs.launchpad .net/filebeat- charm/+ bug/1860591