After installation, lcm-k8s mariadb-k8s, and ui-k8s seem to keep waiting
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical-OSM |
New
|
Undecided
|
Unassigned |
Bug Description
Current status:
ubuntu@
Model Controller Cloud/Region Version SLA Timestamp
osm osm-on-k8s microk8s/localhost 2.6.6 unsupported 14:37:45Z
App Version Status Scale Charm Store Rev OS Address Notes
grafana-k8s active 1 grafana-k8s jujucharms 15 kubernetes 10.152.183.192
kafka-k8s active 1 kafka-k8s jujucharms 1 kubernetes 10.152.183.135
lcm-k8s waiting 1 lcm-k8s jujucharms 21 kubernetes 10.152.183.229
mariadb-k8s waiting 1 mariadb-k8s jujucharms 16 kubernetes 10.152.183.138
mon-k8s active 1 mon-k8s jujucharms 14 kubernetes 10.152.183.7
mongodb-k8s active 1 mongodb-k8s jujucharms 15 kubernetes 10.152.183.65
nbi-k8s active 1 nbi-k8s jujucharms 24 kubernetes 10.152.183.112
pol-k8s active 1 pol-k8s jujucharms 14 kubernetes 10.152.183.81
prometheus-k8s active 1 prometheus-k8s jujucharms 14 kubernetes 10.152.183.9
ro-k8s active 1 ro-k8s jujucharms 20 kubernetes 10.152.183.173
ui-k8s waiting 1 ui-k8s jujucharms 28 kubernetes 10.152.183.45
zookeeper-k8s active 1 zookeeper-k8s jujucharms 16 kubernetes 10.152.183.2
Unit Workload Agent Address Ports Message
grafana-k8s/0* active idle 10.1.1.12 3000/TCP configured
kafka-k8s/0* active idle 10.1.1.21 9092/TCP configured
lcm-k8s/0* active idle 10.1.1.14 80/TCP configured
mariadb-k8s/0* active idle 10.1.1.31 3306/TCP ready
mon-k8s/0* active idle 10.1.1.11 8000/TCP configured
mongodb-k8s/0* active idle 10.1.1.28 27017/TCP configured
nbi-k8s/0* active idle 10.1.1.33 9999/TCP configured
pol-k8s/0* active idle 10.1.1.24 80/TCP configured
prometheus-k8s/0* active idle 10.1.1.25 9090/TCP configured
ro-k8s/0* active idle 10.1.1.32 9090/TCP configured
ui-k8s/0* active idle 10.1.1.17 80/TCP configured
zookeeper-k8s/0* active idle 10.1.1.16 2181/TCP,
ubuntu@
NAME READY STATUS RESTARTS AGE
grafana-
grafana-
kafka-k8s-0 1/1 Running 16 2d23h
kafka-k8s-
lcm-k8s-0 0/1 CrashLoopBackOff 725 2d23h
lcm-k8s-operator-0 1/1 Running 4 3d1h
mariadb-k8s-0 0/1 Running 823 3d1h
mariadb-
mon-k8s-0 1/1 Running 3 2d23h
mon-k8s-operator-0 1/1 Running 4 3d1h
mongodb-k8s-0 1/1 Running 4 3d1h
mongodb-
nbi-k8s-0 1/1 Running 4 2d23h
nbi-k8s-operator-0 1/1 Running 4 3d1h
pol-k8s-0 1/1 Running 21 2d23h
pol-k8s-operator-0 1/1 Running 4 3d1h
prometheus-k8s-0 2/2 Running 4 2d23h
prometheus-
ro-k8s-0 1/1 Running 319 2d23h
ro-k8s-operator-0 1/1 Running 4 3d1h
ui-k8s-
ui-k8s-operator-0 1/1 Running 4 3d1h
zookeeper-k8s-0 1/1 Running 6 3d
zookeeper-
ubuntu@
2019-08-30 14:42:19 0 [Note] mysqld (mysqld 10.3.17-
2019-08-30 14:42:20 0 [Note] InnoDB: Using Linux native AIO
2019-08-30 14:42:20 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-08-30 14:42:20 0 [Note] InnoDB: Uses event mutexes
2019-08-30 14:42:20 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-08-30 14:42:20 0 [Note] InnoDB: Number of pools: 1
2019-08-30 14:42:20 0 [Note] InnoDB: Using SSE2 crc32 instructions
2019-08-30 14:42:20 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2019-08-30 14:42:20 0 [Note] InnoDB: Completed initialization of buffer pool
2019-08-30 14:42:20 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-08-30 14:42:22 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-08-30 14:42:22 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-08-30 14:42:22 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-08-30 14:42:22 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-08-30 14:42:23 0 [Note] InnoDB: 10.3.17 started; log sequence number 1639021; transaction id 21
2019-08-30 14:42:23 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/
2019-08-30 14:42:23 0 [Note] Plugin 'FEEDBACK' is disabled.
2019-08-30 14:42:23 0 [Note] Server socket created on IP: '::'.
2019-08-30 14:42:23 0 [Warning] 'user' entry 'root@mariadb-
2019-08-30 14:42:23 0 [Warning] 'user' entry '@mariadb-k8s-0' ignored in --skip-name-resolve mode.
2019-08-30 14:42:23 0 [Warning] 'proxies_priv' entry '@% root@mariadb-k8s-0' ignored in --skip-name-resolve mode.
2019-08-30 14:42:23 0 [Note] Reading of all Master_info entries succeeded
2019-08-30 14:42:23 0 [Note] Added new Master_info '' to hash table
2019-08-30 14:42:23 0 [Note] mysqld: ready for connections.
Version: '10.3.17-
2019-08-30 14:42:23 0 [Note] InnoDB: Buffer pool(s) load completed at 190830 14:42:23
2019-08-30 14:42:52 8 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:43:03 9 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:43:13 10 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:43:22 11 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:43:32 12 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:43:43 13 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:43:52 14 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:44:03 15 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
ubuntu@
Name: mariadb-k8s-0
Namespace: osm
Priority: 0
PriorityClassName: <none>
Node: osm-rel6-
Start Time: Tue, 27 Aug 2019 13:40:19 +0000
Labels: controller-
Annotations: apparmor.
Status: Running
IP: 10.1.1.31
Controlled By: StatefulSet/
Containers:
mariadb-k8s:
Container ID: containerd:
Image: registry.
Image ID: registry.
Port: 3306/TCP
Host Port: 0/TCP
State: Running
Started: Fri, 30 Aug 2019 14:42:14 +0000
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 30 Aug 2019 14:34:28 +0000
Finished: Fri, 30 Aug 2019 14:37:01 +0000
Ready: False
Restart Count: 823
Liveness: exec [sh -c exec mysqladmin status -uroot -p$MARIADB_
Readiness: exec [sh -c exec mysqladmin status -uroot -p$MARIADB_
Environment:
MARIADB_
MARIADB_
MARIADB_
MARIADB_USER: mano
Mounts:
/
/
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
database-
Type: PersistentVolum
ClaimName: database-
ReadOnly: false
default-
Type: Secret (a volume populated by a Secret)
SecretName: default-token-qnbsj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 49m (x530 over 47h) kubelet, osm-rel6-k8s (combined from similar events): Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to load task: no running task found: process 3caf4aca446c73c
Warning DNSConfigForming 44m (x8790 over 2d) kubelet, osm-rel6-k8s Search Line limits were exceeded, some search paths have been omitted, the applied search line is: osm.svc.
Warning Unhealthy 9m38s (x6270 over 2d) kubelet, osm-rel6-k8s Readiness probe failed: mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
Warning BackOff 4m51s (x6758 over 2d) kubelet, osm-rel6-k8s Back-off restarting failed container
ubuntu@
apiVersion: v1
kind: Pod
metadata:
annotations:
apparmor.
juju.
juju.io/model: 47126947-
seccomp.
creationTimes
generateName: mariadb-k8s-
labels:
controller-
juju-app: mariadb-k8s
statefulset
name: mariadb-k8s-0
namespace: osm
ownerReferences:
- apiVersion: apps/v1
blockOwnerD
controller: true
kind: StatefulSet
name: mariadb-k8s
uid: 321bf25b-
resourceVersion: "398238"
selfLink: /api/v1/
uid: 328162df-
spec:
containers:
- env:
- name: MARIADB_DATABASE
value: database
- name: MARIADB_PASSWORD
value: manopw
- name: MARIADB_
value: osm4u
- name: MARIADB_USER
value: mano
image: registry.
imagePullPo
livenessProbe:
exec:
command:
- sh
- -c
- exec mysqladmin status -uroot -p$MARIADB_
failureTh
initialDe
periodSec
successTh
timeoutSe
name: mariadb-k8s
ports:
- containerPort: 3306
name: main
protocol: TCP
readinessProbe:
exec:
command:
- sh
- -c
- exec mysqladmin status -uroot -p$MARIADB_
failureTh
initialDe
periodSec
successTh
timeoutSe
resources: {}
securityCon
allowPriv
readOnlyR
runAsNonRoot: false
termination
termination
volumeMounts:
- mountPath: /var/lib/mysql
name: database-d3baa19e
- mountPath: /var/run/
name: default-token-qnbsj
readOnly: true
dnsPolicy: ClusterFirst
enableService
hostname: mariadb-k8s-0
imagePullSecrets:
- name: mariadb-
nodeName: osm-rel6-k8s
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccoun
subdomain: mariadb-
terminationGr
tolerations:
- effect: NoExecute
key: node.kubernetes
operator: Exists
tolerationS
- effect: NoExecute
key: node.kubernetes
operator: Exists
tolerationS
volumes:
- name: database-d3baa19e
persistentV
claimName: database-
- name: default-token-qnbsj
secret:
defaultMode: 420
secretName: default-token-qnbsj
status:
conditions:
- lastProbeTime: null
lastTransit
status: "True"
type: Initialized
- lastProbeTime: null
lastTransit
message: 'containers with unready status: [mariadb-k8s]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransit
message: 'containers with unready status: [mariadb-k8s]'
reason: ContainersNotReady
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransit
status: "True"
type: PodScheduled
containerStat
- containerID: containerd:
image: sha256:
imageID: registry.
lastState:
terminated:
exitCode: 0
finishedAt: "2019-08-
reason: Completed
startedAt: "2019-08-
name: mariadb-k8s
ready: false
restartCount: 824
state:
waiting:
message: Back-off 5m0s restarting failed container=
reason: CrashLoopBackOff
hostIP: 10.20.30.66
phase: Running
podIP: 10.1.1.31
qosClass: BestEffort
startTime: "2019-08-
Here is my ubuntu version: osm-rel6- k8s:~$ cat /etc/os-release CODENAME= bionic CODENAME= bionic
ubuntu@
NAME=“Ubuntu”
VERSION=“18.04.2 LTS (Bionic Beaver)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 18.04.2 LTS”
VERSION_ID=“18.04”
…
VERSION_
UBUNTU_
Here are my ressources:
Running in a VM in OpenStack
RAM 12GB
VCPUs 20 VCPU
Disk 100GB