After installation, lcm-k8s mariadb-k8s, and ui-k8s seem to keep waiting

[guilag]

Current status:
ubuntu@osm-rel6-k8s:~$ juju status
Model Controller Cloud/Region Version SLA Timestamp
osm osm-on-k8s microk8s/localhost 2.6.6 unsupported 14:37:45Z

App Version Status Scale Charm Store Rev OS Address Notes
grafana-k8s active 1 grafana-k8s jujucharms 15 kubernetes 10.152.183.192
kafka-k8s active 1 kafka-k8s jujucharms 1 kubernetes 10.152.183.135
lcm-k8s waiting 1 lcm-k8s jujucharms 21 kubernetes 10.152.183.229
mariadb-k8s waiting 1 mariadb-k8s jujucharms 16 kubernetes 10.152.183.138
mon-k8s active 1 mon-k8s jujucharms 14 kubernetes 10.152.183.7
mongodb-k8s active 1 mongodb-k8s jujucharms 15 kubernetes 10.152.183.65
nbi-k8s active 1 nbi-k8s jujucharms 24 kubernetes 10.152.183.112
pol-k8s active 1 pol-k8s jujucharms 14 kubernetes 10.152.183.81
prometheus-k8s active 1 prometheus-k8s jujucharms 14 kubernetes 10.152.183.9
ro-k8s active 1 ro-k8s jujucharms 20 kubernetes 10.152.183.173
ui-k8s waiting 1 ui-k8s jujucharms 28 kubernetes 10.152.183.45
zookeeper-k8s active 1 zookeeper-k8s jujucharms 16 kubernetes 10.152.183.2

Unit Workload Agent Address Ports Message
grafana-k8s/0* active idle 10.1.1.12 3000/TCP configured
kafka-k8s/0* active idle 10.1.1.21 9092/TCP configured
lcm-k8s/0* active idle 10.1.1.14 80/TCP configured
mariadb-k8s/0* active idle 10.1.1.31 3306/TCP ready
mon-k8s/0* active idle 10.1.1.11 8000/TCP configured
mongodb-k8s/0* active idle 10.1.1.28 27017/TCP configured
nbi-k8s/0* active idle 10.1.1.33 9999/TCP configured
pol-k8s/0* active idle 10.1.1.24 80/TCP configured
prometheus-k8s/0* active idle 10.1.1.25 9090/TCP configured
ro-k8s/0* active idle 10.1.1.32 9090/TCP configured
ui-k8s/0* active idle 10.1.1.17 80/TCP configured
zookeeper-k8s/0* active idle 10.1.1.16 2181/TCP,2888/TCP,3888/TCP configured

ubuntu@osm-rel6-k8s:~$ kubectl -n osm get pods
NAME READY STATUS RESTARTS AGE
grafana-k8s-744f68b74b-nbnvh 1/1 Running 2 2d23h
grafana-k8s-operator-0 1/1 Running 4 3d1h
kafka-k8s-0 1/1 Running 16 2d23h
kafka-k8s-operator-0 1/1 Running 4 3d1h
lcm-k8s-0 0/1 CrashLoopBackOff 725 2d23h
lcm-k8s-operator-0 1/1 Running 4 3d1h
mariadb-k8s-0 0/1 Running 823 3d1h
mariadb-k8s-operator-0 1/1 Running 4 3d1h
mon-k8s-0 1/1 Running 3 2d23h
mon-k8s-operator-0 1/1 Running 4 3d1h
mongodb-k8s-0 1/1 Running 4 3d1h
mongodb-k8s-operator-0 1/1 Running 4 3d1h
nbi-k8s-0 1/1 Running 4 2d23h
nbi-k8s-operator-0 1/1 Running 4 3d1h
pol-k8s-0 1/1 Running 21 2d23h
pol-k8s-operator-0 1/1 Running 4 3d1h
prometheus-k8s-0 2/2 Running 4 2d23h
prometheus-k8s-operator-0 1/1 Running 4 3d1h
ro-k8s-0 1/1 Running 319 2d23h
ro-k8s-operator-0 1/1 Running 4 3d1h
ui-k8s-74b576cc99-4bwxz 1/1 Running 575 2d23h
ui-k8s-operator-0 1/1 Running 4 3d1h
zookeeper-k8s-0 1/1 Running 6 3d
zookeeper-k8s-operator-0 1/1 Running 4 3d1h

ubuntu@osm-rel6-k8s:~$ kubectl -n osm logs mariadb-k8s-0
2019-08-30 14:42:19 0 [Note] mysqld (mysqld 10.3.17-MariaDB-1:10.3.17+maria~bionic) starting as process 1 ...
2019-08-30 14:42:20 0 [Note] InnoDB: Using Linux native AIO
2019-08-30 14:42:20 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-08-30 14:42:20 0 [Note] InnoDB: Uses event mutexes
2019-08-30 14:42:20 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-08-30 14:42:20 0 [Note] InnoDB: Number of pools: 1
2019-08-30 14:42:20 0 [Note] InnoDB: Using SSE2 crc32 instructions
2019-08-30 14:42:20 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2019-08-30 14:42:20 0 [Note] InnoDB: Completed initialization of buffer pool
2019-08-30 14:42:20 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-08-30 14:42:22 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2019-08-30 14:42:22 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-08-30 14:42:22 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2019-08-30 14:42:22 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2019-08-30 14:42:23 0 [Note] InnoDB: 10.3.17 started; log sequence number 1639021; transaction id 21
2019-08-30 14:42:23 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2019-08-30 14:42:23 0 [Note] Plugin 'FEEDBACK' is disabled.
2019-08-30 14:42:23 0 [Note] Server socket created on IP: '::'.
2019-08-30 14:42:23 0 [Warning] 'user' entry 'root@mariadb-k8s-0' ignored in --skip-name-resolve mode.
2019-08-30 14:42:23 0 [Warning] 'user' entry '@mariadb-k8s-0' ignored in --skip-name-resolve mode.
2019-08-30 14:42:23 0 [Warning] 'proxies_priv' entry '@% root@mariadb-k8s-0' ignored in --skip-name-resolve mode.
2019-08-30 14:42:23 0 [Note] Reading of all Master_info entries succeeded
2019-08-30 14:42:23 0 [Note] Added new Master_info '' to hash table
2019-08-30 14:42:23 0 [Note] mysqld: ready for connections.
Version: '10.3.17-MariaDB-1:10.3.17+maria~bionic' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
2019-08-30 14:42:23 0 [Note] InnoDB: Buffer pool(s) load completed at 190830 14:42:23
2019-08-30 14:42:52 8 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:43:03 9 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:43:13 10 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:43:22 11 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:43:32 12 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:43:43 13 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:43:52 14 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2019-08-30 14:44:03 15 [Warning] Access denied for user 'root'@'localhost' (using password: YES)

ubuntu@osm-rel6-k8s:~$ kubectl -n osm describe pods mariadb-k8s-0
Name: mariadb-k8s-0
Namespace: osm
Priority: 0
PriorityClassName: <none>
Node: osm-rel6-k8s/10.20.30.66
Start Time: Tue, 27 Aug 2019 13:40:19 +0000
Labels: controller-revision-hash=mariadb-k8s-7bd7799754
                    juju-app=mariadb-k8s
                    statefulset.kubernetes.io/pod-name=mariadb-k8s-0
Annotations: apparmor.security.beta.kubernetes.io/pod: runtime/default
                    juju.io/controller: 91c96579-83a7-4d43-8101-196ca5a41f5c
                    juju.io/model: 47126947-f4af-4749-8a67-adc67a524aa1
                    seccomp.security.beta.kubernetes.io/pod: docker/default
Status: Running
IP: 10.1.1.31
Controlled By: StatefulSet/mariadb-k8s
Containers:
  mariadb-k8s:
    Container ID: containerd://f3744cbf06b6f8e6284fa8914286c093d61ea0e71fa835b25d3e45d9d8f45f2f
    Image: registry.jujucharms.com/charmed-osm/mariadb-k8s/mariadb-image@sha256:779b03a1b7b407644a0526632631d2395716bbb2388572833dc99a57fd319209
    Image ID: registry.jujucharms.com/charmed-osm/mariadb-k8s/mariadb-image@sha256:779b03a1b7b407644a0526632631d2395716bbb2388572833dc99a57fd319209
    Port: 3306/TCP
    Host Port: 0/TCP
    State: Running
      Started: Fri, 30 Aug 2019 14:42:14 +0000
    Last State: Terminated
      Reason: Completed
      Exit Code: 0
      Started: Fri, 30 Aug 2019 14:34:28 +0000
      Finished: Fri, 30 Aug 2019 14:37:01 +0000
    Ready: False
    Restart Count: 823
    Liveness: exec [sh -c exec mysqladmin status -uroot -p$MARIADB_ROOT_PASSWORD] delay=120s timeout=1s period=10s #success=1 #failure=3
    Readiness: exec [sh -c exec mysqladmin status -uroot -p$MARIADB_ROOT_PASSWORD] delay=30s timeout=1s period=10s #success=1 #failure=3
    Environment:
      MARIADB_DATABASE: database
      MARIADB_PASSWORD: manopw
      MARIADB_ROOT_PASSWORD: osm4u
      MARIADB_USER: mano
    Mounts:
      /var/lib/mysql from database-d3baa19e (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qnbsj (ro)
Conditions:
  Type Status
  Initialized True
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  database-d3baa19e:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: database-d3baa19e-mariadb-k8s-0
    ReadOnly: false
  default-token-qnbsj:
    Type: Secret (a volume populated by a Secret)
    SecretName: default-token-qnbsj
    Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning Unhealthy 49m (x530 over 47h) kubelet, osm-rel6-k8s (combined from similar events): Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to load task: no running task found: process 3caf4aca446c73c2456bf3ac210e9d7ee1db0cb450c17edc20989e7bec6c13fb does not exist: not found
  Warning DNSConfigForming 44m (x8790 over 2d) kubelet, osm-rel6-k8s Search Line limits were exceeded, some search paths have been omitted, the applied search line is: osm.svc.cluster.local svc.cluster.local cluster.local lb.videotron.ca ip.videotron.ca int.videotron.com
  Warning Unhealthy 9m38s (x6270 over 2d) kubelet, osm-rel6-k8s Readiness probe failed: mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
  Warning BackOff 4m51s (x6758 over 2d) kubelet, osm-rel6-k8s Back-off restarting failed container

ubuntu@osm-rel6-k8s:~$ kubectl -n osm get pods mariadb-k8s-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    apparmor.security.beta.kubernetes.io/pod: runtime/default
    juju.io/controller: 91c96579-83a7-4d43-8101-196ca5a41f5c
    juju.io/model: 47126947-f4af-4749-8a67-adc67a524aa1
    seccomp.security.beta.kubernetes.io/pod: docker/default
  creationTimestamp: "2019-08-27T13:40:13Z"
  generateName: mariadb-k8s-
  labels:
    controller-revision-hash: mariadb-k8s-7bd7799754
    juju-app: mariadb-k8s
    statefulset.kubernetes.io/pod-name: mariadb-k8s-0
  name: mariadb-k8s-0
  namespace: osm
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: mariadb-k8s
    uid: 321bf25b-c8d0-11e9-ba51-fa163e73b360
  resourceVersion: "398238"
  selfLink: /api/v1/namespaces/osm/pods/mariadb-k8s-0
  uid: 328162df-c8d0-11e9-ba51-fa163e73b360
spec:
  containers:
  - env:
    - name: MARIADB_DATABASE
      value: database
    - name: MARIADB_PASSWORD
      value: manopw
    - name: MARIADB_ROOT_PASSWORD
      value: osm4u
    - name: MARIADB_USER
      value: mano
    image: registry.jujucharms.com/charmed-osm/mariadb-k8s/mariadb-image@sha256:779b03a1b7b407644a0526632631d2395716bbb2388572833dc99a57fd319209
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - sh
        - -c
        - exec mysqladmin status -uroot -p$MARIADB_ROOT_PASSWORD
      failureThreshold: 3
      initialDelaySeconds: 120
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: mariadb-k8s
    ports:
    - containerPort: 3306
      name: main
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - sh
        - -c
        - exec mysqladmin status -uroot -p$MARIADB_ROOT_PASSWORD
      failureThreshold: 3
      initialDelaySeconds: 30
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: false
      runAsNonRoot: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/mysql
      name: database-d3baa19e
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-qnbsj
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: mariadb-k8s-0
  imagePullSecrets:
  - name: mariadb-k8s-mariadb-k8s-secret
  nodeName: osm-rel6-k8s
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: mariadb-k8s-endpoints
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: database-d3baa19e
    persistentVolumeClaim:
      claimName: database-d3baa19e-mariadb-k8s-0
  - name: default-token-qnbsj
    secret:
      defaultMode: 420
      secretName: default-token-qnbsj
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-08-27T13:40:19Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2019-08-27T13:40:19Z"
    message: 'containers with unready status: [mariadb-k8s]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2019-08-27T13:40:19Z"
    message: 'containers with unready status: [mariadb-k8s]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2019-08-27T13:40:18Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://c25d93e861808afc86f07489eced6de5c2fcf79c86b9e76182f0f41cdf28af04
    image: sha256:fce130b42e0be7ec85c3cae68c4dd90ea043bca2b01146161d7f312add05360b
    imageID: registry.jujucharms.com/charmed-osm/mariadb-k8s/mariadb-image@sha256:779b03a1b7b407644a0526632631d2395716bbb2388572833dc99a57fd319209
    lastState:
      terminated:
        containerID: containerd://c25d93e861808afc86f07489eced6de5c2fcf79c86b9e76182f0f41cdf28af04
        exitCode: 0
        finishedAt: "2019-08-30T14:47:23Z"
        reason: Completed
        startedAt: "2019-08-30T14:44:51Z"
    name: mariadb-k8s
    ready: false
    restartCount: 824
    state:
      waiting:
        message: Back-off 5m0s restarting failed container=mariadb-k8s pod=mariadb-k8s-0_osm(328162df-c8d0-11e9-ba51-fa163e73b360)
        reason: CrashLoopBackOff
  hostIP: 10.20.30.66
  phase: Running
  podIP: 10.1.1.31
  qosClass: BestEffort
  startTime: "2019-08-27T13:40:19Z"

[guilag (guillaume-lagace)]

Here is my ubuntu version:
ubuntu@osm-rel6-k8s:~$ cat /etc/os-release
NAME=“Ubuntu”
VERSION=“18.04.2 LTS (Bionic Beaver)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 18.04.2 LTS”
VERSION_ID=“18.04”
…
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Here are my ressources:
Running in a VM in OpenStack
RAM 12GB
VCPUs 20 VCPU
Disk 100GB

[guilag (guillaume-lagace)]

I redid the installation from scratch, this time mariadb seem ok but now its ro, lcm and ui that keep waiting. I believe because lcm depends on ro, and ro and ui can't connect to mariadb. See the logs below that show that ip:port of mariadb is up !

What can I do from here ?

ubuntu@osm-k8s:~$ kubectl -n osm get pods
NAME READY STATUS RESTARTS AGE
grafana-k8s-6dbf74cdf-48pjn 1/1 Running 3 4h41m
grafana-k8s-operator-0 1/1 Running 3 4h59m
kafka-k8s-0 1/1 Running 3 4h42m
kafka-k8s-operator-0 1/1 Running 3 4h58m
lcm-k8s-0 0/1 CrashLoopBackOff 30 3h10m
lcm-k8s-operator-0 1/1 Running 3 4h58m
mariadb-k8s-0 1/1 Running 4 4h54m
mariadb-k8s-operator-0 1/1 Running 3 4h58m
mon-k8s-0 1/1 Running 0 3h10m
mon-k8s-operator-0 1/1 Running 3 4h58m
mongodb-k8s-0 1/1 Running 3 4h53m
mongodb-k8s-operator-0 1/1 Running 3 4h57m
nbi-k8s-0 1/1 Running 0 3h10m
nbi-k8s-operator-0 1/1 Running 3 4h57m
pol-k8s-0 1/1 Running 4 3h10m
pol-k8s-operator-0 1/1 Running 3 4h56m
prometheus-k8s-0 2/2 Running 7 4h50m
prometheus-k8s-operator-0 1/1 Running 3 4h56m
ro-k8s-0 0/1 CrashLoopBackOff 27 4h39m
ro-k8s-operator-0 1/1 Running 3 4h55m
ui-k8s-548fff4886-tc9wd 0/1 CrashLoopBackOff 39 3h8m
ui-k8s-operator-0 1/1 Running 3 4h55m
zookeeper-k8s-0 1/1 Running 5 4h49m
zookeeper-k8s-operator-0 1/1 Running 3 4h54m

ubuntu@osm-k8s:~$ kubectl -n osm logs ro-k8s-0
1/4 Apply config
2/4 Wait for db up
Wait until 120 seconds for MySQL mano Server 10.152.183.37:3306
........................................................................................................................
Can not connect to database 10.152.183.37:3306 during 120 sec

ubuntu@osm-k8s:~$ kubectl -n osm logs ui-k8s-548fff4886-tc9wd
Wait until 120 seconds for MySQL mano Server 10.152.183.37:3306
.............................................................................

ubuntu@osm-k8s:~$ juju status
Model Controller Cloud/Region Version SLA Timestamp
osm osm-on-k8s microk8s/localhost 2.6.8 unsupported 22:45:36Z

App Version Status Scale Charm Store Rev OS Address Notes
grafana-k8s active 1 grafana-k8s jujucharms 21 kubernetes 10.152.183.223
kafka-k8s active 1 kafka-k8s jujucharms 9 kubernetes 10.152.183.186
lcm-k8s ...