libvirt container stuck in Init:0/1

[Nikolas Hermanns]

Following http://openstack-helm.readthedocs.io/en/latest/install/developer/all-in-one.html
although ceph is not specified as a storage backend the libvirt helm chart needs it. So the all in one page does not tell how to setup this.

ubuntu-1604:~/openstack-helm$ kubectl get all --namespace=openstack
NAME READY STATUS RESTARTS AGE
po/etcd-2376361164-b47v0 1/1 Running 0 7m
po/ingress-api-1500747239-bwvhm 1/1 Running 0 7m
po/ingress-error-pages-2146872468-bx5k7 1/1 Running 0 7m
po/libvirt-q311w 0/1 Init:0/1 0 7m
po/mariadb-0 1/1 Running 0 7m
po/mariadb-1 1/1 Running 0 5m
po/mariadb-2 1/1 Running 0 5m
po/memcached-3845003049-4zjj7 1/1 Running 0 7m
po/openvswitch-db-j0lrr 1/1 Running 0 7m
po/openvswitch-vswitchd-s0bd9 1/1 Running 0 7m
po/rabbitmq-2694076863-3h94w 1/1 Running 0 7m
po/rabbitmq-2694076863-mkqds 1/1 Running 0 7m
po/rabbitmq-2694076863-wxqt4 1/1 Running 0 7m

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/etcd 10.106.18.7 <none> 2379/TCP 7m
svc/ingress-error-pages None <none> 80/TCP 7m
svc/mariadb 10.111.227.228 <none> 3306/TCP 7m
svc/mariadb-discovery None <none> 3306/TCP 7m
svc/memcached 10.103.244.20 <none> 11211/TCP 7m
svc/openstack 10.102.123.89 <none> 80/TCP,443/TCP 7m
svc/rabbitmq 10.97.15.216 <none> 5672/TCP,15672/TCP 7m

NAME DESIRED CURRENT AGE
statefulsets/mariadb 3 3 7m

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/etcd 1 1 1 1 7m
deploy/ingress-api 1 1 1 1 7m
deploy/ingress-error-pages 1 1 1 1 7m
deploy/memcached 1 1 1 1 7m
deploy/rabbitmq 3 3 3 3 7m

NAME DESIRED CURRENT READY AGE
rs/etcd-2376361164 1 1 1 7m
rs/ingress-api-1500747239 1 1 1 7m
rs/ingress-error-pages-2146872468 1 1 1 7m
rs/memcached-3845003049 1 1 1 7m
rs/rabbitmq-2694076863 3 3 3 7m

ubuntu-1604:~/openstack-helm$ kubectl describe po/libvirt-q311w --namespace=openstack
Name: libvirt-q311w
Namespace: openstack
Node: ubuntu-1604/192.168.122.195
Start Time: Tue, 26 Sep 2017 16:37:17 +0200
Labels: application=libvirt
  component=libvirt
  controller-revision-hash=202204736
  pod-template-generation=1
  release_group=libvirt
Annotations: configmap-bin-hash=bcdb159aceea0fa683079229c278929c827083288ad314a94f8565b10f22d486
  configmap-etc-hash=a258cd7c9c4446a3946a7b30fba0beb6f84c8647bf94d5577db91d9a46e9038d
  kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"DaemonSet","namespace":"openstack","name":"libvirt","uid":"31c02ee8-a2c8-11e7-b0f8-525400920585","apiVersi...
Status: Pending
IP: 192.168.122.195
Created By: DaemonSet/libvirt
Controlled By: DaemonSet/libvirt
Init Containers:
  ceph-keyring-placement:
    Container ID:
    Image: docker.io/kolla/ubuntu-source-nova-libvirt:3.0.3
    Image ID:
    Port: <none>
    Command:
      /tmp/ceph-keyring.sh
    State: Waiting
      Reason: PodInitializing
    Ready: False
    Restart Count: 0
    Environment:
      CEPH_CINDER_USER: admin
      LIBVIRT_CEPH_SECRET_UUID: 457eb676-33da-42ec-9a8c-9293d545c337
    Mounts:
      /etc/ceph from etcceph (rw)
      /tmp/ceph-keyring.sh from libvirt-bin (ro)
      /tmp/client-keyring from ceph-keyring (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-k3hjk (ro)
Containers:
  libvirt:
    Container ID:
    Image: docker.io/kolla/ubuntu-source-nova-libvirt:3.0.3
    Image ID:
    Port: <none>
    Command:
      /tmp/libvirt.sh
    State: Waiting
      Reason: PodInitializing
    Ready: False
    Restart Count: 0
    Environment:
      CEPH_CINDER_USER: admin
      LIBVIRT_CEPH_SECRET_UUID: 457eb676-33da-42ec-9a8c-9293d545c337
    Mounts:
      /dev from dev (rw)
      /etc/ceph from etcceph (rw)
      /etc/ceph/ceph.conf from ceph-etc (ro)
      /etc/libvirt/libvirtd.conf from libvirt-etc (ro)
      /etc/libvirt/qemu.conf from libvirt-etc (ro)
      /etc/machine-id from machine-id (ro)
      /lib/modules from libmodules (ro)
      /run from run (rw)
      /sys/fs/cgroup from cgroup (rw)
      /tmp/client-keyring from ceph-keyring (ro)
      /tmp/libvirt.sh from libvirt-bin (ro)
      /var/lib/libvirt from var-lib-libvirt (rw)
      /var/lib/nova from var-lib-nova (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-k3hjk (ro)
Conditions:
  Type Status
  Initialized False
  Ready False
  PodScheduled True
Volumes:
  libvirt-bin:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: libvirt-bin
    Optional: false
  libvirt-etc:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: libvirt-etc
    Optional: false
  etcceph:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  ceph-etc:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: ceph-etc
    Optional: false
  ceph-keyring:
    Type: Secret (a volume populated by a Secret)
    SecretName: pvc-ceph-client-key
    Optional: false
  libmodules:
    Type: HostPath (bare host directory volume)
    Path: /lib/modules
  var-lib-libvirt:
    Type: HostPath (bare host directory volume)
    Path: /var/lib/libvirt
  var-lib-nova:
    Type: HostPath (bare host directory volume)
    Path: /var/lib/nova
  run:
    Type: HostPath (bare host directory volume)
    Path: /run
  dev:
    Type: HostPath (bare host directory volume)
    Path: /dev
  cgroup:
    Type: HostPath (bare host directory volume)
    Path: /sys/fs/cgroup
  machine-id:
    Type: HostPath (bare host directory volume)
    Path: /etc/machine-id
  default-token-k3hjk:
    Type: Secret (a volume populated by a Secret)
    SecretName: default-token-k3hjk
    Optional: false
QoS Class: BestEffort
Node-Selectors: openstack-compute-node=enabled
Tolerations: node.alpha.kubernetes.io/notReady:NoExecute
  node.alpha.kubernetes.io/unreachable:NoExecute
Events:
  FirstSeen LastSeen Count From SubObjectPath Type Reason Message
  --------- -------- ----- ---- ------------- -------- ------ -------
  8m 8m 1 kubelet, ubuntu-1604 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "libvirt-bin"
  8m 8m 1 kubelet, ubuntu-1604 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "dev"
  8m 8m 1 kubelet, ubuntu-1604 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "cgroup"
  8m 8m 1 kubelet, ubuntu-1604 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "run"
  8m 8m 1 kubelet, ubuntu-1604 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "var-lib-nova"
  8m 8m 1 kubelet, ubuntu-1604 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "var-lib-libvirt"
  8m 8m 1 kubelet, ubuntu-1604 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "machine-id"
  8m 8m 1 kubelet, ubuntu-1604 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "etcceph"
  8m 8m 1 kubelet, ubuntu-1604 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "libmodules"
  8m 8m 2 kubelet, ubuntu-1604 Normal SuccessfulMountVolume (combined from similar events): MountVolume.SetUp succeeded for volume "default-token-k3hjk"
  6m 1m 3 kubelet, ubuntu-1604 Warning FailedMount Unable to mount volumes for pod "libvirt-q311w_openstack(31c25c20-a2c8-11e7-b0f8-525400920585)": timeout expired waiting for volumes to attach/mount for pod "openstack"/"libvirt-q311w". list of unattached/unmounted volumes=[ceph-etc ceph-keyring]
  6m 1m 3 kubelet, ubuntu-1604 Warning FailedSync Error syncing pod
  8m 1m 11 kubelet, ubuntu-1604 Warning FailedMount MountVolume.SetUp failed for volume "ceph-etc" : configmaps "ceph-etc" not found
  8m 1m 11 kubelet, ubuntu-1604 Warning FailedMount MountVolume.SetUp failed for volume "ceph-keyring" : secrets "pvc-ceph-client-key" not found

[Nikolas Hermanns (nikolas-hermanns)]

ok ceph seems to be there already from the deployment but the key seems to be not there:
ubuntu-1604:~/openstack-helm$ kubectl describe po ceph-bootstrap-56wrx --namespace=ceph
Name: ceph-bootstrap-56wrx
Namespace: ceph
Node: ubuntu-1604/192.168.122.195
Start Time: Tue, 26 Sep 2017 16:58:05 +0200
Labels: application=ceph
  component=bootstrap
  controller-uid=19d21121-a2cb-11e7-b0f8-525400920585
  job-name=ceph-bootstrap
  release_group=ceph
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"Job","namespace":"ceph","name":"ceph-bootstrap","uid":"19d21121-a2cb-11e7-b0f8-525400920585","apiVersion":...
Status: Pending
IP: 192.168.206.212
Created By: Job/ceph-bootstrap
Controlled By: Job/ceph-bootstrap
Init Containers:
  init:
    Container ID: docker://f58000c3b15a14521e777200a9362f5da0b37a6387894f5409507e0160e79a7e
    Image: docker.io/kolla/ubuntu-source-kubernetes-entrypoint:4.0.0
    Image ID: docker-pullable://kolla/ubuntu-source-kubernetes-entrypoint@sha256:75116ab2f9f65c5fc078e68ce7facd66c1c57496947f37b7209b32f94925e53b
    Port: <none>
    Command:
      kubernetes-entrypoint
    State: Running
      Started: Tue, 26 Sep 2017 16:58:42 +0200
    Ready: False
    Restart Count: 0
    Environment:
      POD_NAME: ceph-bootstrap-56wrx (v1:metadata.name)
      NAMESPACE: ceph (v1:metadata.namespace)
      INTERFACE_NAME: eth0
      DEPENDENCY_SERVICE: ceph-mon
      DEPENDENCY_JOBS:
      DEPENDENCY_DAEMONSET:
      DEPENDENCY_CONTAINER:
      COMMAND: echo done
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-zntwb (ro)
Containers:
  ceph-bootstrap:
    Container ID:
    Image: quay.io/attcomdev/ceph-daemon:tag-build-master-jewel-ubuntu-16.04
    Image ID:
    Port: <none>
    Command:
      /tmp/bootstrap.sh
    State: Waiting
      Reason: PodInitializing
    Ready: False
    Restart Count: 0
    Environment: <none>
    Mounts:
      /etc/ceph/ceph.client.admin.keyring from ceph-client-admin-keyring (ro)
      /etc/ceph/ceph.conf from ceph-etc (ro)
      /tmp/bootstrap.sh from ceph-bin (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-zntwb (ro)
Conditions:
  Type Status
  Initialized False
  Ready False
  PodScheduled True
Volumes:
  ceph-bin:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: ceph-bin
    Optional: false
  ceph-etc:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: ceph-etc
    Optional: false
  ceph-client-admin-keyring:
    Type: Secret (a volume populated by a Secret)
    SecretName: ceph-client-admin-keyring
    Optional: false
  default-token-zntwb:
    Type: Secret (a volume populated by a Secret)
    SecretName: default-token-zntwb
    Optional: false
QoS Class: BestEffort
Node-Selectors: openstack-control-plane=enabled
Tolerations: node.alpha.kubernetes.io/notReady:NoExecute for 300s
  node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  FirstSeen LastSeen Count From SubObjectPath Type Reason Message
  --------- -------- ----- ---- ------------- -------- ------ -------
  6m 6m 1 default-scheduler Norma...

[Pete Birley (portdirect)]

The libvirt chart should now be fixed, though more work is being done to harden this area.