openstack-helm

Dep_check init containers fail when quay.io/airshipit/kubernetes-entrypoint:v1.0.0 is used

Bug #1852972 reported by Erkki-T. Peura
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openstack-helm
Invalid
Undecided
Unassigned

Bug Description

I'm deploying openstack-helm charts but init containers fail to finish due error
command.go:14: Cannot find a binary echo : exec: \"echo\": executable file not found in $PATH"
when quay.io/airshipit/kubernetes-entrypoint:v1.0.0 is used, with quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 they succeed to start

Here's example run with both images:

# grep dep_check ../openstack-helm-infra.git/ingress/values.yaml
    dep_check: quay.io/airshipit/kubernetes-entrypoint:v1.0.0
# dep_check: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1
      - dep_check

# ./tools/deployment/multinode/020-test-ingress.sh
+ for NAMESPACE in openstack
+ OSH_EXTRA_HELM_ARGS_INGRESS_NAMESPACE=OSH_EXTRA_HELM_ARGS_INGRESS_openstack
+ tee /tmp/ingress-openstack.yaml
pod:
  replicas:
    ingress: 2
    error_page: 2
+ helm upgrade --install ingress-openstack /root/src/opendev.org/openstack/openstack-helm-infra.git/ingress --namespace=openstack --values=/tmp/ingress-openstack.yaml
Release "ingress-openstack" does not exist. Installing it now.
NAME: ingress-openstack
LAST DEPLOYED: Mon Nov 18 09:14:53 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
ingress-bin 2 0s
ingress-conf 3 0s
ingress-services-tcp 0 0s
ingress-services-udp 0 0s

==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
ingress 0/2 2 0 0s
ingress-error-pages 0/2 2 0 0s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
ingress-64b68f5bc-jdftw 0/1 Init:0/1 0 0s
ingress-64b68f5bc-lmbnm 0/1 Init:0/1 0 0s
ingress-error-pages-58dd9bf45c-xjfz8 0/1 Init:0/1 0 0s
ingress-error-pages-58dd9bf45c-xwvd9 0/1 Init:0/1 0 0s

==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress ClusterIP 10.43.203.174 <none> 80/TCP,443/TCP,18080/TCP 0s
ingress-error-pages ClusterIP None <none> 80/TCP 0s
ingress-exporter ClusterIP 10.43.94.41 <none> 10254/TCP 0s

==> v1/ServiceAccount
NAME SECRETS AGE
ingress-error-pages 1 0s
ingress-openstack-ingress 1 0s

==> v1beta1/ClusterRole
NAME AGE
ingress-openstack-ingress 0s

==> v1beta1/ClusterRoleBinding
NAME AGE
ingress-openstack-ingress 0s

==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
openstack-ingress-openstack *.openstack.svc.cluster.local 80 0s

==> v1beta1/Role
NAME AGE
ingress-openstack-ingress 0s

==> v1beta1/RoleBinding
NAME AGE
ingress-openstack-ingress 0s

+ ./tools/deployment/common/wait-for-pods.sh openstack
Containers failed to start after 900 seconds

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-64b68f5bc-jdftw 0/1 Init:CrashLoopBackOff 7 15m 10.42.2.185 worker-2 <none> <none>
ingress-64b68f5bc-lmbnm 0/1 Init:CrashLoopBackOff 7 15m 10.42.7.205 worker-0 <none> <none>
ingress-error-pages-58dd9bf45c-xjfz8 0/1 Init:CrashLoopBackOff 7 15m 10.42.4.192 worker-1 <none> <none>
ingress-error-pages-58dd9bf45c-xwvd9 0/1 Init:CrashLoopBackOff 7 15m 10.42.2.184 worker-2 <none> <none>

Some pods are in pending state:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-64b68f5bc-jdftw 0/1 Init:CrashLoopBackOff 7 15m 10.42.2.185 worker-2 <none> <none>
ingress-64b68f5bc-lmbnm 0/1 Init:CrashLoopBackOff 7 15m 10.42.7.205 worker-0 <none> <none>
ingress-error-pages-58dd9bf45c-xjfz8 0/1 Init:CrashLoopBackOff 7 15m 10.42.4.192 worker-1 <none> <none>
ingress-error-pages-58dd9bf45c-xwvd9 0/1 Init:CrashLoopBackOff 7 15m 10.42.2.184 worker-2 <none> <none>

# kubectl describe pods -n openstack
Name: ingress-64b68f5bc-jdftw
Namespace: openstack
Priority: 0
Node: worker-2/172.16.0.23
Start Time: Mon, 18 Nov 2019 09:14:54 +0200
Labels: app=ingress-api
                application=ingress
                component=server
                pod-template-hash=64b68f5bc
                release_group=ingress-openstack
Annotations: cni.projectcalico.org/podIP: 10.42.2.185/32
                configmap-bin-hash: b2060f86a8af5df12c634a3ba72114545439eeae4c5e5d14acafc11849343e7c
                configmap-etc-hash: 369dbe44e608890f4d77cbb9c7dd70fe6688196a734ebe3ade647f049c87714b
                openstackhelm.openstack.org/release_uuid:
Status: Pending
IP: 10.42.2.185
IPs: <none>
Controlled By: ReplicaSet/ingress-64b68f5bc
Init Containers:
  init:
    Container ID: docker://bd2c7a97aa223dd09f14963ef5fbbaa728e69f161831a1bbd8ab860160864eb4
    Image: quay.io/airshipit/kubernetes-entrypoint:v1.0.0
    Image ID: docker-pullable://quay.io/airshipit/kubernetes-entrypoint@sha256:038a41765607d40faa5e3864a513efa28a874c7af81829e323418948ebe0857f
    Port: <none>
    Host Port: <none>
    Command:
      kubernetes-entrypoint
    State: Waiting
      Reason: CrashLoopBackOff
    Last State: Terminated
      Reason: Error
      Exit Code: 1
      Started: Mon, 18 Nov 2019 09:35:54 +0200
      Finished: Mon, 18 Nov 2019 09:35:54 +0200
    Ready: False
    Restart Count: 9
    Environment:
      POD_NAME: ingress-64b68f5bc-jdftw (v1:metadata.name)
      NAMESPACE: openstack (v1:metadata.namespace)
      INTERFACE_NAME: eth0
      PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
      DEPENDENCY_SERVICE:
      DEPENDENCY_DAEMONSET:
      DEPENDENCY_CONTAINER:
      DEPENDENCY_POD_JSON:
      DEPENDENCY_CUSTOM_RESOURCE:
      COMMAND: echo done
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-openstack-ingress-token-vmnvx (ro)
Containers:
  ingress:
    Container ID:
    Image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
    Image ID:
    Ports: 80/TCP, 443/TCP, 18080/TCP, 10254/TCP, 8181/TCP
    Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      /tmp/ingress-controller.sh
      start
    State: Waiting
      Reason: PodInitializing
    Ready: False
    Restart Count: 0
    Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness: http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME: ingress-64b68f5bc-jdftw (v1:metadata.name)
      POD_NAMESPACE: openstack (v1:metadata.namespace)
      PORT_HTTP: 80
      PORT_HTTPS: 443
      PORT_STATUS: 18080
      PORT_HEALTHZ: 10254
      DEFAULT_SERVER_PORT: 8181
      RELEASE_NAME: ingress-openstack
      ERROR_PAGE_SERVICE: ingress-error-pages
      INGRESS_CLASS: nginx
    Mounts:
      /tmp from pod-tmp (rw)
      /tmp/ingress-controller.sh from ingress-bin (ro,path="ingress-controller.sh")
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-openstack-ingress-token-vmnvx (ro)
Conditions:
  Type Status
  Initialized False
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  pod-tmp:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
  ingress-bin:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: ingress-bin
    Optional: false
  ingress-openstack-ingress-token-vmnvx:
    Type: Secret (a volume populated by a Secret)
    SecretName: ingress-openstack-ingress-token-vmnvx
    Optional: false
QoS Class: BestEffort
Node-Selectors: openstack-control-plane=enabled
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 22m default-scheduler Successfully assigned openstack/ingress-64b68f5bc-jdftw to worker-2
  Normal Pulled 21m (x5 over 22m) kubelet, worker-2 Container image "quay.io/airshipit/kubernetes-entrypoint:v1.0.0" already present on machine
  Normal Created 21m (x5 over 22m) kubelet, worker-2 Created container init
  Normal Started 21m (x5 over 22m) kubelet, worker-2 Started container init
  Warning BackOff 2m31s (x95 over 22m) kubelet, worker-2 Back-off restarting failed container

Name: ingress-64b68f5bc-lmbnm
Namespace: openstack
Priority: 0
Node: worker-0/172.16.0.4
Start Time: Mon, 18 Nov 2019 09:14:54 +0200
Labels: app=ingress-api
                application=ingress
                component=server
                pod-template-hash=64b68f5bc
                release_group=ingress-openstack
Annotations: cni.projectcalico.org/podIP: 10.42.7.205/32
                configmap-bin-hash: b2060f86a8af5df12c634a3ba72114545439eeae4c5e5d14acafc11849343e7c
                configmap-etc-hash: 369dbe44e608890f4d77cbb9c7dd70fe6688196a734ebe3ade647f049c87714b
                openstackhelm.openstack.org/release_uuid:
Status: Pending
IP: 10.42.7.205
IPs: <none>
Controlled By: ReplicaSet/ingress-64b68f5bc
Init Containers:
  init:
    Container ID: docker://f3a3df580aa919be358860f69b37ac6b7cfe9d375a6f79ca008138159a3d0797
    Image: quay.io/airshipit/kubernetes-entrypoint:v1.0.0
    Image ID: docker-pullable://quay.io/airshipit/kubernetes-entrypoint@sha256:038a41765607d40faa5e3864a513efa28a874c7af81829e323418948ebe0857f
    Port: <none>
    Host Port: <none>
    Command:
      kubernetes-entrypoint
    State: Waiting
      Reason: CrashLoopBackOff
    Last State: Terminated
      Reason: Error
      Exit Code: 1
      Started: Mon, 18 Nov 2019 09:36:16 +0200
      Finished: Mon, 18 Nov 2019 09:36:16 +0200
    Ready: False
    Restart Count: 9
    Environment:
      POD_NAME: ingress-64b68f5bc-lmbnm (v1:metadata.name)
      NAMESPACE: openstack (v1:metadata.namespace)
      INTERFACE_NAME: eth0
      PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
      DEPENDENCY_SERVICE:
      DEPENDENCY_DAEMONSET:
      DEPENDENCY_CONTAINER:
      DEPENDENCY_POD_JSON:
      DEPENDENCY_CUSTOM_RESOURCE:
      COMMAND: echo done
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-openstack-ingress-token-vmnvx (ro)
Containers:
  ingress:
    Container ID:
    Image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0
    Image ID:
    Ports: 80/TCP, 443/TCP, 18080/TCP, 10254/TCP, 8181/TCP
    Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      /tmp/ingress-controller.sh
      start
    State: Waiting
      Reason: PodInitializing
    Ready: False
    Restart Count: 0
    Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness: http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME: ingress-64b68f5bc-lmbnm (v1:metadata.name)
      POD_NAMESPACE: openstack (v1:metadata.namespace)
      PORT_HTTP: 80
      PORT_HTTPS: 443
      PORT_STATUS: 18080
      PORT_HEALTHZ: 10254
      DEFAULT_SERVER_PORT: 8181
      RELEASE_NAME: ingress-openstack
      ERROR_PAGE_SERVICE: ingress-error-pages
      INGRESS_CLASS: nginx
    Mounts:
      /tmp from pod-tmp (rw)
      /tmp/ingress-controller.sh from ingress-bin (ro,path="ingress-controller.sh")
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-openstack-ingress-token-vmnvx (ro)
Conditions:
  Type Status
  Initialized False
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  pod-tmp:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
  ingress-bin:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: ingress-bin
    Optional: false
  ingress-openstack-ingress-token-vmnvx:
    Type: Secret (a volume populated by a Secret)
    SecretName: ingress-openstack-ingress-token-vmnvx
    Optional: false
QoS Class: BestEffort
Node-Selectors: openstack-control-plane=enabled
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 22m default-scheduler Successfully assigned openstack/ingress-64b68f5bc-lmbnm to worker-0
  Normal Pulled 21m (x5 over 22m) kubelet, worker-0 Container image "quay.io/airshipit/kubernetes-entrypoint:v1.0.0" already present on machine
  Normal Created 21m (x5 over 22m) kubelet, worker-0 Created container init
  Normal Started 21m (x5 over 22m) kubelet, worker-0 Started container init
  Warning BackOff 2m35s (x95 over 22m) kubelet, worker-0 Back-off restarting failed container

Name: ingress-error-pages-58dd9bf45c-xjfz8
Namespace: openstack
Priority: 0
Node: worker-1/172.16.0.8
Start Time: Mon, 18 Nov 2019 09:14:54 +0200
Labels: application=ingress
                component=error-pages
                pod-template-hash=58dd9bf45c
                release_group=ingress-openstack
Annotations: cni.projectcalico.org/podIP: 10.42.4.192/32
                configmap-bin-hash: b2060f86a8af5df12c634a3ba72114545439eeae4c5e5d14acafc11849343e7c
                configmap-etc-hash: 369dbe44e608890f4d77cbb9c7dd70fe6688196a734ebe3ade647f049c87714b
                openstackhelm.openstack.org/release_uuid:
Status: Pending
IP: 10.42.4.192
IPs: <none>
Controlled By: ReplicaSet/ingress-error-pages-58dd9bf45c
Init Containers:
  init:
    Container ID: docker://3b18d1555a3602782726c11426263263857ebf97d3b3d7bdcf8e527dab3e302d
    Image: quay.io/airshipit/kubernetes-entrypoint:v1.0.0
    Image ID: docker-pullable://quay.io/airshipit/kubernetes-entrypoint@sha256:038a41765607d40faa5e3864a513efa28a874c7af81829e323418948ebe0857f
    Port: <none>
    Host Port: <none>
    Command:
      kubernetes-entrypoint
    State: Waiting
      Reason: CrashLoopBackOff
    Last State: Terminated
      Reason: Error
      Exit Code: 1
      Started: Mon, 18 Nov 2019 09:36:14 +0200
      Finished: Mon, 18 Nov 2019 09:36:14 +0200
    Ready: False
    Restart Count: 9
    Environment:
      POD_NAME: ingress-error-pages-58dd9bf45c-xjfz8 (v1:metadata.name)
      NAMESPACE: openstack (v1:metadata.namespace)
      INTERFACE_NAME: eth0
      PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
      DEPENDENCY_SERVICE:
      DEPENDENCY_DAEMONSET:
      DEPENDENCY_CONTAINER:
      DEPENDENCY_POD_JSON:
      DEPENDENCY_CUSTOM_RESOURCE:
      COMMAND: echo done
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-error-pages-token-5nqv6 (ro)
Containers:
  ingress-error-pages:
    Container ID:
    Image: gcr.io/google_containers/defaultbackend:1.0
    Image ID:
    Port: 8080/TCP
    Host Port: 0/TCP
    Command:
      /tmp/ingress-error-pages.sh
      start
    State: Waiting
      Reason: PodInitializing
    Ready: False
    Restart Count: 0
    Liveness: http-get http://:8080/healthz delay=30s timeout=5s period=10s #success=1 #failure=3
    Environment: <none>
    Mounts:
      /tmp from pod-tmp (rw)
      /tmp/ingress-error-pages.sh from ingress-bin (ro,path="ingress-error-pages.sh")
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-error-pages-token-5nqv6 (ro)
Conditions:
  Type Status
  Initialized False
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  pod-tmp:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
  ingress-bin:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: ingress-bin
    Optional: false
  ingress-error-pages-token-5nqv6:
    Type: Secret (a volume populated by a Secret)
    SecretName: ingress-error-pages-token-5nqv6
    Optional: false
QoS Class: BestEffort
Node-Selectors: openstack-control-plane=enabled
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 22m default-scheduler Successfully assigned openstack/ingress-error-pages-58dd9bf45c-xjfz8 to worker-1
  Normal Pulled 21m (x5 over 22m) kubelet, worker-1 Container image "quay.io/airshipit/kubernetes-entrypoint:v1.0.0" already present on machine
  Normal Created 21m (x5 over 22m) kubelet, worker-1 Created container init
  Normal Started 21m (x5 over 22m) kubelet, worker-1 Started container init
  Warning BackOff 2m35s (x95 over 22m) kubelet, worker-1 Back-off restarting failed container

Name: ingress-error-pages-58dd9bf45c-xwvd9
Namespace: openstack
Priority: 0
Node: worker-2/172.16.0.23
Start Time: Mon, 18 Nov 2019 09:14:54 +0200
Labels: application=ingress
                component=error-pages
                pod-template-hash=58dd9bf45c
                release_group=ingress-openstack
Annotations: cni.projectcalico.org/podIP: 10.42.2.184/32
                configmap-bin-hash: b2060f86a8af5df12c634a3ba72114545439eeae4c5e5d14acafc11849343e7c
                configmap-etc-hash: 369dbe44e608890f4d77cbb9c7dd70fe6688196a734ebe3ade647f049c87714b
                openstackhelm.openstack.org/release_uuid:
Status: Pending
IP: 10.42.2.184
IPs: <none>
Controlled By: ReplicaSet/ingress-error-pages-58dd9bf45c
Init Containers:
  init:
    Container ID: docker://4ad554e8eacd7c8f6216ff69b44691b8c75fe833fba7dd72862e2d5dfe8ce8bf
    Image: quay.io/airshipit/kubernetes-entrypoint:v1.0.0
    Image ID: docker-pullable://quay.io/airshipit/kubernetes-entrypoint@sha256:038a41765607d40faa5e3864a513efa28a874c7af81829e323418948ebe0857f
    Port: <none>
    Host Port: <none>
    Command:
      kubernetes-entrypoint
    State: Waiting
      Reason: CrashLoopBackOff
    Last State: Terminated
      Reason: Error
      Exit Code: 1
      Started: Mon, 18 Nov 2019 09:36:23 +0200
      Finished: Mon, 18 Nov 2019 09:36:23 +0200
    Ready: False
    Restart Count: 9
    Environment:
      POD_NAME: ingress-error-pages-58dd9bf45c-xwvd9 (v1:metadata.name)
      NAMESPACE: openstack (v1:metadata.namespace)
      INTERFACE_NAME: eth0
      PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/
      DEPENDENCY_SERVICE:
      DEPENDENCY_DAEMONSET:
      DEPENDENCY_CONTAINER:
      DEPENDENCY_POD_JSON:
      DEPENDENCY_CUSTOM_RESOURCE:
      COMMAND: echo done
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-error-pages-token-5nqv6 (ro)
Containers:
  ingress-error-pages:
    Container ID:
    Image: gcr.io/google_containers/defaultbackend:1.0
    Image ID:
    Port: 8080/TCP
    Host Port: 0/TCP
    Command:
      /tmp/ingress-error-pages.sh
      start
    State: Waiting
      Reason: PodInitializing
    Ready: False
    Restart Count: 0
    Liveness: http-get http://:8080/healthz delay=30s timeout=5s period=10s #success=1 #failure=3
    Environment: <none>
    Mounts:
      /tmp from pod-tmp (rw)
      /tmp/ingress-error-pages.sh from ingress-bin (ro,path="ingress-error-pages.sh")
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-error-pages-token-5nqv6 (ro)
Conditions:
  Type Status
  Initialized False
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  pod-tmp:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
  ingress-bin:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: ingress-bin
    Optional: false
  ingress-error-pages-token-5nqv6:
    Type: Secret (a volume populated by a Secret)
    SecretName: ingress-error-pages-token-5nqv6
    Optional: false
QoS Class: BestEffort
Node-Selectors: openstack-control-plane=enabled
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 22m default-scheduler Successfully assigned openstack/ingress-error-pages-58dd9bf45c-xwvd9 to worker-2
  Normal Pulled 21m (x5 over 22m) kubelet, worker-2 Container image "quay.io/airshipit/kubernetes-entrypoint:v1.0.0" already present on machine
  Normal Created 21m (x5 over 22m) kubelet, worker-2 Created container init
  Normal Started 21m (x5 over 22m) kubelet, worker-2 Started container init
  Warning BackOff 2m32s (x96 over 22m) kubelet, worker-2 Back-off restarting failed container

Init pod log:

{
...
  "_source": {
    "message": "Entrypoint ERROR: 2019/11/18 07:31:12 command.go:14: Cannot find a binary echo : exec: \"echo\": executable file not found in $PATH",
    "@version": "1",
...
    },
...
    "kubernetes": {
      "labels": {
        "component": "error-pages",
        "pod-template-hash": "58dd9bf45c",
        "application": "ingress",
        "release_group": "ingress-openstack"
      },
      "namespace": "openstack",
      "container": {
        "name": "init"
      },
      "replicaset": {
        "name": "ingress-error-pages-58dd9bf45c"
      },
      "node": {
        "name": "worker-2"
      },
      "pod": {
        "uid": "1e76ae51-09d3-11ea-a1e9-fa163eca1156",
        "name": "ingress-error-pages-58dd9bf45c-xwvd9"
      }
...

# grep dep_check ../openstack-helm-infra.git/ingress/values.yaml
# dep_check: quay.io/airshipit/kubernetes-entrypoint:v1.0.0
    dep_check: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1
      - dep_check

# ./tools/deployment/multinode/020-nesc-ingress.sh
+ for NAMESPACE in openstack
+ OSH_EXTRA_HELM_ARGS_INGRESS_NAMESPACE=OSH_EXTRA_HELM_ARGS_INGRESS_openstack
+ tee /tmp/ingress-openstack.yaml
pod:
  replicas:
    ingress: 2
    error_page: 2
+ helm upgrade --install ingress-openstack /root/src/opendev.org/openstack/openstack-helm-infra.git/ingress --namespace=openstack --values=/tmp/ingress-openstack.yaml
Release "ingress-openstack" does not exist. Installing it now.
NAME: ingress-openstack
LAST DEPLOYED: Mon Nov 18 09:40:17 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
ingress-bin 2 1s
ingress-conf 3 1s
ingress-services-tcp 0 1s
ingress-services-udp 0 1s

==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
ingress 0/2 2 0 1s
ingress-error-pages 0/2 2 0 1s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
ingress-867ffd5c77-lskzg 0/1 Init:0/1 0 1s
ingress-867ffd5c77-vht82 0/1 Init:0/1 0 1s
ingress-error-pages-895cc9d75-8zgsw 0/1 Init:0/1 0 1s
ingress-error-pages-895cc9d75-kbdbp 0/1 Init:0/1 0 1s

==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress ClusterIP 10.43.227.98 <none> 80/TCP,443/TCP,18080/TCP 1s
ingress-error-pages ClusterIP None <none> 80/TCP 1s
ingress-exporter ClusterIP 10.43.252.28 <none> 10254/TCP 1s

==> v1/ServiceAccount
NAME SECRETS AGE
ingress-error-pages 1 1s
ingress-openstack-ingress 1 1s

==> v1beta1/ClusterRole
NAME AGE
ingress-openstack-ingress 1s

==> v1beta1/ClusterRoleBinding
NAME AGE
ingress-openstack-ingress 1s

==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
openstack-ingress-openstack *.openstack.svc.cluster.local 80 1s

==> v1beta1/Role
NAME AGE
ingress-openstack-ingress 1s

==> v1beta1/RoleBinding
NAME AGE
ingress-openstack-ingress 1s

+ ./tools/deployment/common/wait-for-pods.sh openstack
+ helm status ingress-openstack
LAST DEPLOYED: Mon Nov 18 09:40:17 2019
NAMESPACE: openstack
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
ingress-bin 2 17s
ingress-conf 3 17s
ingress-services-tcp 0 17s
ingress-services-udp 0 17s

==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
ingress 2/2 2 2 17s
ingress-error-pages 2/2 2 2 17s

==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
ingress-867ffd5c77-lskzg 1/1 Running 0 17s
ingress-867ffd5c77-vht82 1/1 Running 0 17s
ingress-error-pages-895cc9d75-8zgsw 1/1 Running 0 17s
ingress-error-pages-895cc9d75-kbdbp 1/1 Running 0 17s

==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress ClusterIP 10.43.227.98 <none> 80/TCP,443/TCP,18080/TCP 17s
ingress-error-pages ClusterIP None <none> 80/TCP 17s
ingress-exporter ClusterIP 10.43.252.28 <none> 10254/TCP 17s

==> v1/ServiceAccount
NAME SECRETS AGE
ingress-error-pages 1 17s
ingress-openstack-ingress 1 17s

==> v1beta1/ClusterRole
NAME AGE
ingress-openstack-ingress 17s

==> v1beta1/ClusterRoleBinding
NAME AGE
ingress-openstack-ingress 17s

==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
openstack-ingress-openstack *.openstack.svc.cluster.local 80 17s

==> v1beta1/Role
NAME AGE
ingress-openstack-ingress 17s

==> v1beta1/RoleBinding
NAME AGE
ingress-openstack-ingress 17s

# kubectl get pods -n openstack
NAME READY STATUS RESTARTS AGE
ingress-867ffd5c77-lskzg 1/1 Running 0 39s
ingress-867ffd5c77-vht82 1/1 Running 0 39s
ingress-error-pages-895cc9d75-8zgsw 1/1 Running 0 39s
ingress-error-pages-895cc9d75-kbdbp 1/1 Running 0 39s

Revision history for this message
Lars Fenneberg (linefeed) wrote :

I think you need a newer version of helm-toolkit/templates/snippets/kubernetes_entrypoint_init_container.tpl to use the new container image. The relevant change is part of this commit: https://github.com/openstack/openstack-helm-infra/commit/b50fae62a4ad0992ce877cd632800e1eed5f71a9#diff-dbbe99bc82a076d89ce11d4423d32cd7

Gage Hugo (gagehugo)
Changed in openstack-helm:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.