kuryr-kubernetes

Kuryr-kubernetes services throws Dummy exception while creating a POD

Bug #1640446 reported by Preethi Dsilva
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kuryr-kubernetes
Invalid
Undecided
Ilya Chukhnakov

Bug Description

1.Created two node devstack k8s with kuryr setup.
2.The kubectl get nodes shows node to be ready
vagrant@container1:~$ kubectl get node
NAME STATUS AGE
container1 Ready 3h
container2 Ready 3h
vagrant@container1:~$
3.now tried creating the pod ,During the creation of the pod kuryr-kubernetes service throws below given error
4.On this error tried applying the review https://review.openstack.org/#/c/394066/ but no luck :(

Logs:
=====

ERROR in kuyr-kubernetes.log
------------------------------
09T08:30:29Z'}], u'containerStatuses': [{u'restartCount': 0, u'name': u'ubuntu', u'image': u'ubuntu:14.04', u'imageID': u'', u'state': {u'waiting': {u'reason': u'ContainerCreating'}}, u'ready': False, u'lastState': {}}], u'startTime': u'2016-11-09T08:30:29Z', u'hostIP': u'10.10.210.212'}, u'kind': u'Pod', u'spec': {u'dnsPolicy': u'ClusterFirst', u'securityContext': {}, u'serviceAccountName': u'default', u'serviceAccount': u'default', u'terminationGracePeriodSeconds': 30, u'restartPolicy': u'Always', u'volumes': [{u'secret': {u'secretName': u'default-token-95u3b'}, u'name': u'default-token-95u3b'}], u'containers': [{u'terminationMessagePath': u'/dev/termination-log', u'name': u'ubuntu', u'image': u'ubuntu:14.04', u'volumeMounts': [{u'readOnly': True, u'mountPath': u'/var/run/secrets/kubernetes.io/serviceaccount', u'name': u'default-token-95u3b'}], u'imagePullPolicy': u'IfNotPresent', u'resources': {}}], u'nodeName': u'minion2'}, u'apiVersion': u'v1', u'metadata': {u'name': u'ubuntu-1518966169-5vgvz', u'labels': {u'pod-template-hash': u'1518966169', u'run': u'ubuntu'}, u'namespace': u'default', u'resourceVersion': u'1882', u'generateName': u'ubuntu-1518966169-', u'creationTimestamp': u'2016-11-09T08:30:28Z', u'annotations': {u'kubernetes.io/created-by': u'{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"ubuntu-1518966169","uid":"c520eb0c-a656-11e6-a67d-080027e01b47","apiVersion":"extensions","resourceVersion":"1872"}}\n'}, u'selfLink': u'/api/v1/namespaces/default/pods/ubuntu-1518966169-5vgvz', u'uid': u'c532bf0c-a656-11e6-a67d-080027e01b47'}}, u'type': u'ADDED'}
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/logging.py", line 38, in __call__
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/retry.py", line 66, in __call__
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging ex.reraise = False
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging self.force_reraise()
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging six.reraise(self.type_, self.value, self.tb)
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/retry.py", line 61, in __call__
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/service.py", line 47, in __call__
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging raise Exception(_LE("Dummy exception %s") % self.event_seq)
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging Exception: Dummy exception 119
2016-11-09 11:03:42.492 24242 ERROR kuryr_kubernetes.handlers.logging
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'status': {u'loadBalancer': {}}, u'kind': u'Service', u'spec': {u'clusterIP': u'10.0.0.1', u'type': u'ClusterIP', u'ports': [{u'targetPort': 443, u'protocol': u'TCP', u'name': u'https', u'port': 443}], u'sessionAffinity': u'ClientIP'}, u'apiVersion': u'v1', u'metadata': {u'name': u'kubernetes', u'labels': {u'component': u'apiserver', u'provider': u'kubernetes'}, u'namespace': u'default', u'resourceVersion': u'10', u'creationTimestamp': u'2016-11-09T06:47:17Z', u'selfLink': u'/api/v1/namespaces/default/services/kubernetes', u'uid': u'5afd91d1-a648-11e6-807b-080027e01b47'}}, u'type': u'ADDED'}
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/logging.py", line 38, in __call__
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/retry.py", line 66, in __call__
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging ex.reraise = False
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging self.force_reraise()
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging six.reraise(self.type_, self.value, self.tb)
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/retry.py", line 61, in __call__
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/service.py", line 48, in __call__
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging super(DummyHandler, self).__call__(event)
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/k8s_base.py", line 62, in __call__
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging self.on_added(obj)
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/service.py", line 52, in on_added
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging event['object']['metadata']['selfLink'])
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging KeyError: 'object'
2016-11-09 11:04:34.404 24242 ERROR kuryr_kubernetes.handlers.logging
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'kind': u'Endpoints', u'subsets': [{u'addresses': [{u'ip': u'10.10.210.210'}], u'ports': [{u'protocol': u'TCP', u'name': u'https', u'port': 6443}]}], u'apiVersion': u'v1', u'metadata': {u'name': u'kubernetes', u'namespace': u'default', u'resourceVersion': u'11', u'creationTimestamp': u'2016-11-09T06:47:17Z', u'selfLink': u'/api/v1/namespaces/default/endpoints/kubernetes', u'uid': u'5b0d368c-a648-11e6-807b-080027e01b47'}}, u'type': u'ADDED'}
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/logging.py", line 38, in __call__
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/retry.py", line 66, in __call__
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging ex.reraise = False
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging self.force_reraise()
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging six.reraise(self.type_, self.value, self.tb)
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/retry.py", line 61, in __call__
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/service.py", line 47, in __call__
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging raise Exception(_LE("Dummy exception %s") % self.event_seq)
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging Exception: Dummy exception 121
2016-11-09 11:07:22.071 24242 ERROR kuryr_kubernetes.handlers.logging

I am attaching the local.conf used for the same

Master:
---------------
[[local|localrc]]
GIT_BASE=https://git.openstack.org
enable_plugin kuryr-kubernetes \
    https://git.openstack.org/openstack/kuryr-kubernetes

# If you do not want stacking to clone new versions of the enabled services,
# like for example when you did local modifications and need to ./unstack.sh
# and ./stack.sh again, uncomment the following
RECLONE="yes"
HOST_IP=10.10.100.206

# Log settings for better readability
LOGFILE=devstack.log
LOG_COLOR=False
# If you want the screen tabs logged in a specific location, you can use:
# SCREEN_LOGDIR="${HOME}/devstack_logs"

# Credentials
ADMIN_PASSWORD=pass
DATABASE_PASSWORD=pass
RABBIT_PASSWORD=pass
SERVICE_PASSWORD=pass
SERVICE_TOKEN=pass
# Enable Keystone v3
IDENTITY_API_VERSION=3

# In pro of speed and being lightweight, we will be explicit in regards to
# which services we enable
ENABLED_SERVICES=""

# Neutron services
enable_service neutron
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-svc

# Keystone
enable_service key

# dependencies
enable_service mysql
enable_service rabbit
# By default use all the services from the kuryr-kubernetes plugin

# Docker
# ======
# If you already have docker configured, running and with its socket writable
# by the stack user, you can omit the following line.
enable_service docker

# Etcd
# ====
# The default is for devstack to run etcd for you. You can select the image and
# version of it by uncommenting and modifying the following defaults.
# KURYR_ETCD_IMAGE="quay.io/coreos/etcd"
# KURYR_ETCD_VERSION="v3.0.8"
#
# You can select the listening and advertising client and peering Etcd
# addresses by uncommenting and changing from the following defaults:
# KURYR_ETCD_ADVERTISE_CLIENT_URL=http://my_host_ip:2379}
# KURYR_ETCD_ADVERTISE_PEER_URL=http://my_host_ip:2380}
# KURYR_ETCD_LISTEN_CLIENT_URL=http://0.0.0.0:2379}
# KURYR_ETCD_LISTEN_PEER_URL=http://0.0.0.0:2380}
#
# If you already have etcd configured and running, you can just comment out
enable_service etcd
# then uncomment and set the following line:
# KURYR_ETCD_CLIENT_URL="http://etcd_ip:etcd_client_port"

# Kubernetes
# ==========
#
# Kubernetes is run from the hyperkube docker image
# If you already have a Kubernetes deployment, you can use it instead and omit
# enabling the Kubernetes service (except Kubelet, which must be run by
# devstack so that it uses our development CNI driver.
#
# The default is, again, for devstack to run the Kubernetes services:
enable_service kubernetes-api
enable_service kubernetes-controller-manager
enable_service kubernetes-scheduler

# We use hyperkube to run the services. You can select the hyperkube image and/
# or version by uncommenting and setting the following ENV vars different
# to the following defaults:
# KURYR_HYPERKUBE_IMAGE="gcr.io/google_containers/hyperkube-amd64"
# KURYR_HYPERKUBE_VERSION="v1.3.7"
#
# If you have the 8080 port already bound to another service, you will need to
# have kubernetes API server bind to another port. In order to do that,
# uncomment and set a different port number in:
# KURYR_K8S_API_PORT="8080"
#
# If you want to test with a different range for the Cluster IPs uncomment and
# set the following ENV var to a different CIDR
# KURYR_K8S_CLUSTER_IP_RANGE="10.0.0.0/24"
#
# If, however, you are reusing an existing deployment, you should uncomment and
# set an ENV var so that the Kubelet devstack runs can find the API server:
# KURYR_K8S_API_URL="http://k8s_api_ip:k8s_api_port"
#

# Kubelet
# =======
#
# Kubelet should almost invariably be run by devstack
enable_service kubelet

# You can specify a different location for the hyperkube binary that will be
# extracted from the hyperkube container into the Host filesystem:
# KURYR_HYPERKUBE_BINARY=/usr/local/bin/hyperkube
#
# NOTE: KURYR_HYPERKUBE_IMAGE, KURYR_HYPERKUBE_VERSION also affect which
# the selected binary for the Kubelet.

# Kuryr watcher
# =============
#
# Just like the Kubelet, you'll want to have the watcher enabled. It is the
# part of the codebase that connects to the Kubernetes API server to read the
# resource events and convert them to Neutron actions
enable_service kuryr-kubernetes
minion:
-----===
[[local|localrc]]
GIT_BASE=https://git.openstack.org
enable_plugin kuryr-kubernetes \
    https://git.openstack.org/openstack/kuryr-kubernetes

# If you do not want stacking to clone new versions of the enabled services,
# like for example when you did local modifications and need to ./unstack.sh
# and ./stack.sh again, uncomment the following
RECLONE="yes"
RABBIT_HOST=10.10.100.206
Q_HOST=10.10.100.206
HOST_IP=10.10.100.207
DATA_IP=10.10.100.207
KEYSTONE_AUTH_HOST=10.10.100.206
KEYSTONE_SERVICE_HOST=10.10.100.206
MULTI_HOST=1

# Log settings for better readability
LOGFILE=devstack.log
LOG_COLOR=False
# If you want the screen tabs logged in a specific location, you can use:
# SCREEN_LOGDIR="${HOME}/devstack_logs"

# Credentials
ADMIN_PASSWORD=pass
DATABASE_PASSWORD=pass
RABBIT_PASSWORD=pass
SERVICE_PASSWORD=pass
SERVICE_TOKEN=pass
# Enable Keystone v3
IDENTITY_API_VERSION=3

# In pro of speed and being lightweight, we will be explicit in regards to
# which services we enable
ENABLED_SERVICES=""

# Neutron services
#enable_service neutron
enable_service q-agt
#enable_service q-dhcp
#enable_service q-l3
#enable_service q-svc
# Keystone
enable_service key

# dependencies
#enable_service mysql
enable_service rabbit

# By default use all the services from the kuryr-kubernetes plugin

# Docker
# ======
# If you already have docker configured, running and with its socket writable
# by the stack user, you can omit the following line.
enable_service docker

# Etcd
# ====
# The default is for devstack to run etcd for you. You can select the image and
# version of it by uncommenting and modifying the following defaults.
# KURYR_ETCD_IMAGE="quay.io/coreos/etcd"
# KURYR_ETCD_VERSION="v3.0.8"
#
# You can select the listening and advertising client and peering Etcd
# addresses by uncommenting and changing from the following defaults:
# KURYR_ETCD_ADVERTISE_CLIENT_URL=http://my_host_ip:2379}
# KURYR_ETCD_ADVERTISE_PEER_URL=http://my_host_ip:2380}
# KURYR_ETCD_LISTEN_CLIENT_URL=http://0.0.0.0:2379}
# KURYR_ETCD_LISTEN_PEER_URL=http://0.0.0.0:2380}
#
# If you already have etcd configured and running, you can just comment out
enable_service etcd
# then uncomment and set the following line:
# KURYR_ETCD_CLIENT_URL="http://etcd_ip:etcd_client_port"
# Kubernetes
# ==========
#
# Kubernetes is run from the hyperkube docker image
# If you already have a Kubernetes deployment, you can use it instead and omit
# enabling the Kubernetes service (except Kubelet, which must be run by
# devstack so that it uses our development CNI driver.
#
# The default is, again, for devstack to run the Kubernetes services:
#enable_service kubernetes-api
#enable_service kubernetes-controller-manager
#enable_service kubernetes-scheduler

# We use hyperkube to run the services. You can select the hyperkube image and/
# or version by uncommenting and setting the following ENV vars different
# to the following defaults:
# KURYR_HYPERKUBE_IMAGE="gcr.io/google_containers/hyperkube-amd64"
# KURYR_HYPERKUBE_VERSION="v1.3.7"
#
# If you have the 8080 port already bound to another service, you will need to
# have kubernetes API server bind to another port. In order to do that,
# uncomment and set a different port number in:
# KURYR_K8S_API_PORT="8080"
#
# If you want to test with a different range for the Cluster IPs uncomment and
# set the following ENV var to a different CIDR
# KURYR_K8S_CLUSTER_IP_RANGE="10.0.0.0/24"
#
# If, however, you are reusing an existing deployment, you should uncomment and
# set an ENV var so that the Kubelet devstack runs can find the API server:
KURYR_K8S_API_URL="http://10.10.100.206:8080"
#
# Kubelet
# =======
#
# Kubelet should almost invariably be run by devstack
enable_service kubelet

# You can specify a different location for the hyperkube binary that will be
# extracted from the hyperkube container into the Host filesystem:
# KURYR_HYPERKUBE_BINARY=/usr/local/bin/hyperkube
#
# NOTE: KURYR_HYPERKUBE_IMAGE, KURYR_HYPERKUBE_VERSION also affect which
# the selected binary for the Kubelet.

# Kuryr watcher
# =============
#
# Just like the Kubelet, you'll want to have the watcher enabled. It is the
# part of the codebase that connects to the Kubernetes API server to read the
# resource events and convert them to Neutron actions
enable_service kuryr-kubernetes

Ilya Chukhnakov (ichukhnakov)
Changed in kuryr-kubernetes:
assignee: nobody → Ilya Chukhnakov (ichukhnakov)
status: New → In Progress
Revision history for this message
Ilya Chukhnakov (ichukhnakov) wrote :

This is an expected behaviour. kuryr-kubernetes is currently under development. The version you are running does not implement specific handlers but uses a stub DummyHandler to test the logging/retrying aspects of the event handling process. These exceptions will go away once [1] is merged (note, however, that [1] only implements the controller-side of pod port handling and until the CNI driver is implemented, it will still produce some exceptions).

[1] https://review.openstack.org/#/c/376044

Revision history for this message
Ilya Chukhnakov (ichukhnakov) wrote :

The patch [1] was merged (the temporary code causing Dummy exceptions removed).

[1] https://review.openstack.org/#/c/376044/

Changed in kuryr-kubernetes:
status: In Progress → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.