[R5.0-k8s]: Kube DNS doesn't come up in multi interface setup

Bug #1767094 reported by Pulkit Tandon on 2018-04-26
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R5.0
Fix Released
Critical
Yuvaraja Mariappan
Trunk
Fix Released
Critical
Yuvaraja Mariappan

Bug Description

In multi interface setup, Kube DNS will continue to restart due to probe failures.

There are 2 reasons for that:
1. By default, firewalld is enabled.
2. Kubelet is using management interface.
   When it sends probe, it sends via management interface.
   Management interface do not have reachability to Control data interface.
   Thus the readiness and Liveliness probes are missed and DNS keeps on restarting.

Workaround to this issue is:
1. Disable probes
2. The 2 networks should be reachable

Pulkit Tandon (pulkitt) on 2018-04-26
tags: added: releasenote
Pulkit Tandon (pulkitt) on 2018-07-02
tags: added: sanityblocker
Pulkit Tandon (pulkitt) on 2018-07-16
tags: removed: sanityblocker
Jeba Paulaiyan (jebap) on 2018-08-08
tags: added: k8s
Venkatesh Velpula (vvelpula) wrote :

Nodeport service failure also tracked under the same bug as the root cause looks to be the same.

thanks
-Venky

tags: added: sanityblocker
Yuvaraja Mariappan (ymariappan) wrote :

route has to be added for pod-network to reach vhost0 and gateway should be vrotuer gateway.
For example, 10.32.0.0/12 is the pod-network and vrouter gateway is 192.168.10.100, the below
command has to be added in all the compute-nodes.

"route add -net 10.32.0.0/12 gw 192.168.10.100 dev vhost0"

It should be taken care in ansible. Reassigning to ram.

Thanks,
Yuvaraja

Ramprakash R (ramprakash) wrote :

It looks like this is best done inside the agent container as the status of the vhost0 device is not apparent from inside the deployer playbooks. Reassigning to Yuvaraja.

Review in progress for https://review.opencontrail.org/45467
Submitter: Yuvaraja Mariappan

Review in progress for https://review.opencontrail.org/45468
Submitter: Yuvaraja Mariappan

Review in progress for https://review.opencontrail.org/45467
Submitter: Yuvaraja Mariappan

Review in progress for https://review.opencontrail.org/45468
Submitter: Yuvaraja Mariappan

Review in progress for https://review.opencontrail.org/45467
Submitter: Yuvaraja Mariappan

Review in progress for https://review.opencontrail.org/45468
Submitter: Yuvaraja Mariappan

Reviewed: https://review.opencontrail.org/45468
Committed: http://github.com/Juniper/contrail-container-builder/commit/0387135452b5e28beeab41e8a8f4d7e6c5f05047
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 0387135452b5e28beeab41e8a8f4d7e6c5f05047
Author: Yuvaraja Mariappan <email address hidden>
Date: Thu Aug 9 19:09:08 2018 -0700

Route added for pod_cidr to handle k8s probe in multi interface nodes

In mulitple interfaces nodes, kubelet may use mgmt interface
and vrouter may use the control data interface. If the default gw
is mgmt, kubelet probes wont go via vrouter. Added route for pod_cidr
to go via vhost0.

Change-Id: If74f7bfbc7cb3d03f43a7bd5003a5f85ba513c8f
Closes-bug: 1767094

Reviewed: https://review.opencontrail.org/45467
Committed: http://github.com/Juniper/contrail-container-builder/commit/68ab08ca0ae5314ea3f1e1937d1186903d537878
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 68ab08ca0ae5314ea3f1e1937d1186903d537878
Author: Yuvaraja Mariappan <email address hidden>
Date: Thu Aug 9 19:09:08 2018 -0700

Route added for pod_cidr to handle k8s probe in multi interface nodes

In mulitple interfaces nodes, kubelet may use mgmt interface
and vrouter may use the control data interface. If the default gw
is mgmt, kubelet probes wont go via vrouter. Added route for pod_cidr
to go via vhost0.

Change-Id: If74f7bfbc7cb3d03f43a7bd5003a5f85ba513c8f
Closes-bug: 1767094

Venkatesh Velpula (vvelpula) wrote :

Verified ..with 5.0 queens 221 image on ha multi interface setup .

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers