Juniper Openstack

[k8s-R5.0-Single-Yaml-Prov]: k8s provisioning using single yaml failed

Bug #1752657 reported by Pulkit Tandon on 2018-03-01

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R5.0	Fix Committed	Critical	Sachchidanand Vaidya	Juniper Openstack r5.0.0
Trunk	In Progress	Critical	Sachchidanand Vaidya	Juniper Openstack r5.1.0

Bug Description

BUG Tempelate:

Configuration:
K8s 1.9.2
Docker version 17.03.1-ce
Centos -7.4

Setup:
3 node setup.
1 Kube master. 1 Controller.
2 Agent+ K8s slaves

Followed steps suggested by Sachin to provision k8s contrail cluster using single yaml provisioning.
Agent containers keep on restarting and do not get spawned.
State of agent containers was:
kube-system contrail-agent-czx2x 1/2 CrashLoopBackOff 7 8m
kube-system contrail-agent-ffmz5 1/2 CrashLoopBackOff 6 8m

There are multiple problem leading to this state:

1. By default in contrail.yaml, the PHYSICAL_INTERFACE: is always set to:
PHYSICAL_INTERFACE: eno1
At that time the agent will give following errors:
2018-03-01 Thu 16:49:30:061.102 UTC nodec58 [Thread 140193993205952, Pid 1]: Error reading interface <eno1>. Error number 19 : No such device
2018-03-01 Thu 16:54:33:048.405 UTC nodec58 [Thread 140017763444928, Pid 1]: Error reading interface <eno1>. Error number 19 : No such device

2. To correct this, this line can be deleted manually from contrail.yaml and the file can be loaded again.
This resulted in losing the connectivity to agents as they lose default routes.
Workaround to this is adding the default routes manually.

3. After above problems are corrected, it lands up in an issue to connect to the API server.
ERROR Tue Feb 27 06:53:43 UTC 2018: Config API servers 10.204.217.108 are not responding on port 8082. Exiting...
WARNING Tue Feb 27 06:53:43 UTC 2018: Some of Config API servers 10.204.217.108 are not responding on port 8082.
Provisioning: provision_vrouter.py --oper add --host_name nodec60 --host_ip 10.204.217.100: 0/10
Traceback (most recent call last):
  File “/opt/contrail/utils/provision_vrouter.py”, line 274, in <module>
    main()
  File “/opt/contrail/utils/provision_vrouter.py”, line 270, in main
    VrouterProvisioner(args_str)
  File “/opt/contrail/utils/provision_vrouter.py”, line 35, in __init__
    api_server_use_ssl=self._args.api_server_use_ssl)
  File “/opt/contrail/utils/vnc_admin_api.py”, line 35, in __init__
    auth_host=self.auth_host)
  File “/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py”, line 460, in __init__
    retry_on_error=False)
  File “/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py”, line 943, in _request
    raise ConnectionError
requests.exceptions.ConnectionError
Provisioning: provision_vrouter.py --oper add --host_name nodec60 --host_ip 10.204.217.100: 1/10
Traceback (most recent call last):
  File “/opt/contrail/utils/provision_vrouter.py”, line 274, in <module>
    main()
  File “/opt/contrail/utils/provision_vrouter.py”, line 270, in main
    VrouterProvisioner(args_str)
  File “/opt/contrail/utils/provision_vrouter.py”, line 35, in __init__
    api_server_use_ssl=self._args.api_server_use_ssl)
  File “/opt/contrail/utils/vnc_admin_api.py”, line 35, in __init__
    auth_host=self.auth_host)
  File “/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py”, line 460, in __init__
    retry_on_error=False)
  File “/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py”, line 943, in _request
    raise ConnectionError
requests.exceptions.ConnectionError

Tags: