pods not created fine on R4.1

Bug #1729247 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Committed
Critical
Dinesh Bakiaraj
R4.1
Fix Committed
Critical
Dinesh Bakiaraj
Trunk
Fix Committed
Critical
Dinesh Bakiaraj

Bug Description

R4.1 Build 32

Config Nodes : [u'testbed-1-vm1']
Control Nodes : [u'testbed-1-vm1']
Compute Nodes : [u'testbed-1-vm2', u'testbed-1-vm3']
WebUI Node : [u'testbed-1-vm1']
Analytics Nodes : [u'testbed-1-vm1']
Database Nodes : [u'testbed-1-vm1']
Physical Devices : [u"'mx1'"]

All k8s sanity cases were failing. It was seen that pods are not getting their IP
kube-manager log have this exception :

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/vnc_kubernetes.py", line 539, in vnc_process
    self.ingress_mgr.process(event)
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/vnc_ingress.py", line 829, in process
    self._update_ingress(name, uid, event)
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/vnc_ingress.py", line 772, in _update_ingress
    self._create_lb(uid, name, ns_name, event)
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/vnc_ingress.py", line 652, in _create_lb
    lb_obj = self._vnc_create_lb(uid, name, ns_name, annotations)
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/vnc_ingress.py", line 320, in _vnc_create_lb
    name, proj_obj, vn_obj, vip_address, pod_ipam_subnet_uuid)
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/loadbalancer.py", line 187, in create
    vn_obj, service_ns, service_name, vip_address, subnet_uuid)
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/loadbalancer.py", line 109, in _create_virtual_interface
    self._vnc_lib.instance_ip_create(iip_obj)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 42, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 483, in _object_create
    rest.OP_POST, obj_cls.create_uri, data=json_body)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 872, in _request_server
    retry_after_authn=retry_after_authn, retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 973, in _request
    raise BadRequest(status, content)
BadRequest: Virtual-Network(['default-domain', 'default', 'cluster-network']) has invalid subnet(a0fa8fc0-293d-4e5d-ab91-a6d80631f363)

root@testbed-1-vm1:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
ctest-nginx-pod-13035722 0/1 ContainerCreating 0 50s
ctest-nginx-pod-28556556 0/1 ContainerCreating 0 50s
ctest-nginx-pod-87611617 0/1 ContainerCreating 0 50s
ctest-nginx-pod-89246889 0/1 ContainerCreating 0 50s
root@testbed-1-vm1:~#

Changed in juniperopenstack:
milestone: none → r4.1.0.0-fcs
Revision history for this message
Sachchidanand Vaidya (vaidyasd) wrote : Re: [Bug 1729247] [NEW] pods not created fine on R4.1
Download full text (10.5 KiB)

(Pdb) pp vn_obj.__dict__
{'_address_allocation_mode': u'user-defined-subnet-only',
 '_display_name': u'cluster-network',
 '_flood_unknown_unicast': False,
 '_id_perms': permissions = owner = cloud-admin, owner_access = 7, group =
cloud-admin-group, group_access = 7, other_access = 7, uuid = uuid_mslong
= 14433757573316626795, uuid_lslong = 11486240802766775397, enable = True,
created = 2017-11-01T08:47:41.567299, last_modified =
2017-11-01T12:48:57.603803, description = None, user_visible = True,
creator = None,
 '_layer2_control_word': False,
 '_mac_aging_time': 300,
 '_mac_learning_enabled': False,
 '_pbb_etree_enable': False,
 '_pbb_evpn_enable': False,
 '_pending_field_list_updates': {},
 '_pending_field_map_updates': {},
 '_pending_field_updates': set([]),
 '_pending_ref_updates': set([]),
 '_perms2': owner = None, owner_access = 7, global_access = 0, share = [],
 '_port_security_enabled': True,
 '_server_conn': <vnc_api.vnc_api.VncApi object at 0x7f4f42a758d0>,
 '_type': 'virtual-network',
 '_uuid': u'c84f020a-23a4-496b-9f67-53274582e465',
 '_virtual_network_network_id': 4,
 '_virtual_network_properties': allow_transit = None, network_id = None,
vxlan_network_identifier = None, forwarding_mode = l3, rpf = None,
mirror_destination = False,
 'fq_name': [u'default-domain', u'default', u'cluster-network'],
 'name': u'cluster-network',

 'network_ipam_refs': [{u'attr': ipam_subnets = [subnet = ip_prefix =
10.96.0.0, ip_prefix_len = 12, default_gateway = 10.111.255.254,
dns_server_address = 10.111.255.253, subnet_uuid =
a4bffca6-d42d-42b8-b27e-8d7ceda48352, enable_dhcp = True, dns_nameservers
= [], allocation_pools = [], addr_from_start = None, dhcp_option_list =
None, host_routes = None, subnet_name = None, alloc_unit = 1, created =
None, last_modified = None], host_routes = None,
                        u'href':
u'http://10.204.217.194:8082/network-ipam/4721a611-d9cb-4180-a198-4e07fdc6b
110',
                        u'to': [u'default-domain',
                                u'default',
                                u'service-ipam¹], <<<<<<<===== SERVICE-IPAM
                        u'uuid': u'4721a611-d9cb-4180-a198-4e07fdc6b110'},

                       {u'attr': ipam_subnets = [subnet = None,
default_gateway = None, dns_server_address = None, subnet_uuid =
2c08a134-9c83-443d-becf-a50051e79aee, enable_dhcp = True, dns_nameservers
= [], allocation_pools = [], addr_from_start = None, dhcp_option_list =
None, host_routes = None, subnet_name = None, alloc_unit = 1, created =
None, last_modified = None], host_routes = None,
                        u'href':
u'http://10.204.217.194:8082/network-ipam/964bc1a7-6953-4684-a064-5e54170e1
5fd',
                        u'to': [u'default-domain', u'default',
u'pod-ipam¹], <<<<======= POD-IPAM
u'uuid': u'964bc1a7-6953-4684-a064-5e54170e15fd'}],

 'network_policy_refs': [{u'attr': sequence = major = 0, minor = 0, timer
= None,
                          u'href':
u'http://10.204.217.194:8082/network-policy/376241a6-7c17-4591-9bca-42b4713
42f26',
                          u'to': [u'default-domain',
                                  u'default',
                                  u'ip-fabric-cluster-network-d...

Changed in juniperopenstack:
assignee: Sachchidanand Vaidya (vaidyasd) → Dinesh Bakiaraj (dineshb)
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/37140
Submitter: Dinesh Bakiaraj (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/37141
Submitter: Dinesh Bakiaraj (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/37140
Committed: http://github.com/Juniper/contrail-controller/commit/2d98189401e6db158c4214fff0bd6fde39a01f9b
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 2d98189401e6db158c4214fff0bd6fde39a01f9b
Author: dineshb-jnpr <email address hidden>
Date: Thu Nov 2 23:49:50 2017 -0700

Exit kube-manager if mandatory args are missing.

This commit introduces a framework to specify mandatory
arguments for kube-manager. The persence and the contents
of the mandatory arguments will be enforced at kube-manager
arg parse phase. If validation fails, kube-manager will exit
with an error.

Change-Id: I6a645ff9d4a9981dc26515498fe1d9eab26fecb6
Partial-bug: #1729247

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/37204
Submitter: Dinesh Bakiaraj (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/37205
Submitter: Dinesh Bakiaraj (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/37141
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/26b5e6507a3c66e3a8370fe0f626059112aff334
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 26b5e6507a3c66e3a8370fe0f626059112aff334
Author: dineshb-jnpr <email address hidden>
Date: Fri Nov 3 00:03:15 2017 -0700

Restart kube-manager service forever, on failure.

The systemd defualt is give up on restart of kube-manager, if the
process fails 5 times within 10 secs. But the desired behavior is
to keep retrying start of kubemanager. This commit sets the systemd
restart settings such that kubemanager failure will not hit the
failure threshold of systemd and hence systemd will keep restarting
kube-manager on failure.

Change-Id: Ic1ee7d781739b5a857d0c02cf6929f1dddcb72af
Partial-bug: #1729247

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/37204
Submitter: Dinesh Bakiaraj (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/37321
Submitter: Dinesh Bakiaraj (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/37322
Submitter: Dinesh Bakiaraj (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/37204
Submitter: Dinesh Bakiaraj (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/37321
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/b9eaf661a4444d6c13b0ab143f448c893e7b0a91
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit b9eaf661a4444d6c13b0ab143f448c893e7b0a91
Author: dineshb-jnpr <email address hidden>
Date: Fri Nov 3 00:03:15 2017 -0700

Restart kube-manager service forever, on failure.

The systemd defualt is give up on restart of kube-manager, if the
process fails 5 times within 10 secs. But the desired behavior is
to keep retrying start of kubemanager. This commit sets the systemd
restart settings such that kubemanager failure will not hit the
failure threshold of systemd and hence systemd will keep restarting
kube-manager on failure.

Change-Id: Ic1ee7d781739b5a857d0c02cf6929f1dddcb72af
Partial-bug: #1729247

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/37322
Committed: http://github.com/Juniper/contrail-controller/commit/c4dabbd1180981f3ee42e03044b74cf4f485328b
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit c4dabbd1180981f3ee42e03044b74cf4f485328b
Author: dineshb-jnpr <email address hidden>
Date: Wed Nov 8 12:31:05 2017 -0800

Exit kube-manager if mandatory args are missing.

This commit introduces a framework to specify mandatory
arguments for kube-manager. The persence and the contents
of the mandatory arguments will be enforced at kube-manager
arg parse phase. If validation fails, kube-manager will exit
with an error.

Change-Id: I91aa7b1534eb41a3441a83fae32989547cedd79b
Partial-bug: #1729247

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/37205
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/24b7cc891196a9ab864e790ec012c8c81069bb4b
Submitter: Zuul (<email address hidden>)
Branch: master

commit 24b7cc891196a9ab864e790ec012c8c81069bb4b
Author: dineshb-jnpr <email address hidden>
Date: Fri Nov 3 00:03:15 2017 -0700

Restart kube-manager service forever, on failure.

The systemd defualt is give up on restart of kube-manager, if the
process fails 5 times within 10 secs. But the desired behavior is
to keep retrying start of kubemanager. This commit sets the systemd
restart settings such that kubemanager failure will not hit the
failure threshold of systemd and hence systemd will keep restarting
kube-manager on failure.

Change-Id: Ic1ee7d781739b5a857d0c02cf6929f1dddcb72af
Partial-bug: #1729247

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/37204
Submitter: Dinesh Bakiaraj (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/37204
Committed: http://github.com/Juniper/contrail-controller/commit/409a6674ad1e260609a790313bbb5eb03b7f18c0
Submitter: Zuul (<email address hidden>)
Branch: master

commit 409a6674ad1e260609a790313bbb5eb03b7f18c0
Author: dineshb-jnpr <email address hidden>
Date: Thu Nov 9 11:02:15 2017 -0800

Exit kube-manager if mandatory args are missing.

This commit introduces a framework to specify mandatory
arguments for kube-manager. The persence and the contents
of the mandatory arguments will be enforced at kube-manager
arg parse phase. If validation fails, kube-manager will exit
with an error.

Change-Id: I6a645ff9d4a9981dc26515498fe1d9eab26fecb6
Partial-bug: #1729247

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.