k8s:Compute node reboots when ping initiated from pod to/ from mx when ip_fabric_forwarding is enabled

[Venkatesh Velpula]

orchestrator :k8s
build :ocata-master-117
hostOS :centos7.5

Topology
========
                     192.168.10.0/24
 master -----192.168.10.2 -------192.168.10.5---slave
 pod :192.168.10.20
 fabric network :192.168.10.32/29

on the destination machine
===========================
11:08:08.062972 Out arp who-has 192.168.10.20 tell 192.168.10.100
11:08:08.563473 In IP 192.168.10.20 > 192.168.10.100: ICMP echo request, id 181, seq 248, length 64

00:25:90:c3:af:ab 192.168.10.1 192.168.10.1 ge-1/2/2.10 none
00:25:90:c3:08:6b 192.168.10.2 192.168.10.2 ge-1/2/2.10 none
00:25:90:c3:3f:13 192.168.10.3 192.168.10.3 ge-1/2/2.10 none
00:25:90:c5:10:92 192.168.10.5 192.168.10.5 ge-1/2/2.10 none

on the node
===========
6:39:41.814551 IP 192.168.10.20 > 192.168.10.100: ICMP echo request, id 181, seq 999, length 64
16:39:42.814554 IP 192.168.10.20 > 192.168.10.100: ICMP echo request, id 181, seq 1000, length 64

[root@nodei16 /]# arp
Address HWtype HWaddress Flags Mask Iface
gateway ether 80:ac:ac:f0:a2:c1 C eno1
192.168.10.254 (incomplete) vhost0
nodec20.local ether 00:25:90:c3:08:6a C eno1
192.168.10.2 ether 00:25:90:c3:08:6b C vhost0
nodei40.englab.juniper. ether 00:25:90:e7:7e:7a C eno1
192.168.10.3 ether 00:25:90:c3:3f:13 C vhost0
puppet ether 00:e0:81:ca:5a:75 C eno1
192.168.10.6 ether 00:25:90:e4:08:8e C vhost0
nodei21.englab.juniper. ether 00:25:90:e7:80:9a C eno1
192.168.10.100 ether 4c:96:14:98:23:e1 C vhost0
10.204.217.240 ether 80:71:1f:c0:38:70 C eno1

  252132<=>474640 192.168.10.20:181 1 (0)
                         192.168.10.100:0
(Gen: 1, K(nh):35, Action:F, Flags:, QOS:-1, S(nh):35, Stats:3547/347606,
 SPort 64936, TTL 0, Sinfo 4.0.0.0)

   474640<=>252132 192.168.10.100:181 1 (0)
                         192.168.10.20:0
(Gen: 1, K(nh):35, Action:F, Flags:, QOS:-1, S(nh):15, Stats:0/0, SPort 52383,
 TTL 0, Sinfo 0.0.0.0)

[root@nodei16 /]# dropstats
Invalid IF 0
Trap No IF 0
IF TX Discard 0
IF Drop 0
IF RX Discard 0

Flow Unusable 0
Flow No Memory 0
Flow Table Full 0
Flow NAT no rflow 0
Flow Action Drop 0
Flow Action Invalid 0
Flow Invalid Protocol 0
Flow Queue Limit Exceeded 0
New Flow Drops 0
Flow Unusable (Eviction) 0

Original Packet Trapped 0

Discards 64699<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
TTL Exceeded 0
Mcast Clone Fail 0
Cloned Original 16

2018-05-28 Mon 16:54:54:434.828 IST nodei16 [Thread 139701471885056, Pid 22313]: SANDESH: Send FAILED: 1527506694434304 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = nodei16:Compute:contrail-vrouter-agent:0 client_info= [ status = ClientInit successful_connections = 1 pid = 22313 http_port = 8085 start_time = 1527506694427660 collector_name = collector_ip = 10.204.217.5:8086 collector_list= [ [ (*_iter40) = 10.204.217.5:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_diff= [ [ _iter107->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter107->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter107->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter107->first = VrfObjectLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_diff= [ [ _iter111->first = dropped_sending_disabled _iter111->second = 1, _iter111->first = dropped_wrong_client_sm_state _iter111->second = 6, ] ] ]
2018-05-28 Mon 16:54:54:500.201 IST nodei16 [Thread 139701653199168, Pid 22313]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodei16:192.168.10.2 RECV Mode Client: Event: Tcp Connected peer ip: 192.168.10.2 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1293
2018-05-28 Mon 16:54:56:476.384 IST nodei16 [Thread 139701653199168, Pid 22313]: XMPP [SYS_NOTICE]: XmppEventLog: XMPP Peer nodei16:192.168.10.2 RECV Mode Client: Event: Tcp Connected peer ip: 192.168.10.2 ( <email address hidden> ) controller/src/xmpp/xmpp_state_machine.cc 1293
2018-05-28 Mon 16:59:38:969.607 IST nodei16 [Thread 139701471885056, Pid 22313]: ID-PERM not set for object <default-domain:fabric-test:k8s-fabric-test-pod-network> Table <__ifmap__.virtual_network.0>. Ignoring it
================================================
yaml
===============================================
global_configuration:
  REGISTRY_PRIVATE_INSECURE: True
  CONTAINER_REGISTRY: 10.204.217.152:5000
provider_config:
  bms:
    ssh_pwd: c0ntrail123
    ssh_user: root
    ssh_public_key: /root/.ssh/id_rsa.pub
    ssh_private_key: /root/.ssh/id_rsa
    domainsuffix: local

instances:
  nodec19:
    provider: bms
    ip: 10.204.217.4
    roles:
       config_database:
       config:
       control:
       analytics_database:
       analytics:
       webui:
       k8s_master:
       kubemanager:
  nodec21:
    provider: bms
    ip: 10.204.217.6
    roles:
       k8s_node:
       vrouter:
         PHYSICAL_INTERFACE: enp1s0f1

contrail_configuration:
  CONTAINER_REGISTRY: 10.204.217.152:5000
  CONTRAIL_VERSION: ocata-master-115
  KUBERNETES_IP_FABRIC_SUBNETS: 192.168.10.32/29
  KUBERNETES_PUBLIC_FIP_POOL: {'project': 'default', 'domain': 'default-domain', 'name': '__fip_pool_public__', 'network': '__public__'}
  CLOUD_ORCHESTRATOR: kubernetes
  CONTROLLER_NODES: 10.204.217.4
  CONTROL_NODES: 192.168.10.1
  VROUTER_GATEWAY: 192.168.10.100

[Venkatesh Velpula (vvelpula)]

investigating little more on my setup ...will update

[Pulkit Tandon (pulkitt)]

This issue is reproducible on k8s sanity setup.
Hence reopening it.

Affected sanity script:
TestFabricFWD.test_deployment_with_replica_update_for_fabric_fwd[k8s_sanity]

[Venkatesh Velpula (vvelpula)]

when the family inet is enabled between mx and the controller and initiate aping from pod to mx or mx to pod will result in a crash in the compute node where the pod resides

core file is @

below is my setup and steps to reproduce the issue and right now its in the same state

1) enabled fabric forwarding

2) create a a pod in the fabric forwarding VN (namespace)

3) create a bgp session over family inet between the controller and mx gw

4) initiate a ping either from pod ..or from gw ..will result in the compute node reboot

Note: prior to configuring family inet, I had bgp session over family inet-vpn , then modified inet-vpn to inet on both controller and gateway side .thats when i started hitting this issue

++++++++++++++++++ +++++++++ +++ +++++++++++++++++

Node:192.16.10.5 | | | | |

-----------------| | controller | |blr-mx2(router) |

pod | +++++++192.168.10.1| +++++ | 192.168.10.100 |

192.168.10.68 | | | | |

----------------- | | | |

+++++++++++++++++ +++++++++++++ +----------------+

[root@nodec19 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
nodec19 NotReady master 5d v1.9.2
nodei16 Ready <none> 5d v1.9.2

[root@nodec19 ~]# kubectl get pods --all-namespaces -o wide | grep fab
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
fabric-test fabtest 1/1 Running 0 21s 192.168.10.68 nodei18

nodec19:
  mgmtip:10.204.217.4
  controldataip :192.168.10.1

nodei16:
  mgmtip:10.204.217.
  controldataip :192.168.5

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44252
Submitter: Saurabh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44252
Committed: http://github.com/Juniper/contrail-vrouter/commit/d459a4ae39c050e853aea701f04434d58e7c9ac0
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit d459a4ae39c050e853aea701f04434d58e7c9ac0
Author: Saurabh Gupta <email address hidden>
Date: Fri Jun 29 21:48:42 2018 +0530

GRE tunneled packet on fabric interface crashing vrouter kernel when
the GRE tunnel is not configured on the interface. Reworked how the
relaxed policy check is enforeced int he code to avoid crashing. Crash
happened because packet processing went into a loop in vr_ip_rcv().

Change-Id: I6de49c70a30393c9d752273880b31f415dca1aa5
closes-bug: #1773979

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44333
Submitter: Saurabh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44333
Committed: http://github.com/Juniper/contrail-vrouter/commit/dad83252487be67dc32da75434e7d4210fcf7989
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit dad83252487be67dc32da75434e7d4210fcf7989
Author: Saurabh Gupta <email address hidden>
Date: Fri Jun 29 21:48:42 2018 +0530

GRE tunneled packet on fabric interface crashing vrouter kernel when
the GRE tunnel is not configured on the interface. Reworked how the
relaxed policy check is enforeced int he code to avoid crashing. Crash
happened because packet processing went into a loop in vr_ip_rcv().

Change-Id: I6de49c70a30393c9d752273880b31f415dca1aa5
closes-bug: #1773979
(cherry picked from commit d459a4ae39c050e853aea701f04434d58e7c9ac0)

[Venkatesh Velpula (vvelpula)]

verified

root@nodec19 ~]# kubectl get pods -n fabric-test -o wide
NAME READY STATUS RESTARTS AGE IP NODE
fabtest 1/1 Running 0 21h 192.168.10.68 nodei16
[root@nodec19 ~]# kubectl describe namespace fabric-test
Name: fabric-test
Labels: app=fabtest
Annotations: opencontrail.org/ip_fabric_forwarding=true
              opencontrail.org/isolation=true
Status: Active

No resource quota.

No resource limits.

[root@nodec19 ~]# kubectl exec -it fabtest -n fabric-test ping 192.168.10.100
PING 192.168.10.100 (192.168.10.100) 56(84) bytes of data.
64 bytes from 192.168.10.100: icmp_seq=1 ttl=63 time=1.40 ms
64 bytes from 192.168.10.100: icmp_seq=2 ttl=63 time=0.830 ms
^C

[Sunil Basker (sunilbasker)]

Guys,

When ‘configure_instances’ playbook fails when run on the VM, to install k8. Any idea on how to fix it?

TASK [configure_instances : restart network service] ***************************************************************************************************************************
fatal: [1.1.1.3]: FAILED! => {"changed": false, "msg": "Unable to start service network: Job for network.service failed because the control process exited with error code. See \"systemctl status network.service\" and \"journalctl -xe\" for details.\n"}
fatal: [1.1.1.5]: FAILED! => {"changed": false, "msg": "Unable to start service network: Job for network.service failed because the control process exited with error code. See \"systemctl status network.service\" and \"journalctl -xe\" for details.\n"}
fatal: [1.1.1.4]: FAILED! => {"changed": false, "msg": "Unable to start service network: Job for network.service failed because the control process exited with error code. See \"systemctl status network.service\" and \"journalctl -xe\" for details.\n"}
        to retry, use: --limit @/root/contrail-ansible-deployer/playbooks/configure_instances.retry

on rerun however, the playbook works and the same task in skipped

TASK [configure_instances : restart network service] ***************************************************************************************************************************
skipping: [1.1.1.3]
skipping: [1.1.1.5]
skipping: [1.1.1.4]

Sunil