Kernel is crashed on compute node while bringing up vlan interface

Bug #1794536 reported by musharani
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
Trunk
Fix Committed
Critical
haji mohamed ashraf ali

Bug Description

Brought up the setup with the build ocata-master-294 on multi node setup.

While bringing up the vlan interface kernel is crashed. Because of that compute nodes got rebooted.

Steps to repro:
1. Create 2 VNs vn1 and vn2
2. Create a port called port1 for vn1 and launch vm for that port
3. Create a sub interface called sub-intf with vlan id 101 for port1 under vn2 network
4. Create a sub interface with the vlan 101 by the command sudo vconfig add eth0 101 inside vm.
5. Bring up the sub interface on vm using the command sudo ifconfig eth0.101 up.

While performing step5 kernel is crashed.

Logs are saved in /cs-shared/bugs/1794536

Logs:
=====
[ 1.598363] ACPI: Enabled 4 GPEs in block 00 to 3F
[ 1.598552] ACPI Error: [\_SB_.PRAD] Namespace lookup failure, AE_NOT_FOUND (20130517/psargs-359)
[ 1.598559] ACPI Error: Method parse/execution failed [\_GPE._L24] (Node ffff9b521b8dff78), AE_NOT_FOUND (20130517/psparse-536)
[ 1.598567] ACPI Exception: AE_NOT_FOUND, while evaluating GPE method [_L24] (20130517/evgpe-635)
[ 1.617237] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e])

[ 9016.571852] IPv6: ADDRCONF(NETDEV_CHANGE): tapb8047eff-0d: link becomes ready
[ 9061.713944] ------------[ cut here ]------------
[ 9061.713991] kernel BUG at net/core/dev.c:4143!
[ 9061.714017] invalid opcode: 0000 [#1] SMP
[ 9061.714053] Modules linked in: vhost_net vhost macvtap macvlan ebtable_filter ebtables ip6table_filter ip6_tables devlink tun ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc overlay(T) vrouter(OE) sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif pcspkr joydev i2c_i801 mei_me sg mei lpc_ich ioatdma shpchp ipmi_si wmi ipmi_devintf ipmi_msghandler ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
[ 9061.714779] ttm ahci libahci ixgbe drm libata igb crct10dif_pclmul crct10dif_common crc32c_intel mdio i2c_algo_bit ptp i2c_core pps_core dca dm_mirror dm_region_hash dm_log dm_mod
[ 9061.714919] CPU: 2 PID: 7129 Comm: vhost-7126 Kdump: loaded Tainted: G OE ------------ T 3.10.0-862.11.6.el7.x86_64 #1
[ 9061.715016] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
[ 9061.715102] task: ffff9b504a7a0000 ti: ffff9b515d82c000 task.ti: ffff9b515d82c000
[ 9061.715166] RIP: 0010:[<ffffffff991f0c40>] [<ffffffff991f0c40>] __netif_receive_skb_core+0x7f0/0xa20
[ 9061.715266] RSP: 0018:ffff9b515d82fba8 EFLAGS: 00010286
[ 9061.715312] RAX: 00000000ffffffff RBX: ffff9b5138253900 RCX: 0000000000000000
[ 9061.715372] RDX: 00000000ffffffff RSI: 0000000000000000 RDI: ffff9b5138253900
[ 9061.715432] RBP: ffff9b515d82fc10 R08: 000000000001bb00 R09: ffffffff991d94e7
[ 9061.715492] R10: ffff9b515fa9bb00 R11: ffffea34dfe09480 R12: ffff9b515d82fbb8
[ 9061.715552] R13: 0000000000000000 R14: ffff9b681f3cc000 R15: 0000000000000001
[ 9061.715613] FS: 0000000000000000(0000) GS:ffff9b515fa80000(0000) knlGS:0000000000000000
[ 9061.715680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9061.715735] CR2: 0000556519352bf0 CR3: 0000002ff927c000 CR4: 00000000001627e0
[ 9061.715799] Call Trace:
[ 9061.715830] [<ffffffff991f0e88>] __netif_receive_skb+0x18/0x60
[ 9061.715885] [<ffffffff991f0f10>] netif_receive_skb_internal+0x40/0xc0
[ 9061.715942] [<ffffffff991f0fac>] netif_receive_skb+0x1c/0x70
[ 9061.715998] [<ffffffffc071ad55>] tun_get_user+0x815/0xbc0 [tun]
[ 9061.716053] [<ffffffffc071b166>] tun_sendmsg+0x66/0x90 [tun]
[ 9061.716106] [<ffffffffc0915344>] handle_tx+0x214/0x650 [vhost_net]
[ 9061.716165] [<ffffffffc09157b5>] handle_tx_kick+0x15/0x20 [vhost_net]
[ 9061.716225] [<ffffffffc0900321>] vhost_worker+0xb1/0x110 [vhost]
[ 9061.716285] [<ffffffffc0900270>] ? vhost_flush_work+0x20/0x20 [vhost]
[ 9061.716347] [<ffffffff98cbdf21>] kthread+0xd1/0xe0
[ 9061.716394] [<ffffffff98cbde50>] ? insert_kthread_work+0x40/0x40
[ 9061.716455] [<ffffffff993255f7>] ret_from_fork_nospec_begin+0x21/0x21
[ 9061.716512] [<ffffffff98cbde50>] ? insert_kthread_work+0x40/0x40
[ 9061.716563] Code: 41 bf f4 ff ff ff 0f 1f 44 00 00 4c 8d 64 24 10 e9 18 fa ff ff 66 0f 1f 44 00 00 66 41 89 84 24 c0 00 00 00 e9 81 f8 ff ff 66 90 <0f> 0b 66 0f 1f 44 00 00 45 31 ff e9 88 fc ff ff 45 31 ed 48 85
[ 9061.716885] RIP [<ffffffff991f0c40>] __netif_receive_skb_core+0x7f0/0xa20
[ 9061.716948] RSP <ffff9b515d82fba8>

Instances.yaml
==============
instances:
  nodec57:
      ip: 10.204.216.153
      provider: bms
      roles:
          analytics: null
          analytics_database: null
          config: null
          config_database: null
          control: null
          openstack: null
          webui: null
  nodec7:
      ip: 10.204.216.64
      provider: bms
      roles:
          analytics: null
          analytics_database: null
          config: null
          config_database: null
          control: null
          openstack: null
          webui: null
  nodec8:
      ip: 10.204.216.65
      provider: bms
      roles:
          analytics: null
          analytics_database: null
          config: null
          config_database: null
          control: null
          openstack: null
          webui: null
  nodei1:
      ip: 10.204.216.150
      provider: bms
      roles:
          openstack_compute: null
          vrouter:
              PHYSICAL_INTERFACE: eno2
  nodei2:
      ip: 10.204.217.114
      provider: bms
      roles:
          openstack_compute: null
          vrouter:
              PHYSICAL_INTERFACE: eno2
  nodei3:
      ip: 10.204.217.115
      provider: bms
      roles:
          openstack_compute: null
          vrouter:
              PHYSICAL_INTERFACE: eno2

contrail_configuration:
  AAA_MODE: rbac
  CLOUD_ORCHESTRATOR: openstack
  CONFIG_API_VIP: 10.204.216.64
  CONTAINER_REGISTRY: 10.204.217.152:5000
  CONTRAIL_VERSION: ocata-master-294
  CONTROLLER_NODES: 10.204.216.64,10.204.216.65,10.204.216.153
  CONTROL_NODES: 192.168.192.6,192.168.192.5,192.168.192.7
  ENCAP_PRIORITY: VXLAN,MPLSoUDP,MPLSoGRE
  IPFABRIC_SERVICE_HOST: 10.204.216.130
  KEYSTONE_AUTH_ADMIN_PASSWORD: c0ntrail123
  KEYSTONE_AUTH_HOST: 10.204.216.130
  KEYSTONE_AUTH_URL_VERSION: /v3
  OPENSTACK_NODES: 10.204.216.64,10.204.216.65,10.204.216.153
  RABBITMQ_NODE_PORT: 5673
  REGISTRY_PRIVATE_INSECURE: true
  VROUTER_GATEWAY: 192.168.192.253
  XMPP_SSL_ENABLE: true

  RABBITMQ_NODE_PORT: "5673"
orchestrator_configuration:
  contrail_api_interface_address: 10.204.216.64
  enable_haproxy: 'yes'
  enable_ironic: 'no'
  enable_swift: 'no'
  external_vip: 10.204.216.130
  internal_vip: 10.204.216.130
  keystone:
      password: c0ntrail123
      version: v3
  vrrp_id: 151

musharani (musharani)
description: updated
tags: added: sanityblocker
removed: sanity
tags: added: vrouter
musharani (musharani)
description: updated
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/48024
Submitter: haji mohamed ashraf ali (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/48024
Committed: http://github.com/Juniper/contrail-vrouter/commit/d52b1ae80bbc998a9a01aa6e493ec9b4ce7c93a5
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit d52b1ae80bbc998a9a01aa6e493ec9b4ce7c93a5
Author: hajim <email address hidden>
Date: Wed Dec 5 11:19:41 2018 +0530

Revert "Change functions in Rx code path for batch processing"

This reverts commit 7085f6b125a0966da0180f7388dccc159aa9d11b.
Closes-Bug: #1794536

Change-Id: I955b386213cd2959fa60284eed8b31025427cc64

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.