networking-odl

Deadlock occurs when deleting bgp peer

Bug #1836841 reported by jingting on 2019-07-17

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	networking-odl	New	Undecided	Unassigned

Bug Description

We have BGP peer established. ODL-BGP is receiving routes from router ,meanwhile we delete peer and find that ODL-BGP didn't close peer correctly.

DELETING PEER URL（/restconf/config/openconfig-network-instance:network-instances/network-instance/global-bgp/openconfig-network-instance:protocols/protocol/openconfig-policy-types:BGP/bgp-example/bgp/neighbors）

Below is the log of jstack:

Found one Java-level deadlock:
=============================
"epollEventLoopGroup-7-2":
  waiting to lock monitor 0x00007fe09411bca8 (object 0x00007fd1906f0390, a org.opendaylight.protocol.bgp.rib.impl.BGPPeer),
  which is held by "opendaylight-cluster-data-notification-dispatcher-46"
"opendaylight-cluster-data-notification-dispatcher-46":
  waiting to lock monitor 0x00007fe09411d148 (object 0x00007fd1906f0ab0, a org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl),
  which is held by "epollEventLoopGroup-7-2"

Java stack information for the threads listed above:
===================================================
"epollEventLoopGroup-7-2":
        at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onUpdateMessage(BGPPeer.java:302)
        - waiting to lock <0x00007fd1906f0390> (a org.opendaylight.protocol.bgp.rib.impl.BGPPeer)
        at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:255)
        at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:298)
        - locked <0x00007fd1906f0ab0> (a org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl)
        at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.channelRead0(BGPSessionImpl.java:523)
        at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.channelRead0(BGPSessionImpl.java:83)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:382)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:335)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
"opendaylight-cluster-data-notification-dispatcher-46":
        at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.close(BGPSessionImpl.java:255)
        - waiting to lock <0x00007fd1906f0ab0> (a org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl)
        at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.releaseConnection(BGPPeer.java:566)
        - locked <0x00007fd1906f0390> (a org.opendaylight.protocol.bgp.rib.impl.BGPPeer)
        at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.close(BGPPeer.java:246)
        - locked <0x00007fd1906f0390> (a org.opendaylight.protocol.bgp.rib.impl.BGPPeer)
        at org.opendaylight.protocol.bgp.rib.impl.config.BgpPeer$BgpPeerSingletonService.closeServiceInstance(BgpPeer.java:360)
        - locked <0x00007fd1906f0290> (a org.opendaylight.protocol.bgp.rib.impl.config.BgpPeer$BgpPeerSingletonService)
        at org.opendaylight.protocol.bgp.rib.impl.config.BgpPeer$BgpPeerSingletonService.access$200(BgpPeer.java:237)
        at org.opendaylight.protocol.bgp.rib.impl.config.BgpPeer.closeServiceInstance(BgpPeer.java:187)
        - locked <0x00007fd1906d77e0> (a org.opendaylight.protocol.bgp.rib.impl.config.BgpPeer)
        at org.opendaylight.protocol.bgp.rib.impl.config.BGPClusterSingletonService.lambda$closeAllBindedPeers$0(BGPClusterSingletonService.java:225)
        at org.opendaylight.protocol.bgp.rib.impl.config.BGPClusterSingletonService$$Lambda$1460/1043302599.accept(Unknown Source)
        at java.util.HashMap.forEach(HashMap.java:1289)
        at org.opendaylight.protocol.bgp.rib.impl.config.BGPClusterSingletonService.closeAllBindedPeers(BGPClusterSingletonService.java:223)
        - locked <0x00007fd18c9f38d0> (a org.opendaylight.protocol.bgp.rib.impl.config.BGPClusterSingletonService)
        at org.opendaylight.protocol.bgp.rib.impl.config.BGPClusterSingletonService.onGlobalChanged(BGPClusterSingletonService.java:161)
        - locked <0x00007fd18c9f38d0> (a org.opendaylight.protocol.bgp.rib.impl.config.BGPClusterSingletonService)
        at org.opendaylight.protocol.bgp.rib.impl.config.BgpDeployerImpl.onGlobalChanged(BgpDeployerImpl.java:250)
        - locked <0x00007fd18bbfe2e8> (a org.opendaylight.protocol.bgp.rib.impl.config.BgpDeployerImpl)
        at org.opendaylight.protocol.bgp.rib.impl.config.BgpDeployerImpl.handleGlobalChange(BgpDeployerImpl.java:184)
        at org.opendaylight.protocol.bgp.rib.impl.config.BgpDeployerImpl.handleDeletions(BgpDeployerImpl.java:173)
        at org.opendaylight.protocol.bgp.rib.impl.config.BgpDeployerImpl.onDataTreeChanged(BgpDeployerImpl.java:143)
        - locked <0x00007fd18bbfe2e8> (a org.opendaylight.protocol.bgp.rib.impl.config.BgpDeployerImpl)
        at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:42)
        at org.opendaylight.controller.sal.core.compat.LegacyDOMDataBrokerAdapter$ProxyListener.onDataTreeChanged(LegacyDOMDataBrokerAdapter.java:353)
        at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:81)
        at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:42)
        at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:40)
        at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
        at akka.actor.Actor.aroundReceive(Actor.scala:517)
        at akka.actor.Actor.aroundReceive$(Actor.scala:515)
        at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
        at akka.actor.ActorCell.invoke(ActorCell.scala:557)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
        at akka.dispatch.Mailbox.run(Mailbox.scala:225)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Found 1 deadlock.

[root@localhost log]#

Revision history for this message

jingting (kangjingting) wrote on 2019-07-17:

Version: Neon release.

summary:	- deadlock when deleting bgp peer + Deadlock when deleting bgp peer
summary:	- Deadlock when deleting bgp peer + Deadlock occurs when deleting bgp peer

Revision history for this message

Lajos Katona (lajos-katona) wrote on 2019-08-29:

Hi, Could you please give some extra details like versions you used, and reproduction steps.
As I see this is more an ODL bug, not a networking-odl, am I right?
If the answer to the last uestion is yes, the ODL bug tracker is here I think: https://bugs.opendaylight.org/

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.