Juniper Openstack

contrail-collector going berserk over "nw_ace_uuid: () INVALID" messages

Series trunk
Bug #1400617

Bug #1400617 reported by Martin Gerhard Loschwitz on 2014-12-09

This bug affects 5 people

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R2.20	Fix Committed	Medium	Sachin Bansal	Juniper Openstack r2.23 "r2.23"
R2.21.x	Fix Committed	Medium	Raj Reddy	Juniper Openstack r2.21.2
R2.22.x	Fix Committed	Medium	Sachin Bansal	Juniper Openstack r2.22.2
Trunk	Fix Committed	Medium	Raj Reddy

Bug Description

On our systems, we see contrail-collector going berserk from time to time shouting messaes like this into its logfile:

2014-12-09 Tue 09:10:43:707.538 CET bka-001-02 [Thread 140431438345984, Pid 15786]: FlowRecordTable: nw_ace_uuid: () INVALID
2014-12-09 Tue 09:10:43:707.565 CET bka-001-02 [Thread 140430779852544, Pid 15786]: FlowRecordTable: nw_ace_uuid: () INVALID
2014-12-09 Tue 09:10:43:707.611 CET bka-001-02 [Thread 140431438345984, Pid 15786]: FlowRecordTable: nw_ace_uuid: () INVALID
2014-12-09 Tue 09:10:43:707.619 CET bka-001-02 [Thread 140430779852544, Pid 15786]: FlowRecordTable: nw_ace_uuid: () INVALID
2014-12-09 Tue 09:10:43:707.679 CET bka-001-02 [Thread 140430779852544, Pid 15786]: FlowRecordTable: nw_ace_uuid: () INVALID
2014-12-09 Tue 09:10:43:707.691 CET bka-001-02 [Thread 140431438345984, Pid 15786]: FlowRecordTable: nw_ace_uuid: () INVALID
2014-12-09 Tue 09:10:43:707.727 CET bka-001-02 [Thread 140430779852544, Pid 15786]: FlowRecordTable: nw_ace_uuid: () INVALID
2014-12-09 Tue 09:10:43:707.773 CET bka-001-02 [Thread 140430779852544, Pid 15786]: FlowRecordTable: nw_ace_uuid: () INVALID

It starts eating 1500% of CPU time. The event is triggered by no obvious reason and comes to an end for no obvious reason, too.

Tags:

Martin Gerhard Loschwitz (martin-loschwitz) on 2014-12-09

information type:

Proprietary → Public

Nagabhushana R (bhushana) on 2014-12-09

tags:

added: analytics

Raj Reddy (rajreddy) on 2014-12-09

Changed in juniperopenstack:
assignee:	nobody → Sundaresan Rajangam (srajanga)

Raj Reddy (rajreddy) on 2014-12-23

Changed in juniperopenstack:
importance:	Undecided → Medium

Revision history for this message

Sundaresan Rajangam (srajanga) wrote on 2015-01-02:

This error message is logged by the contrail-collector when the vrouter-agent sends invalid uuid string. In this specific case, vrouter-agent seem to have sent empty string for nw_ace_uuid in the flow log. Assigning the bug to Ashok to check if the vrouter-agent sends "" nw_ace_uuid in any case.

As far as the high cpu utilization is concerned, there is not enough information to analyze further. Perhaps, it could be due to high rate of messages sent by the agent/other generators. Any information regarding the number of generators connected to the contrail-collector and the rate at which the contrail-collector received messages during period of high cpu utilization would be helpful.

Changed in juniperopenstack:
assignee:	Sundaresan Rajangam (srajanga) → Ashok Singh (ashoksr)
tags:	added: vrouter

Revision history for this message

Sundaresan Rajangam (srajanga) wrote on 2015-01-02:

Similar issue was reported sometime back.
https://bugs.launchpad.net/juniperopenstack/+bug/1344716
As part of the fix, the boost::uuid::sting_generator (throws exception when invalid uuid string is received) was replaced with stream operator.

Revision history for this message

Martin Gerhard Loschwitz (martin-loschwitz) wrote on 2015-02-02:

We still see this and contrail-collector still eats massive amounts of CPU and writes this stuff into its logfile all the time. Now it's happening with Juno and Contrail 2.0. Any help will be appreciated.

Revision history for this message

Ashok Singh (ashoksr) wrote on 2015-02-03:

rule.tar.gz Edit (7.4 KiB, application/x-tar)

The "" uuid is being sent from agent because "" uuid is being received by agent as part of acl rule configuration. On checking configuration in API server it was found that null UUIDs were present for some ACL rules. I have attached a tar file which has ACL entries and rules for one of the ACL (default-domain:admin:float24:float24) which had null UUIDs

Ashok Singh (ashoksr) on 2015-02-03

tags:	added: config
Changed in juniperopenstack:
assignee:	Ashok Singh (ashoksr) → Sachin Bansal (sbansal)

Revision history for this message

Sachin Bansal (sbansal) wrote on 2015-02-03:

Martin, could you please tell us what version of contrail you are running? And if you had upgraded from an earlier version, please let us know that too.

Revision history for this message

Martin Gerhard Loschwitz (martin-loschwitz) wrote on 2015-02-04:

This is Contrail R2.0 with a few patches applied on top of it (mostly taken from your git repos). We updated this installation from 1.20 to 2.0 a week ago, but as we have pointed out earlier, we saw the same messages from contrail-collector with 1.20 as well.

Hari Prasad Killi (haripk) on 2015-06-22

tags:

removed: vrouter

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-09: [Review update] master

Review in progress for https://review.opencontrail.org/17051
Submitter: Raj Reddy (<email address hidden>)

Raj Reddy (rajreddy) on 2016-02-09

Changed in juniperopenstack:
assignee:	Sachin Bansal (sbansal) → Raj Reddy (rajreddy)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-02-13: A change has been merged

Reviewed: https://review.opencontrail.org/17051
Committed: http://github.org/Juniper/contrail-controller/commit/f32f0c9921f1d3809181e5bc092be8fbaa05ac80
Submitter: Zuul
Branch: master

commit f32f0c9921f1d3809181e5bc092be8fbaa05ac80
Author: Raj Reddy <email address hidden>
Date: Tue Feb 9 13:39:05 2016 -0800

There are some corner cases where the ace uuid comes as null, we
can just ignore in the collector than printing and error message
as that causes log file to be overwhelmed.
Closes-Bug: #1400617

Change-Id: I4943f0d0621330ccb105182d9e6f8f29c177561a

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-11: [Review update] R2.21.x

#10

Review in progress for https://review.opencontrail.org/19216
Submitter: Jean-Philippe Braun (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-29: [Review update] R2.20

#12

Review in progress for https://review.opencontrail.org/19744
Submitter: Raj Reddy (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-04-29: [Review update] R2.22.x

#14

Review in progress for https://review.opencontrail.org/19745
Submitter: Raj Reddy (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-02: A change has been merged

#16

Reviewed: https://review.opencontrail.org/19216
Committed: http://github.org/Juniper/contrail-controller/commit/a1a9ab44a4814b6f4ce7a9bf622d2459ca4fda28
Submitter: Zuul
Branch: R2.21.x

commit a1a9ab44a4814b6f4ce7a9bf622d2459ca4fda28
Author: Raj Reddy <email address hidden>
Date: Tue Feb 9 13:39:05 2016 -0800

There are some corner cases where the ace uuid comes as null, we
can just ignore in the collector than printing and error message
as that causes log file to be overwhelmed.
Closes-Bug: #1400617

Change-Id: I4943f0d0621330ccb105182d9e6f8f29c177561a
(cherry picked from commit f32f0c9921f1d3809181e5bc092be8fbaa05ac80)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-03:

#17

Reviewed: https://review.opencontrail.org/19745
Committed: http://github.org/Juniper/contrail-controller/commit/9bc4ff65e9bcc3d906256c7ac7b3b0e46275d502
Submitter: Zuul
Branch: R2.22.x

commit 9bc4ff65e9bcc3d906256c7ac7b3b0e46275d502
Author: Raj Reddy <email address hidden>
Date: Tue Feb 9 13:39:05 2016 -0800

There are some corner cases where the ace uuid comes as null, we
can just ignore in the collector than printing and error message
as that causes log file to be overwhelmed.
Closes-Bug: #1400617

Change-Id: I4943f0d0621330ccb105182d9e6f8f29c177561a
(cherry picked from commit f32f0c9921f1d3809181e5bc092be8fbaa05ac80)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2016-05-03:

#18

Reviewed: https://review.opencontrail.org/19744
Committed: http://github.org/Juniper/contrail-controller/commit/31b42b19ffae918d0b633aff5d44e5a9f078a560
Submitter: Zuul
Branch: R2.20

commit 31b42b19ffae918d0b633aff5d44e5a9f078a560
Author: Raj Reddy <email address hidden>
Date: Tue Feb 9 13:39:05 2016 -0800

There are some corner cases where the ace uuid comes as null, we
can just ignore in the collector than printing and error message
as that causes log file to be overwhelmed.
Closes-Bug: #1400617

Change-Id: I4943f0d0621330ccb105182d9e6f8f29c177561a
(cherry picked from commit f32f0c9921f1d3809181e5bc092be8fbaa05ac80)