Export control node KPIs using statistics infra

[Raj Reddy]

Need to export control node KPIs (Key Performance Indicators) using stats
infra. Doing so allows aggregation, slicing and dicing of dials/counters.
UI can do queries and provide visualization in the form of histograms etc.

In some cases counters are already sent as part of UVEs - they simply need
additional annotation to use the stats infra. In other cases, new UVEs may
need to be defined.

The initial set of KPIs are:

1. Number of total and primary paths received from each bgp/xmpp peer. The
refcount_ field in BgpPeer can be renamed to total_path_count_. The total
and secondary path counts should be added to UVEs for bgp and xmpp peers.

2. Number of reach/unreach route updates sent/received to/from each xmpp
and bgp peer. This should also be sent as part of bgp and xmpp peer UVEs.

3. Total number of routes and paths in each BgpTable. This will require
defining a new UVE for a table.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20141
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20141
Committed: http://github.org/Juniper/contrail-controller/commit/a82a822b730516b89ab6f0ac8c2b3d2a5fa87bdb
Submitter: Zuul
Branch: master

commit a82a822b730516b89ab6f0ac8c2b3d2a5fa87bdb
Author: Ananth Suryanarayana <email address hidden>
Date: Wed May 11 19:06:08 2016 -0700

Update BGP and XMPP UVEs

o Add UVEs for BgpTables (total, primary, secondary and infeasible path counts)
o Add total_path_count and primary_path_count to bgp and xmpp peer UVEs
o Send UVEs periodically off a new bgp::Uve task. This runs mutually
exclusively with (only) bgp::Config task
o Convert all counters in IPeerDebygStats to tbb::atomic, as they can now be
changed and read at the same from different threads
o Cleanup stats collection in control-node/main.c by moving code into
respective modules BgpServer, BgpXmppChannelManager and IFMapServer

TODO: Integrate UVEs (selectively) with KPIs

Change-Id: Ia53e766956bb2590d417a1f4ce71e5df570cf040
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20843
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20968
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20994
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22049
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22050
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22049
Committed: http://github.org/Juniper/contrail-controller/commit/7ac06d88c96b7d7db2aff3b4b72f1840008e8f5d
Submitter: Zuul
Branch: master

commit 7ac06d88c96b7d7db2aff3b4b72f1840008e8f5d
Author: Ananth Suryanarayana <email address hidden>
Date: Fri Jul 15 10:47:59 2016 -0700

Change table-stats from list to map so that infra can analyze the stats easily

map makes it easer to diff, merge etc. when compared with a vector

Change-Id: Iadf4bb1c7ebff96e57ebcd94bb49b92916eddf3d
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22050
Committed: http://github.org/Juniper/contrail-controller/commit/0dbf7927f84e61eddb493de7ed34cc6cbf9b7c12
Submitter: Zuul
Branch: R3.1

commit 0dbf7927f84e61eddb493de7ed34cc6cbf9b7c12
Author: Ananth Suryanarayana <email address hidden>
Date: Fri Jul 15 10:47:59 2016 -0700

Change table-stats from list to map so that infra can analyze the stats easily

map makes it easer to diff, merge etc. when compared with a vector

Change-Id: Iadf4bb1c7ebff96e57ebcd94bb49b92916eddf3d
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22327
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22328
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22327
Committed: http://github.org/Juniper/contrail-controller/commit/003c45d7e9b8833e29d95327c9383ff11cb482a7
Submitter: Zuul
Branch: master

commit 003c45d7e9b8833e29d95327c9383ff11cb482a7
Author: Ananth Suryanarayana <email address hidden>
Date: Thu Jul 21 17:26:39 2016 -0700

Remove redundant address-family field from table structure

This is already present as the key of the map that encloses these stats

Change-Id: I28f4d8fbd79dd8b1fe55872a34fd867a32b24fec
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22511
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22512
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22328
Committed: http://github.org/Juniper/contrail-controller/commit/8b4877504374edbe5bba43d14ceb7cb4b2e25f05
Submitter: Zuul
Branch: R3.1

commit 8b4877504374edbe5bba43d14ceb7cb4b2e25f05
Author: Ananth Suryanarayana <email address hidden>
Date: Thu Jul 21 17:26:39 2016 -0700

Remove redundant address-family field from table structure

This is already present as the key of the map that encloses these stats

Change-Id: I28f4d8fbd79dd8b1fe55872a34fd867a32b24fec
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22512
Committed: http://github.org/Juniper/contrail-controller/commit/6ebe8385fdfd6917514f7759a8510437a69b4e07
Submitter: Zuul
Branch: R3.1

commit 6ebe8385fdfd6917514f7759a8510437a69b4e07
Author: Ananth Suryanarayana <email address hidden>
Date: Tue Jul 26 15:38:44 2016 -0700

Generate derevied stats for BGP and XMPP route updates and flap counts

o BGP and XMPP route updates are sent periodically and will last the lifetime
of the peer
o Flap counts are also sent as separate UVE.
o Xmpp Peer flap counts which are maintained across session flips are sent
even after xmpp peer is deleted (By walking XmppConnectionEndPoint map)

Change-Id: I90825f4367d3295042e35d13bd64291a5cd388fd
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22511
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22586
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22586
Committed: http://github.org/Juniper/contrail-controller/commit/25b0e18647132972c3a8d94b5060236c1e625304
Submitter: Zuul
Branch: R3.1

commit 25b0e18647132972c3a8d94b5060236c1e625304
Author: Ananth Suryanarayana <email address hidden>
Date: Thu Jul 28 14:03:09 2016 -0700

Fix dns build issues by re-ordering linked libraries as appropriate

Change-Id: Ide67443cc0d4085ade6fe3137a4e917dbd82f566
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22587
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22511
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22593
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22596
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22511
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22601
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22601
Committed: http://github.org/Juniper/contrail-controller/commit/fea512c9505e6f4cca528453d256148f89498f35
Submitter: Zuul
Branch: R3.1

commit fea512c9505e6f4cca528453d256148f89498f35
Author: Ananth Suryanarayana <email address hidden>
Date: Thu Jul 28 20:48:07 2016 -0700

Fix query-engine compile issues of derived stats

Change-Id: I0d32547e28f1989e47fa14948ba1aa85b449469e
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22602
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22602
Committed: http://github.org/Juniper/contrail-controller/commit/eedbb1116d9c7cbbfe1add9ecc8635c2ef73d0e9
Submitter: Zuul
Branch: master

commit eedbb1116d9c7cbbfe1add9ecc8635c2ef73d0e9
Author: Ananth Suryanarayana <email address hidden>
Date: Thu Jul 28 20:48:07 2016 -0700

Fix query-engine compile issues of derived stats

Change-Id: I0d32547e28f1989e47fa14948ba1aa85b449469e
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22587
Committed: http://github.org/Juniper/contrail-controller/commit/553ba0ed593547f7ecf93198a8c4b6f76ed86f69
Submitter: Zuul
Branch: master

commit 553ba0ed593547f7ecf93198a8c4b6f76ed86f69
Author: Ananth Suryanarayana <email address hidden>
Date: Thu Jul 28 14:03:09 2016 -0700

Fix dns build issues by re-ordering linked libraries as appropriate

Change-Id: Ide67443cc0d4085ade6fe3137a4e917dbd82f566
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22596
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22753
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22511
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22511
Committed: http://github.org/Juniper/contrail-controller/commit/66ebe20c31807b0f52efd3559a2189e18bf750cd
Submitter: Zuul
Branch: master

commit 66ebe20c31807b0f52efd3559a2189e18bf750cd
Author: Ananth Suryanarayana <email address hidden>
Date: Tue Jul 26 15:38:44 2016 -0700

Generate derevied stats for BGP and XMPP route updates and flap counts

o BGP and XMPP route updates are sent periodically and will last the lifetime
of the peer
o Flap counts are also sent as separate UVE.
o Xmpp Peer flap counts which are maintained across session flips are sent
even after xmpp peer is deleted (By walking XmppConnectionEndPoint map)

Change-Id: I90825f4367d3295042e35d13bd64291a5cd388fd
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22593
Committed: http://github.org/Juniper/contrail-controller/commit/c5cda239dd5ded34f408703a45b0aa6dfd1d2d0c
Submitter: Zuul
Branch: R3.1

commit c5cda239dd5ded34f408703a45b0aa6dfd1d2d0c
Author: Ananth Suryanarayana <email address hidden>
Date: Thu Jul 28 14:52:14 2016 -0700

Delete BGP Peer Flap count UVE when the peer is destroyed

Change-Id: I6e4e5646fce4e48552d63b926ae65a2b7bf7421f
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22753
Committed: http://github.org/Juniper/contrail-controller/commit/a5933f9f4bc6764857e882e775eebdb1a08cdd8d
Submitter: Zuul
Branch: R3.1

commit a5933f9f4bc6764857e882e775eebdb1a08cdd8d
Author: Ananth Suryanarayana <email address hidden>
Date: Mon Aug 1 22:28:48 2016 -0700

Send BGP Peer Flap stats periodically

This is required in order to generate derived stats correctly

Change-Id: I74fa4c6b6ed7b5ed456d91541bb7677d7f6e064e
Partial-Bug: 1576437

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22596
Committed: http://github.org/Juniper/contrail-controller/commit/7ed7bb7d52827d6173ac24753bc0ed3e18d00c81
Submitter: Zuul
Branch: R3.1

commit 7ed7bb7d52827d6173ac24753bc0ed3e18d00c81
Author: Ananth Suryanarayana <email address hidden>
Date: Thu Jul 28 16:42:10 2016 -0700

Remove periodic flap stats for XMPP Agent peers

At the moment, we do not have any configuration associated with XMPP Agent peer.
Hence lifetime of flap-stats cannot be cleanly managed

Change-Id: I4fd9a8cbc8504bf9aed9bde073bc22af43637240
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24313
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24648
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24649
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24649
Committed: http://github.org/Juniper/contrail-controller/commit/6b089c58cf98a317a50ec7847bce1c74367c3d7e
Submitter: Zuul
Branch: master

commit 6b089c58cf98a317a50ec7847bce1c74367c3d7e
Author: Ananth Suryanarayana <email address hidden>
Date: Tue Oct 4 10:49:47 2016 -0700

Reset stats structure and compute total (reach, unreach, eor) rx/tx correctly

Change-Id: Id0cad78e033907e176c4b7d3cd7b42db1b657430
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24313
Committed: http://github.org/Juniper/contrail-controller/commit/f3132e14942108d3b4abf29b0c3102805ce038ca
Submitter: Zuul
Branch: R3.1

commit f3132e14942108d3b4abf29b0c3102805ce038ca
Author: Ananth Suryanarayana <email address hidden>
Date: Tue Sep 20 13:31:17 2016 -0700

Remove instance stats uve upon instance deletion

Also send infeasible paths count as part of table stats uve

Change-Id: I8ef25b52ec643eb1f2111893846004c8047836f7
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26003
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26004
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26003
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26004
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26003
Committed: http://github.org/Juniper/contrail-controller/commit/3685d4a2d2718b5674d78d88794794f3e8214d8c
Submitter: Zuul
Branch: master

commit 3685d4a2d2718b5674d78d88794794f3e8214d8c
Author: Ananth Suryanarayana <email address hidden>
Date: Fri Nov 11 10:50:13 2016 -0800

Tag fields in BgpRouterState to derive statistics based on their changes

Change-Id: I2f70ee954af8f9518979c7837434e372b8027b0b
Partial-Bug: #1576437

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26004
Committed: http://github.org/Juniper/contrail-controller/commit/fca809b5c136699c1b0213d43cab7bea87cbfbbc
Submitter: Zuul
Branch: R3.2

commit fca809b5c136699c1b0213d43cab7bea87cbfbbc
Author: Ananth Suryanarayana <email address hidden>
Date: Fri Nov 11 10:50:13 2016 -0800

Tag fields in BgpRouterState to derive statistics based on their changes

Change-Id: I2f70ee954af8f9518979c7837434e372b8027b0b
Partial-Bug: #1576437