Reviewed:  https://review.opencontrail.org/49393
Committed: http://github.com/Juniper/contrail-controller/commit/77df3b58265b3fab414dfbc00e1ff39d19f0a99c
Submitter: Zuul v3 CI (<email address hidden>)
Branch:    R6.0-WIP

commit 77df3b58265b3fab414dfbc00e1ff39d19f0a99c
Author: Shivayogi Ugaji <email address hidden>
Date:   Mon Nov 5 22:07:18 2018 -0800

Apply commits from master onto R6.0-WIP

db_resync_done lock is used to indicate the amqp thread to wait for resync to
complete. In this case, when we call SchemaTransformer.destroy_instance()
due to Casandra connection failure, this lock remains locked blocking
destroy_instance. destroy_instance calls _vnc_subscribe_callback to drain the
amqp queue which waits infinitely for db_resync_done lock to be released.
This fix releases db_resync_done lock so that destroy_instance doesnt get
blocked.
Closes-Bug: #1801474

[DM] Hitless image upgrade implementation
Closes-Bug: #1799322

Provisioner for the devicemanager node.
usage:
from /opt/contrail/utils
python provision_devicemgr_node.py --host_name aio --host_ip 10.87.82.2
--oper add --admin_user admin --admin_password contrail123 --admin_tenant_name
admin --openstack_ip 10.87.82.2 --api_server_ip 10.87.82.2
Closes-Bug: #1805303

CFM: Changes for onboarding L3PNF
- Add new platform SRX240
- Add L3PNF subnet is schema
- Add new namespace, VN and IPAM for L3PNF during brownfield onboarding
Closes-Bug: 1800701

Add entrypoint to vrouter-agent service on Windows
Introduce entrypoint for agent similar in design to that from
microservice deployment. For now it will only start agent,
actual features will be added in following changes.
Partial-Bug: #1806677

Check build dependencies for tbb, SimpleAmqpClient and rabbitmq
Closes-Bug: #1806719

Make agent's entrypoint update agent's config on Windows
In future we will generate the whole config from scratch
as on Linux, but for now we only update the vhost's ifname.
It's the only field that can change upon restart.
Partial-Bug: #1806677

bgp-peer selection support for bgpaas
1. Listener BgpRouterConfig is added for BgpRouter and ControlNodeZone
2. BgpRouterConfig builds BgpRouterTree and ControlNodeZoneTree
from IFMapNode
3. BGPaaS gets BgpRouter for configured ControlNodeZone from
BgpRouterConfig and Updates bgp-peer-ip and bgp-peer-port in
the flow.
4. Step 3 is followed for xmpp based peer-selection also.
5. BGPaaS sandesh is updated with primary_control_node_zone,
secondary_control_node_zone, bgp_peer_ip and bgp_peer_port
Partial-bug: #1775872

[DM] Inside-outside workflow - lag/mH
1. Change the exisiting business logic to adhere to the new data model for lag/mH workflow
2. Multi-vlan support
Partial-Bug: #1799329

Rework nodemgr before fixing ntp issue
- move windows/linux code to separate classes instead of same condition through the code
- simplify main.py
- remove copy duplication
Closes-Bug: 1800704

[fabric] Added playbook retry support to job manager
1) When playbook return retry_devices in the output, job manager will retry the playbooks against those devices
2) remove obsolete playbooks from 5.0
3) remove obsolete ansible roles from 5.0
4) added a warning log on missing loopback interface when creating bgp router
5) import os version to prouter object
6) Fixed pep8 warnings and errors
Partial-Bug: #1805237

nodemgr: add checking time sync by crhony
Closes-Bug: #1800704

Dont include "lost+found" folder in /var/crashes
If /var/crashes is mounted, fsck can create
"lost+found" folder in it. The core list reported
by nodemgr will contain this folder, even though its
not a core file. This assumes there is no core file
reported by the name "lost+found" and it can also
report other folders/files reported in var/crashes.
Closes-Bug: 1784069

Adding Introspect for mesos manager
Fix for bringing introspect for mesos-manager
objects on port 8109
Partial-bug: #1795295

Agent changes for inter-as option c part1
1. added new table int.3 to process labelled inet routes
2. added new nexthop type labelled tunnel nh
3. vpn routes with tunnel type as mpls depends on inet.3
routes to resolve its nexthop
partial-Bug: #1743517

Mesos api server not responing to 127.0.0.1 ip
Setting api server ip to be accessed through the node ip
Closes-Bug: #1807302

Adding default as_path for 2 byte asn neighbor
With 4 byte asn changes, by default as_path with 4 byte asn
was being added. It should be as_path with 2 byte asn if the
peer does not support 4 byte asn.
Closes-Bug: #218663

build fix for inter-as changes
Partial-Bug: #1743517

Account for flow count on interface if
flow is ingress flow , increase flow count by 2
for interface one for forward and another for reverse.
If zero is set on vmi take value from virtual network.
Closes-Bug: #1805337
Closes-Bug: #1806866
Closes-Bug: #1804823

Agent crash @ assert in VxlanRoutingManager::BridgeVnNotify
Porting the fix from R5.0 - removed the assert
Closes-bug: #1767044

Split flow management to multiple files
Partial-Bug: #1800894

[Fabric] Enhancements for Generic Device Operations
Included the following changes on the controller:
1. Included the output_ui_schema in the predef paylaods.json for job_template_object
2. Included the actual output_ui_schema in the schema.json files
3. included the output_ui_schema in load_data under api-server
4. Resolved merge conflicts and changed description to job_template_description in predef_payloads.json
5. Changed description in schema for better understanding
Closes-Bug: #1806508

Fix for multiple interfaces in pod showing in ECMP endpoints
This addresses the issue of multiple interfaces showing up as ECMP
endpoints when a loadbalancer is created for a multi interface pod.
With this fix only the default interface is added as an endpoint
Closes-Bug: 1791460

Make XMPP talk TLSv1.2
Use SSLv23 for TLS connections and then set the context options to exclude
SSLv1, SSLv2 and TLSv1 thereby forcing the connection to use TLS v1.2
Partial-Bug: #1807296

Fix most of level 1 warnings on Windows
- Invalid escape sequence
- Invalid unused field and getter
- Invalid preprocessor usage
- Wrong type of field
- Missing return in non-void returning functions
Partial-Bug: #1800894

Fix warning related to not called destructor
Partial-Bug: #1800894

[DM] Hitless image upgrade
Files and roles added to support hitless image upgrade
Closes-Bug: #1803626

Revert "Fix for active TCP flows not transitioning"
This reverts commit c0c44258aa9c9f1ecd1b8d6f9541fb1accca3bb0.
Closes-Bug: #1786924

schema-transfomer: don't update RT list when SI init isn't done
* When RoutingInstanceST are initialized, static routes are updated.
* This update trigger an update of route targets list which should not
be updated in the case where the RI is owned by an SNAT SI and that
latter has not been initialized yet in the reinit function.
* Prevent the schema to remove RT of SNAT's RI and so prevent to lose SNAT
traffic temporarily when restarting schema-transformer.
Closes-Bug: #1787371
(cherry picked from commit 182f70500ca104b50b26bc27906939eba49e0109)

Asn should be treated as uint
With 4 byte ASN, introspect is displaying higher asn as -ve
number because those are defined as u32 in sandesh files. They
should be defined as u32 instead.
Also Rtarget should take into account 4 byte asn properly.
If asn <= 0xFFFF, we'll create rtarget with 2 byte asn value,
otherwise, rtarget will have 4 byte asn
Partial-Bug: #218680

[DM] Add device health check for hitless upgrade
Closes-Bug: 1803614

bgp-peer selection support for bgpaas
1. Added introspect to BgpRouter and ControlNodeZone
2. Added Unit tests to validate secondary control-node-zone
Partial-bug: #1775872
Partial-jira-bug: #JCB-184435

[Config] Add provisioning scripts for Config Database and WEBUI
Inorder to support the following feature
https://github.com/Juniper/contrail-specs/blob/master/5.1/ansible-delete-roles-and-nodes.md
we need to add support to register two new nodes, WEBUI and CONFIG DB.
In this change, added two files for provisioning webui and config database nodes.
These files are similar to provision_database_node.py
Closes-Bug: #1799269
Closes Jira Bug: JCB-216383

Support DNS reverse zones for IPv6
Automatically generate reverse zones for IPv6 addresses OpenContrail
assigns to tenant VMs. Also, make contrail-api accept ip6.arpa names
alongside in-addr.arpa ones, and adapt provision tools to create them.
Finally, support having NS records in reverse zones to delegate a
sub-zone to another nameserver.
Closes-Bug: #1730584
Signed-off-by: Valentin Sinitsyn <email address hidden>
Signed-off-by: Alexey I. Froloff <email address hidden>

Adding support to add timer interval through config
Changing hard coded logic of forever syncing to time based which can be
acquired by conf file.
Closes Jira Bug:JCB-168187

Remove unused variables
Partial-Bug: #1800894

Fixed boundary condtion check for ksync bulk msgs
Agent assert is seen when the size of the bulk message sent to vrouter
is equal to 4096 as the message size should be less than 4096.The code
allows to form bulk message upto 4096.
Fixed this boundary condition issue.
Closes-Bug: #218722

Active TCP flows not moving to deny upon SG attach
The bug manifested coz we reflexively update a reverse flow,
when a forward flow is updated. We call UpdateReflexiveFlow()
for reverse flow whenever a forward flow is updated.
After the SG attach(allows only egress) to Server port, the
forward flow moves to 'D' and reverse flow to 'H'. When TcpAck
from server port hit the reverse flow, flow is reevaluated and
it becomes F(since egress is allowed and this is TcpAck). And
this reverse flow becomes forward and previous forward flow(which
is 'D') becomes reverse. Now the new reverse flow is updated
reflexively and copies the sg_action_summary from Forward flow to
be toggled to 'F', which leads to the reported problem. This will
allow back traffic from client to server.
Fix here has been to not reflexively update the reverse flow for
forward flows, triggered because of TcpAck packets. This allows the
keepalives from server to work, while allowing the C->S flow to be
dropped.
The change has been made in ResyncFlow since this is called first
for reverse flow and then for the forward flow(expecting the reverse
flow to be updated again as part of the ResyncFlow code).
Closes-Bug: #1786924

[DM] Add strategy playbook for hitless upgrade
Closes-Bug: 1805981

[DM] Include LAG configuration for maintenance mode
Update jinja template to include LAG configuration to put a leaf into maintenance mode
closes-jira-bug: CF-29

Mark graceful_restart_flap_all_test7 as flaky
Partial-Bug: 1733446

schema-transfomer: clean stale route-target properly
When RouteTargets are reinitialized, the stale route targets (route
targets which doesn't have logical router backref or routing instance
back ref) are getting removed. But its not removed properly. Route
target remains on cls.__dict. Call the 'delete_vnc_obj' method which
cleans the route target properly.
Closes-Bug: #1808518

Firewall: traffic filter between two AGs as EP fails
We have made changes for address group labels to do 'OR' operation
instead 'AND' between endpoints
Closes-Bug: #1770518

DCI: DCI UT (covers API tests) and global system config Loopback NS update issue
Partial-Bug: #1794010

Don't overwrite createvrf flag for mirror config updates
Agent sets create vrf flag to true if it creates VRF as part of mirror
entry addition. when mirror entry is deleted , Agent checks vreate vrf flag,
if it is set then deletes teh VRF. In this scenario, mirror config updates
are overwriting the flag to false and that blocks the VRF deletion
made changes not to overwrite the flag if it set.
Closes-bug: #1803264

vnc_cassandra: encode keys before calling pycassa
Closes-Bug: #1808132
(cherry picked from commit d7cad149c26325790f5f0dfb9059c8cb50452fe4)

Disable log level checks for flows
The flow messages are sent at LOG_INFO by default.
If the configured log level is anything more flow
messages will get dropped. Default configured log
level is SYS_NOTICE. For flow messages to be written
to the syslog, the configured log level has to be
SYS_INFO; but this can cause a huge churn. By
disabling this check only for flow messages, they
will be logged as long as the destination is set as
Syslog.
Partial-bug: #1811001
Partial-jira-bug: #CBS-7

bgp-peer selection support for bgpaas
Added missing link delete in cleanup
Partial-bug: #1775872
Partial-jira-bug: JCB-184435

Add analytics-alarm and alalytics-snmp nodemgr.
Adding provisiong code for alarm/snmp nodes
Partial-Bug: #1784493

[DM] Set the IRB IP as the default gateway IP for ERB
closes-jira-bug: CF-41

Job Manager and ZTP refactoring
- Moved Job Manager to DM
- DM manages ZTP
- Using DNSMASQ as DHCP and TFTP server
- Using Rabbit MQ for Job Manager and ZTP communication
- Implemented ZTP config and TFTP update in DM
- Added Kombu AMQP client
- Added APIs to api-server for AMQP
Implements: https://blueprints.launchpad.net/juniperopenstack/+spec/dm-pod
Closes-Bug: #1780580
Closes-Bug: #1793811
Closes-Bug: #1794887
Closes-Bug: #1796762

Fat flow AggrDst: IPv6: ICMP6 Flow in hold state
- This is a day1 issue where fat flow for ICMP6 was not supported/working.
Agent Changes (Also see vrouter changes) :-
1) ICMPv6 rule will be stored with proto 1 and port 0 (similar to ICMP rule)
2) Rule lookup for ICMPv6 will happen using proto 1.
Additional fix for IPv6 rule match when src/dst prefix is empty.
Closes Jira Bug:JCB-206313

Chaning the index of route target if going beyond boudary
With 4 byte ASN, index field in route target can only be 2 bytes.
However, schema generates default rtargets with 4 byte value. Real
fix would have been to change those values in schema itself but it
has its own challenges. Here, adding a hack in control node to
change index of a rtarget if asn is 4 byte and index is also 4 byte
(only if in the range of auto generated rtargets)
Partial-Bug: #1807123

Manage rtarget routes associated with bgpaas sessions
With bgpaas static configuration for associated control-nodes, it is possible
that bgpaas sessions can connect to a control-node in a routing-instance, even
though there is no explicit xmpp agent subscription for the same. If this
happens, then as is, control-node will not attract any routes of the instance
if route-target-filtering is enabled (which is the default configuration)
It is only when agents subscribe do we add import-target to the bgp.target.0
table so that instance routes are attracted in bgp.l3vpn.0 table.
With this change, when ever bgpaas session comes up in a routing-instance,
then all its associated import route targets are added to bgp.rtarget.0
table with source peer as bgpaas peer pointer. This is similar to what we
do when an agent subscribes. On the other hand, when bgpaas session goes
down, then those routes are removed from the bgp.rtarget.0 table.
Also, current set of route-targets configured for an instance are tracked so
that in case of configuration change to the import route targets list, routes
are correctly updated.
TODO: Add more unit tests to cover routing-instance configuration changes.
Partial-bug: #1775872

[Config] Add support to log remote IP address
As per the current implementation, WEBUI host IP is sent as
remote IP in the logs sent to Analytics.
This change will add actual remote IP address of the end user
when sending the log information to Analytics.
To do this, a WEBUI change will be done seperately. This code
review is to handle the changes in vnc_cfg only.
Closes-Bug: #1808574
closes-jira-bug: JCB-189711

[DM] Improve hitless upgrade error handling
closes-jira-bug: CEMDA-60

pass ssl options to all instances of cassandra's client
partial-jira-bug: JCB-218753

[DM] Job concurrency support
Added zookeeper lock based fabric and device level concurrency control
for jobs
closes-jira-bug: CF-25

Added a defensive check in casting NH in AddReceiveRoute
AddReceiveRoute assumes the active path to have an NH with interface. This
resulted in agent coring when the active path pointed to composite NH. Adding a
defensive check to proceed only if NH is INTERFACE or RECEIVE.
Closes-Bug: #1795090
(cherry picked from commit 879df7e1c20f19adefcb69c7ed42d341117b3137)

[DM] Vendor agnostic changes - initial checkin
1. Initial checkin for framework testing
2. Device import tested for vendor agnostic device querying / command execution
3. Device import tested for filter plugin invoking
4. Device import tested for functionality with refactoring
5. Device import including debug logging functionality
6. Topology discovery tested for vendor agnostic device querying / command execution
7. Topology discovery tested for filter plugin invoking
8. Topology discovery tested for functionality with refactoring
9. Topology discovery including debug logging functionality
10.Unified logging across all modules into a single instance
11.Included device name in debug logs also
12.Included validators at the beginning of parsing
13. Included schema checks.
14. To Modify parsing methods and include some annotations
Patch Set #23 changes:
1. Resolved merge conflicts
Patch Set #24 changes:
1. restructured files into different folder  strucure
2. deleted unwanted files
Patch Set #28 changes:
1. Restructured directory organization and file names using appropriate naming convention
2. Included doc strings for modules
3. Removed vnc_util and added functionality in JobVncApi
Patch Set #32 changes:
1. Changed lo0 ip address tracking to list in order to save the value in jinja processing
Patch Set #35 changes:
1. Resolved merge conflicts
Closes-Bug: #1785127
Closes-Bug: #1784968
Closes-Bug: #1786319

Metadata requests should use fqdn instead IP
partial-jira-bug: JCB-218839

[DM] Use container label to find and restart
closes-jira-bug: JCB-218842

[DM] Simplified plugin manager logic
partial-jira-bug: JCB-218842

[APISERVER] - Adding 'None' (str) check for gateway and service address while creating Subnet object
Closes-Jira-Bug: JCB-211810

[CFM] enable multi-homing unit test
Partial-jira-Bug: CF-39

Add Support for: Server Discovery, Server Import and Node Profile Import
Servers and Node Profiles can be imported from YAML or JSON Files
Patches:
Fixed Job Ctx to use Single Auth token end to end
Fixed Try Catch Blocks
Removed whitespaces
Removed auth vars from group_vars/all.yml -> Should be in test code
Can import nodes without ports
IPMI Port Range format (Min Max)
Fixes for Auth Token
Added decription for schema
Removed Ironic in Import schema (not needed)
Added YAML/JSON enum for validation of format
Removed failed import
Partial-Bug: #1807762

Fixing some flaky test in agent
There were some tests marked as flaky. Fixing them so that there is
better code coverage for agent code.
closes-jira-bug:JCB-218754

Added validation check for BgpRouter Object in Contrail-api
Control-Node-Zone should be linked only BgpRouter type of
"control-node". Added check to validate the same in BgpRouter
pre_dbe_create() and pre_dbe_update()
Closes-jira-bug: CNET-75

Use semicolons properly with 'while (false)' in macros
Partial-Bug: #1800894

get inet labeled address family enable info
partial changes to get inet labeled address family info
from bgp router config.
workaround for UI issue:
setting tunnel encapsulation type as MPLS over MPLS
if labeld inet is enabled, will remove these changes
once UI fix is in.
Closes Jira Bug:JCB-218795

[DM] fix regression issue in device job manager
closes-jira-bug: CEMDA-89

[DM] Device states not updated properly during hitless image upgrade
closes-jira-bug: CEMDA-69

enable SSL for cassandras' clients
partial-jira-bug: JCB-218753

Agent EVPN Multicast Changes
partial-jira-bug: JCB-218799

increase version of db_manage.py script
partial-jira-bug: JCB-218753

Fixed flaky test-cases in test_sg_flow and test_sg_tcp_flow
Fixes in test_sg_flow.cc
1. XML parsing code had extra white-spaces, hence removed it.
2. Call AddIPAM() from Init() and remove DelIPAM() from TearDown() as it
is already getting called in Shutdown. This way it will be similar to
test_sg_flowv6 which is already working.
Fixes in test_sg_tcp_flow.cc
1. XML parsing code had extra white-spaces, hence removed it.
2. We were not deleting bgp_peer_
Updated SConscript to make above test-cases as non-flaky.
Closes Jira Bug:JCB-218833

AMQP Client: Fixed typo in keyfile attribute
closes-jira-bug: JCB-218897

[vnc openstack] Add support of Neutron FWaaS v2 API
Add support to the Neutron FWaaS v2 API extension [1] as described in
the spec [2].
[1] https://developer.openstack.org/api-ref/network/v2/?expanded=id341-detail,id346-detail#fwaas-v2-0-current-fwaas-firewall-groups-firewall-policies-firewall-rules
[2] https://github.com/Juniper/contrail-specs/blob/master/neutron_FWaaSv2.md
Partial-Jira-bug: JCB-204217

Route Reflector code clean up in ST
Cleaned up route reflector related code that got added previously.
It was causing some unexpected behaviour like calling update peering
on all the bgp routers on an update to any one of the bgp routers
even when it's not required.
Closes-Jira-Bug: JCB-218900

[DM] Filter logic for hitless upgrade flag
closes-jira-bug: CEMDA-59

[DM] Regression in logging - Vendor Agnostic Checkin
Refactoring logging to a single file was missed out for some places
in fabric.py for role_assignment.
Also included the refactored model of logging for hitless_upgrade_filters.py
and write_to_file_filter.py.
closes-jira-bug: CEMDA-113

Packaging Ironic and Contrail Command Plugins
These were missing and causing import error when any fabric job is called
closes-jira-bug: CNET-101

Check for consumers before draining events
If the AMQP client proceeds with draining events without having any consumers
that could delay the consumer creation and loss of messages
closes-jira-bug: JCB-218915

db_manage: Make Individual connection timeout and buffer size user configurable
closes-jira-bug: JCB-218813

[DM] Hitless Upgrade Health Check Improvements
closes-jira-bug: CEMDA-117

FWAAS support
Partial-Jira-bug: JCB-204217
Add agent changes to support FWAAS policies
Added UT to test the FWAAS gate

[DM] Create QFX-specific basic config template
closes-jira-bug: CEMDA-119

To enable api-server SSL, enable bottle ssl. The following parameters
need to be passed from deployer to the api-server configuration file:
'config_api_ssl_enable': <boolean>
'config_api_ssl_keyfile':
'config_api_ssl_certfile':
'config_api_ssl_ca_cert':
the webui_job_1 & webui_web_1 uses VncApi client, it need to be configured
to use ssl as well.
closes-jira-bug: JCB-218811

AS4PATH was not being processed for 2 byte neighbours
AS4PATH attribute was added to handle cases where 2byte node interacts
with 4 byte node but somehow it was not added in the list of attributes
to be handle for 2 byte neighbours.
Closes-Jira-Bug: JCB-218927

[CFM] Ansible filter to read from db directly
Helps in reading objects from DB directly
Closes-jira-bug: CEMDA-31

Prefer EBGP over IBGP first before looking at path-id
During BGP best-path selection, EBGP paths should be preferred over IBGP
and only then should we look at the path-id (which is typically the nexthop
address which is pretty much a random selction)
Because BGPaaS peer based ebgp route-target route was not preferred as the
best path, route target filtering caused routes not to get advertised
Closes-Jira-Bug: CNET-121

[DM] Populate default annotations on fabric object
- Example annotations listed in bug CF-44 comments
- Currently annotations only used for hitless upgrade and fabric creation
closes-jira-bug: CF-44

- Changes for schema transformer to be aware of fabric when handling
mp-bgp peering.
- Added UT to handle the fabric aware schema transformer mp-bgp peering
closes-jira-bug: JCB-218878

Enabling agent UT test script.
Closes Jira Bug:JCB-218757

Fix type conversion related warnings for common/base/task*
Partial-Bug: #1800894

[DM] Enhance information provided to the user during job execution
- Fixes bug with JobAnnotations json import
closes-jira-bug: CEMDA-44

[DM][fabric] IPv6 support
- Updated abstract config to have family and gateway for every IP address
- Deprecated 'ip_list'
- Updated Jinja templates to use address family from abstract config
closes-jira-bug: CEMDA-142

Set psutil version to last working one for nodemgr
Psutil==5.5.0 throws exceptions in nodemgr on Windows.
This issue was reported to the maintainers
and should be fixed in future versions.
Closes Jira Bug: WIND-158

[fabric] Fixed JSON error
closes-jira-bug: JCB-218973

[fabric] Enable IGMP snooping for ERB
closes-jira-bug: JCB-218983

[DM] Deletion of an onboarded fabric device fails
closes-jira-bug: JCB-218968

[DM] Error handling for maintenance mode
During any failure through the entire workflow, deactivate MM if it is in the active state
closes-jira-bug: JCB-218949

Fix HA issue with VncApi usage
Use api server ip list instead of a single ip while creating the vnc api client.
Randomize the list to ensure load balancing
Closes-jira-bug: JCB-218962

[DM] Check VN is valid when setting IRB
closes-jira-bug: JCB-209995

Mark graceful_restart_flap_some_test2 as flaky (for now)
Partial Jira Bug: JCB-218991

[DM] change LinkAggregationGroup to VirtualPortGroup
Partial-jira-Bug: CF-39

[DM] Hitless upgrade health report printing problems
closes-jira-bug: CEMDA-149

Kubemanager should wait till Global vrouter object is created by VNC API server
Closes-Jira-Bug: JCB-218957

Rtarget should be valid when attaching to RI
Whenever a route target is attached to a routing instance, its validity
should be checked, especially with 4 byte ASN.
Closes-Jira-Bug: JCB-219010

add missing files/folder to fabric-ansible packaging
closes-jira-bug: CNET-101

[DM] import lldp playbook bails out if there an interface has multiple neighbors
This bug has been fixed for shared interfaces with multiple neighbors.
closes-jira-bug: JCB-218966

Marking igmp_test as flaky
Mark igmp_test UT as flaky for now. Will be addressed as part of cleaning up UTs
Closes Jira Bug: JCB-219033

[fabric] Enable igmp for IRB and set multicast-replication mode
closes-jira-bug: JCB-219035
closes-jira-bug: JCB-219036

[DM] Remove rb_routing roles from hitless upgrade logic
closed-jira-bug: CEMDA-156

[fabric] Use FilterLog for logging in db_filter
closes-jira-bug: JCB-219027

Core file deletion in nodemgr
The nodemgr instances delete the core files of processes it
controls. If the core files belonging to a process exceeds 4,
nodemgr deletes the oldest entry. At any point in time only
4 core files for each process that nodemgr controls in allowed
to exist.
Partial-Bug: 1776170

[DM] Percentage calculation change to include two modes
Percentage change to include both hitless and non-hitles mode
closes-jira-bug: CEM-299

[DM][fabric] Fixed error in job status notification
closes-jira-bug: JCB-219030
closes-jira-bug: JCB-219031

[config] Split resource API server code
Split the huge 'vnc_cfg_types.py' file in one file per Contrail
resources type. All new file will follow pep8 recommendations and the
patch adds that pep8 checks in the scons test target to be sure pep8
compliance will not be broken. it also moves from testr to stestr as
testr is not anymore maintained.
Closes-jira-bug: JCB-218755
Partial-jira-bug: JCB-218756

nexthops are not added correctly in vrouter.
Made changes in nexthop ksync compare function to compare
transport label for labelled tunnel nexthops.
changes to support tunnel type change( MPLS over GRE/MPLS over UDP)
for MPLS over MPLS tunnel based on global tunnel encapsulation
priority list.
Closes Jira Bug:JCB-218985

[vnc openstack] Add config knob to enable FWaaS on VNC OpenStack API
When the config flag 'NEUTRON.fwaas_enabled' is set to true, the default
firewall group is automatically provisioned in project and applied to VM
port by default.
Partial-Jira-bug: JCB-204217

[fabric] Add ipv6 static route under rib for MX
closes-jira-bug: CEM-305

[DM] Use updated docker api
Partial-Jira-Bug: JCB-218848

Reverting the changes related to db_filter
Removing the db_filter since we cannot use the vnc_db_client
within the ansible process space.
Closes-jira-bug: JCB-219027

[DM] Hitless image upgrade device_list processing
closes-jira-bug: CEM-144

[schema transformer] Clean RT when a VN is deleted
API server is now in charge of the creation and deletion of the primary
RI of a VN but the schema stills in charge to allocate a unique route
target number and create/delete the corresponding RT. But actually, when
a VN is deleted, schema tries to delete all referenced RI (in memory and on the
VNC API for all the RI and only in memory for the primary RI). But frequently,
it deletes the primary RI before the VNC API have deleted it and it
tries to delete the allocated RT but that failed as it still referenced
by the primary RI not yet deleted. Then when the primary RI is deleted,
the schema ignore the notification as that RI is not anymore in its memory and
the RT remains (lock in zk was removed and could be re-used).
That patch changes the schema to not delete the primary RI when a VN is
deleted and lets the notification of the primary RI deletion warns the
schema to delete the RI and its allocated RT.
Closes-Jira-Bug: JCB-219068

[DM] Move vrf import reject maintenance mode to evpn overlay
closes-jira-bug: CEM-306

parse string to boolean flag for api_ssl_enable option
partial-jira-bug: JCB-218849

[DM] Handle untagged vlan for lag/mH
Closes-jira-bug: JCB-219032
Closes-jira-bug: JCB-218974

Fixed flaky test-cases in test_vrf_assign_acl.cc
1. XML parsing code had extra white-spaces, hence removed it.
2. While verifying flow action we need checking if particular
flag bit is set. Made it same as class VerifyFlowAction.
3. Updated SConscript to make above test-case as non-flaky.
Closes Jira Bug:JCB-218833

Agent support for Multicast Policy defined in schema.
Implementation of schema field "virtual-network-multicast-policy"
in contrail agent.
Also includes changes required for integration with IGMP module
for <S,G> policy checks.
partial-jira-bug: JCB-218799

[config] Add validation check for SubnetType
Validate any create or update of all SubnetType attribute with netaddr
python library.
Closes-Jira-Bug: JCB-218787

support new interface of python-docker library in nodemgr
nodemgr can work with both interface with this fix.
later old interface support can be removed and code
can be rerworked to use new features.
Partial-Jira-Bug: JCB-218848

Added dependency path in bgp-as-a-service
Dependency path was missing between bgpaas-control-node-zone
and bgpaas-virtual-machine-interface in bgp-as-a-service.
Due to that bgpaas-virtual-machine-interface event got missed
in virtual-machine-interface node which caused the issue.
Added missed dependency path in bgp-as-a-service.
Closes-jira-bug: JCB-219072

[DM] - Added API server validation while creating service appliance object
- Added logic to allocate VLAN, each left and right per service instance
- Fix build compilation
- Fix unit tests
- Allocate two internal VN's for management of RT's for left and right service VRF
- Fix VLAN allocation logic for service instance creation
- Fix physical interface ref creation during service appliance creation
- Fix port tuple deletion errors
- Added logic to allocate IRB units for spine device using virtual network ID
- Added logic to create dummy loopback interface for PNF device to be used for
BGP peering on PNF device
- Fix build error
- First clean-up, removing un-wanted code
- Added pre dbe create validation for service instance
- Handled most of Suresh review comments
- Handle Edouard review comments
Closes-Bug: #1807426

fixes for getloadavg using wmi
Partial-Bug: #1789754

[DM] Missing error handler in Sconscript
closes-jira-bug: CEM-325

[DM] - Updating local cache for new VNC objects
- Defining new PNF plugin for PNF physical role
- Updated the reaction map based on new logic and data-model changes
- Fix DM integration issues
- Add initial logic for rendering PNF abstract config
- Add business logic for QFX abstract config
- Added a check to check for PNF physical role for PR's with no BGP router
- Handled Suresh review comments
- Added checks while generating abstract config, so that config push happens
when all required parameters are present
Closes-Bug: #1800701

CFM: PNF Service chaining ansible playbooks
Closes-Bug: 1800701

[Config] Fix for lost update problem in Security Group Rules
When updating a security group in parallel to add/delete multiple rules,
we see that there's a lost update problem leading to undefined behavior.
This change is to fix this problem by implementing a
Zookeeper lock inside neutron_plugin_db of API Server.
Closes-Jira-Bug: JCB-218976
Closes-Jira-Bug: JCB-181802

Fixed flaky test-cases in test_flow_error
1. On setting block_msg_processing_ as true flow_stats_manager was
not able to collect short flows for deletion and hence it was
not getting deleted.
2. Flows are actually not deleted, we just reset them. Hence
fe->deleted flag is reset to false and hence assertion happens.
So we need to delete check for fe->deleted().
3. Added a check to see that flow count should be 0.
4. Added a check to see if it is shortflow.
5. Updated SConscript to make above test-case as non-flaky.
Closes Jira Bug:JCB-218833

[DM] Fix hitless upgrade file
Fix the typo in hitless upgrade file that was changed during percentage update.
closes-jira-bug: CEM-337

master: slow vrouter convergence with BFD health check
In BFD Poll phase, BFD Detection time is not computed properly.
1) Detection time Fix
After the BFD session is UP,  when remotes bfd.DesiredminTxmin
decreased due the detection time is not recomputed.
Earlier code was only computing if the bfd.DesiredminTxmin is
greater than its previous advertised value but not when the
value is decreased.
2) Also fixed if in poll phase if the remote session's
minRxInterval is changed. If the value is increased,
the new value will be used when recomputing the SendTimer().
3) Added UT BasicSingleHop_CfgChange_DetectionTime to validate
the detectiontime.
Closes-jira-bug: CEM-294
(cherry picked from commit beb33969652c5a55449de49f50af8b421fa860a2)

Server-Disocvery: PI-Port Ref, EndSystem-Node-Profile ref fixes
Partial-Bug: #1805237
1. on create , create ref_link
2. on update, remove old link and create new link
3. for deleting port, ref link needs to be deleted by client.
4. Moved from Node to EndSystem
5. if node_profile_ref is found in EndSystem, remove old tag ref from
EndSystem-Ports and then re-create new ref to tag
6. create port-groups if specificed in node-profile
7. remove port-groups refs before creating new ones
updated code based on newer hierarchy and recent schema changes
changes to comply with pep8
fixed any unit-test failures
fixed pep8 failures as well

do not call keystone_connect from extension init
it leads to exception because keystone is not ready yet.
and due to excpetion the extension becomes not loaded.
and config-api tells that keystone connection is not ready until config-api is not restarted.
closes-jira-bug: JCB-176427

DM: Use vlan-id none for irb interfaces in the case of Spines.
We were assigning vn-id as vlna-id which is not recommended.
https://junipernetworks.sharepoint.com/sites/EMC-multi-cloud-NPI-docs-site/Shared%20Documents/InterLock/EVPN-VXLAN-Configuration-Examples-for-Quick-Referance.pdf
close-jira-bug: JCB-210290

Set key name in routing-instance UVE during deletion
Because key was not, UVE remains in the cache locally and
in the remote collector
Closes-Jira-Bug: JCB-218965

Fixed kombu client consuming and hearbeat logic
- Run hearbeat check only if there are consumers
- Handle connection status explicitly
- Wait for consumers in consumer greenlet
- Payload check when publishing
- Fixed error in docker api usage
partial-jira-bug: JCB-219115

ISSU run docker under root
- in case of OSP for ssh heat-admin user is used.
Partial-Bug: #1801956

Move the multi_device flag from the job template to playbook
Use this flag to control the spawning of greenlets within the job manager
instead of the device json.
Closes-jira-bug: CEMDA-31

[DM] Read leases file line by line
partial-jira-bug: JCB-219115

Support comma separated list of interfaces
As per the multi-interface spec, annotations for networks can now
be a comma separated list.
Also fixes un-necessary update of networks and display formatting for
network-status annotation
Closes-jira-bug: JCB-219125

For local ECMP NHs, pick the vif from the child member NH while creating
fat flows
Today, in case of local ECMP NHs, the flow gets created using the ECMP NH
as Key NH because of which the vif interface couldn't be retrieved to apply
fat flow config.
With this fix, we will check for local ECMP NH, and if so we will pick the
vif from the first member NH. If the first member vif interface has
fat flow configured, then it will be applied.
closes-jira-bug: JCB-219075

[config] Fix vitual DNS resource name typo
During the refactoring code patch, the Virtual DNS resource type name
was not correctly wrote. Resource type name is 'virtual_DNS' instead of
'virtual_dns'.
Closes-Jira-Bug: JCB-219182

Check for non-FDN vrouter name if FQDN names do not match.
closes-jira-bug: JCB-219162

FWAAS support
Partial-Jira-bug: JCB-204217
Add agent changes to support FWAAS policies
Missed to checkin flow_mgmt changes

If Kubernetes API server watch is not set during event process, call watch
register again.
closes-jira-bug: JCB-219181

Crash with sub-interface on VM side only.
Sub-interface not created on contrail side.
VN IPAM is not picked resulting in ipam being NULL.
Hence the crash. putting a check for ipam being NULL.
Unexpected use-case, but should be handled.
partial-jira-bug: JCB-219107

[DM] Removing routing instances delete as part of fabric deletion
Removed explicit deletion of Routing Instances
Also fixed one of validate role assignment checks
closes-jira-bug: JCB-219195

[Fabric] - Changing int to str before concatenation
closes-jira-bug: JCB-219172

Change-Id: I0fd2d010fe5448d9ced5061690acbc85a31bc3f5
Depends-on: Icd111bf92e4656fe7187726cd95ceabb80b9415f
Partial-jira-bug: JBE-925