[K8s-R5.0]: Priority order of attached FW policy to default APS named k8s turns wrong and k8s-Ingress policy loses its association with k8s APS

[Pulkit Tandon]

Configuration:
K8s 1.9.2
coat-5.0-45
Centos-7.4

Setup:
5 node setup.
1 Kube master. 3 Controller.
2 Agent+ K8s slaves

In a freshly configured setup, ran a test case which involve creating of new namespaces, pods and network policies.
Noticed that the association of k8s-Ingress Firewall Policy with APS "k8s" is lost.
Also, the policy priority order is not maintained.
This result in k8s-allow policy to have the highest precedence.

Because of this, no firewall policy is going to take any effect and all traffic will pass.

Attached are the snapshots to explain the issue

[Pulkit Tandon (pulkitt)]

Workaround is to restart all the contrail-kube-manager

[Pulkit Tandon (pulkitt)]

Observed the same issue again on build 49.
Logs kept at <email address hidden>

Path:
bhushana@mayamruga:/home/bhushana/Documents/technical/bugs/1768475/

Initially, nodeg31 was the the active kube-manager.
With the timestamp of logs of contrail-api.log, I think that the old logs are lost due to more logging. I don't see the old backup logging files as well.

[Dinesh Bakiaraj (dineshb)]

This issue needs to be debugged from Config team.

The issue is as follows (illustrated by VncApiConfigLog logs from analyticsdb) :

1.Kube-manager adds a FW policy called k8s-denyall to an empty APS.

2018 May 09 23:09:31.767435 nodec58 [Config:contrail-api:0:__default__][SYS_INFO] : VncApiConfigLog:21 [VncApiCommon: identifier_uuid = 9010d94b-d2ae-4eb8-a0c4-2a263f625b84, object_type = application_policy_set, identifier_name = default-policy-management:k8s, url = http://77.77.1.11:8082/ref-update, operation = ref-update, useragent = nodeg31:/usr/bin/contrail-kube-manager, remote_ip = 77.77.1.11:8082, body = {"ref-type": "firewall-policy", "uuid": "9010d94b-d2ae-4eb8-a0c4-2a263f625b84", "ref-fq-name": ["default-policy-management", "k8s-denyall"], "ref-uuid": "f1a4b630-1bbb-477c-9f00-9f8e1e078722", "operation": "ADD", "type": "application-policy-set", "attr": {"sequence": "00001.0"}}, domain = default-domain]

2. Kube-manager then adds a second FWP called k8s-allowall. It tries to insert it after k8s-denyall. As a part of this add, k8s-denyall is also modified to a new sequence number. In the vnc lib, this modify is treated as Delete followed by ADD. So the expectation is that the k8s-denyall policy will first be deleted and then be added.

However, in a multi-config node setup it is observed that the sequence of operation performed is ADD followed by Delete.

2018 May 09 23:09:32.018468 nodec58 [Config:contrail-api:0:__default__][SYS_INFO] : VncApiConfigLog:22 [VncApiCommon: identifier_uuid = 9010d94b-d2ae-4eb8-a0c4-2a263f625b84, object_type = application_policy_set, identifier_name = default-policy-management:k8s, url = http://77.77.1.11:8082/ref-update, operation = ref-update, useragent = nodeg31:/usr/bin/contrail-kube-manager, remote_ip = 77.77.1.11:8082, body = {"ref-type": "firewall-policy", "uuid": "9010d94b-d2ae-4eb8-a0c4-2a263f625b84", "ref-fq-name": ["default-policy-management", "k8s-denyall"], "ref-uuid": "f1a4b630-1bbb-477c-9f00-9f8e1e078722", "operation": "ADD", "type": "application-policy-set", "attr": {"sequence": "00002.0"}}, domain = default-domain]

2018 May 09 23:09:32.018468 nodec58 [Config:contrail-api:0:__default__][SYS_INFO] : VncApiConfigLog:22 [VncApiCommon: identifier_uuid = 9010d94b-d2ae-4eb8-a0c4-2a263f625b84, object_type = application_policy_set, identifier_name = default-policy-management:k8s, url = http://77.77.1.11:8082/ref-update, operation = ref-update, useragent = nodeg31:/usr/bin/contrail-kube-manager, remote_ip = 77.77.1.11:8082, body = {"ref-type": "firewall-policy", "uuid": "9010d94b-d2ae-4eb8-a0c4-2a263f625b84", "ref-fq-name": ["default-policy-management", "k8s-denyall"], "ref-uuid": "f1a4b630-1bbb-477c-9f00-9f8e1e078722", "operation": "ADD", "type": "application-policy-set", "attr": {"sequence": "00002.0"}}, domain = default-domain]

This causes the k8s-denyall FWP to be removed from the APS altogether.
This results in incorrect ordering of the FWP's in the APS used by kube-manager to configure K8s network policy.

[Sachchidanand Vaidya (vaidyasd)]

Pls see AuditTrail #5

[Pulkit Tandon (pulkitt)]

Another occurrence of similar issue observed on k8s sanity run on build R5.0-58.

This time, k8s-allow all loses its association with APS.
This happens after kube-manager restart. Restart of Kube manger was done together on all 3 nodes.

Snapshot and kube-api+ kube manager logs are attached

[Pulkit Tandon (pulkitt)]

Also, the rules inside the k8s-allowall firewall policy were not correct.
Please see the attached snapshot

[Shivayogi Ugaji (shivayogi123)]

can we remove the sanity blocker status if you are not able to reproduce the issue and collect the logs discussed offline.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/43768
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/43768
Committed: http://github.com/Juniper/contrail-container-builder/commit/aa4b48ae4c24dba67a75f7416255cfbfa337f483
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit aa4b48ae4c24dba67a75f7416255cfbfa337f483
Author: dgautam <email address hidden>
Date: Tue Jun 12 16:49:52 2018 -0700

add curl_log variable for vnc_api_ini file

Partial-Bug: #1768475

Change-Id: Ia751fc0f74eb25859b9d75e1d9c8826113c39537

[Pulkit Tandon (pulkitt)]

After the fix to this bug, following issue was observed:
https://bugs.launchpad.net/juniperopenstack/+bug/1777277

Waiting for the above bug to get fixed.

[Pulkit Tandon (pulkitt)]

Fix committed for https://bugs.launchpad.net/juniperopenstack/+bug/1777277
Waiting for the new build to come post R5.0-108

[Pulkit Tandon (pulkitt)]

The issue was seen again with logging enabled on R5.0-111
I have locked the setup for investigation of logs. Please give it a look. Let me know if any info is required.

Setup details:

Sorry. Correct one are:
nodeg12(10.204.217.52) – Controller + Kube manager
nodeg31(10.204.217.71) - Controller + Kube manager + kube master
nodec58(10.204.217.98) – Controller + Kube manager

nodec60(10.204.217.100) - Compute + k8s Minion
nodec61(10.204.217.101) - Compute + k8s Minion

[Édouard Thuleau (ethuleau)]

From attached logs, I see the ref to the deny FP was updated (ref-update delete then add) with a new sequence number but then ref disappeared. The ref-update add POST gone well and I don't see any other PUT or POST on that APS which could remove it. I see the ref disappeared on a next GET on the APS (6 seconds after the ref-update ADD POST).

Global APS 'default-policy-management:k8s' (1c21aa9f-5cee-4c98-b0c8-e62c8e01c0ea)
Global deny FP 'default-policy-management:k8s-denyall' (22c9214f-da27-4fd5-a867-e4810ff22ad0)

12:00:27 GET /application-policy-set/1c21aa9f-5cee-4c98-b0c8-e62c8e01c0ea (line 5674 nodeg31/vnc_logs_k8s.log)
FP refs:
- 2 k8s-Ingress
- 4 ctest-ns1-47749395-policy-on-all-pods-of-deployment
- 6 k8s-denyall
- 7 k8s-allowall

12:00:27 multiple POST /ref-update (from line 5689 nodeg31/vnc_logs_k8s.log)
## action FP host req ID notify update received
1/ DELETE k8s-allowall nodec58 req-ef8f5ca1-9132-4fc6-b658-1d681203ec1f (nodeg12, nodeg31)
2/ DELETE k8s-denyall nodeg12 req-e6d47003-6a5a-42cb-86b0-d538eff15ae3 (nodeg31, nodec58)
3/ ADD 8 k8s-denyall nodeg31 req-e7c4e891-38a7-42a6-8d2d-041c87cc1811 (nodeg12, nodec58)
4/ ADD 9 k8s-allowall nodec58 req-bb25ec49-8e61-4e27-b2ab-4b11d49bd08d (nodeg12, nodeg31)
5/ ADD 6 non-default-egress-policy-test nodeg12 req-13a210f3-be29-4ed1-a8a2-78f120e16bd8 (nodeg31, nodec58)

12:00:33 GET /application-policy-set/1c21aa9f-5cee-4c98-b0c8-e62c8e01c0ea (line 5798 nodeg31/vnc_logs_k8s.log)
FP refs:
- 2 k8s-Ingress
- 4 ctest-ns1-47749395-policy-on-all-pods-of-deployment
- 6 non-default-egress-policy-test
- 9 k8s-allowall

[Édouard Thuleau (ethuleau)]

logs nodeg12

[Édouard Thuleau (ethuleau)]

logs nodeg31

[Édouard Thuleau (ethuleau)]

logs nodec58

[Édouard Thuleau (ethuleau)]

Need new setup deployed by the test team to investigate further. Pulkit will hold one when he meets the issue again.

[Édouard Thuleau (ethuleau)]

From kube-manager service logs, it seems sometimes multiple kube-manager service run in the same time. That needs to be confirmed with zk logs but zk container is not correctly configured and logs are lost.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44812
Submitter: Édouard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44812
Committed: http://github.com/Juniper/contrail-controller/commit/57faba1186fba3ddb3b845b655d6ff1478f28f48
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 57faba1186fba3ddb3b845b655d6ff1478f28f48
Author: Édouard Thuleau <email address hidden>
Date: Fri Jul 20 16:54:50 2018 +0200

[config] Add log to object UUID cassandra cache manager

Add debugging logs to the cassandra object UUID cache for tracking issue.
That log are only available if the server log level is set at least to
debug and if some resource types we want to debug are defined in the
option flag 'DEFAULT.debug_object_cache_types'.

Change-Id: Iacd10017c236cc0595bc525ef7b1e3f3c8c756d6
Partial-Bug: #1768475

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44919
Submitter: Édouard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44919
Committed: http://github.com/Juniper/contrail-controller/commit/a18bdba494cd61466122214e7660ff44f112a228
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit a18bdba494cd61466122214e7660ff44f112a228
Author: Édouard Thuleau <email address hidden>
Date: Fri Jul 20 16:54:50 2018 +0200

[config] Add log to object UUID cassandra cache manager

Add debugging logs to the cassandra object UUID cache for tracking issue.
That log are only available if the server log level is set at least to
debug and if some resource types we want to debug are defined in the
option flag 'DEFAULT.debug_object_cache_types'.

Change-Id: Iacd10017c236cc0595bc525ef7b1e3f3c8c756d6
Partial-Bug: #1768475
(cherry picked from commit 57faba1186fba3ddb3b845b655d6ff1478f28f48)

[Édouard Thuleau (ethuleau)]

Patch to add traces on object UUID cassandra table was merged on master and 5.0 branches.
Need to configure log trace like this in the contrail config API server containers and wait error to be reproduced with a 5.0/master sanity run:

[DEFAULTS]
log_level = SYS_DEBUG
debug_object_cache_types = application-policy-set

[Venkatesh Velpula (vvelpula)]

Hi Edouard ,
     I tried couple of sanity run on master and 5.0 release builds. So far haven't hit this issue .
the way that cam configuring the logs traces mentioned above are

-add the below statements in entrypont.sh of config_api container and restart the container .
log_level = SYS_DEBUG
debug_object_cache_types = application-policy-set

 but is there anyway that I can add "debug_object_cache_types = application-policy-set " during provisioning itself so that restart of the config_api can be avoided .I know how to populate log_level.

thanks
-Venky

[Venkatesh Velpula (vvelpula)]

Observed this issue on ocata-5.0.172/173 builds and re marking it as a sanity blocker .

[Shivayogi Ugaji (shivayogi123)]

Config team had a look at the setup with the help of ignatious, contrail-api.conf did not have “debug_object_cache_types = application-policy-set” enabled.
Without this logs added by Edouard will not be generated.

(config-api)[root@nodec58 /etc/contrail]$ cat contrail-api.conf
[DEFAULTS]
listen_ip_addr=10.204.217.98
listen_port=8082
http_server_port=8084
log_file=/var/log/contrail/contrail-api.log
log_level=SYS_DEBUG
log_local=1
list_optimization_enabled=True
auth=noauth
aaa_mode=no-auth
cloud_admin_role=admin
global_read_only_role=
cassandra_server_list=10.204.217.52:9161 10.204.217.71:9161 10.204.217.98:9161
zk_server_ip=10.204.217.52:2181,10.204.217.71:2181,10.204.217.98:2181

From the logs, we saw kube manager being restarted multiple times(13 times) and from the code we saw kube manager was trying reconfigure the same config without deleting config setup by the previous kube manager instance. Needs further discussion with kube manager experts.

It will be good if the problem is reproduced with the “debug_object_cache_types = application-policy-set” enabled.

[Shivayogi Ugaji (shivayogi123)]

as discussed offline in detail, When mater-ship is toggling across kube manager containers this issue can happen when one kube manager sets up partially baked config and next kube manager tries to operate on it without cleaning previous config or without validating previous kube manager config.

kube manager needs to be changed to validate the config set up by previous kube manager and repair the config if anything is partially setup.

[Dinesh Bakiaraj (dineshb)]

Incorrect ordering is a realistic scenario if kube-manager is restarted/killed in the middle of fw configuration.
Kube-manager has logic to reconcile incorrect ordering of firewall policies.
This reconciliation kicks in approx every 60 secs depending on config load.
It may take a little longer to kick in at the time of boot up.
Sanity scripts needs to be changed to wait a little longer. The current guidance is 5 mins before declaring failure due to network policy.

We have identified one more place where reconciliation will help.
This will be addressed in 5.0.2 timeframe.
So moving this bug to 5.0.2 per discussion with team.

[Sachchidanand Vaidya (vaidyasd)]

Problem doesn't happen anymore.

[Sachchidanand Vaidya (vaidyasd)]

Problem doesn't happen anymore due changes in kube-manager.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/46344
Submitter: Dinesh Bakiaraj (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/46383
Submitter: Dinesh Bakiaraj (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/46344
Committed: http://github.com/Juniper/contrail-controller/commit/fd3cfbeaeb99d000d3966327d8edc342b3664f42
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit fd3cfbeaeb99d000d3966327d8edc342b3664f42
Author: dineshb-jnpr <email address hidden>
Date: Wed Sep 19 19:23:11 2018 -0700

Validation of FWP policies at kube-manager start.

This commit adds changes to validate sequence of firewall policies
created by K8s network policy by kube-manager, at the time of
the policy add and at the time of kube-manager startup/restart.

Previously validations were done only at periodic intervals at runtime.
This commit, extends the validation to the time of kube-manager startup.

Change-Id: I23060f34f9dda0aef960bb9a5b429d4ad2339879
Closes-Bug: #1768475

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/46383
Committed: http://github.com/Juniper/contrail-controller/commit/bf1e326b5fc23e99c96b34c02e641a58afeb4912
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit bf1e326b5fc23e99c96b34c02e641a58afeb4912
Author: dineshb-jnpr <email address hidden>
Date: Wed Sep 19 19:23:11 2018 -0700

Validation of FWP policies at kube-manager start.

This commit adds changes to validate sequence of firewall policies
created by K8s network policy by kube-manager, at the time of
the policy add and at the time of kube-manager startup/restart.

Previously validations were done only at periodic intervals at runtime.
This commit, extends the validation to the time of kube-manager startup.

Change-Id: I23060f34f9dda0aef960bb9a5b429d4ad2339879
Closes-Bug: #1768475

[Venkatraman Venkatapathy (vvenkatapath)]

Hi Nick,

The code just got merged yesterday. Will let you know soon if we have a good build.

+ Yuvaraja, Venky

Thanks,
Venkat V
Ph | [phone]
Loc | A.6.361

From: Nick Davey <email address hidden>
Date: Tuesday, November 6, 2018 at 9:14 AM
To: Dinesh Bakiaraj <email address hidden>, Venkatraman Venkatapathy <email address hidden>
Cc: Sachchidanand Vaidya <email address hidden>
Subject: Container Service Chaining Demo Environment

Hi folks,
Would anyone be able to point me at a stable nightly build that features the service chaining code? Anything i need to know to get started building a demo environment?

Thanks!

[Sarath (nsarath)]

Hi,

Please find the detailed QA testplan for Mesos R5.1 based on blueprint doc and as few of clarificiations pending which is on chat-channel, so leaving those LB tests not included and this testplan has all other planned testcases detailed.

Please review and provide your comments.

Note: As per our Team process insisted by Jeba, we need to have “Unit test results” and “Integration results” for acceptance to QA. As talked to Venkat, he has most of provisioning procedure ready and also verified unit tests of most sections.

Thanks
*Sarath

[Sarath (nsarath)]

As we not yet heard for queries sent which also keep QA testplan incomplete with only ~85% document for review and so marking the Tracking sheet
Status Red for this tracking for Mesos/Contrail integration,
https://docs.google.com/spreadsheets/d/12UjwumTPmvceqaYVr2Ri9TaITKB9skX_ZRQquI62YyY/edit#gid=1827649949

Also, per R5.1 process highlighted, we need to have CI sanity running for qualifiying dev builds and opened below bug for tracking this,
https://docs.google.com/spreadsheets/d/12UjwumTPmvceqaYVr2Ri9TaITKB9skX_ZRQquI62YyY/edit#gid=1827649949

Thanks
*Sarath

From: Sarathbabu Narasimhan <email address hidden>
Date: Thursday, November 29, 2018 at 12:09 AM
To: Venkatraman Venkatapathy <email address hidden>, Aniket Gawade <email address hidden>, Aniket Daptari <email address hidden>
Cc: Sachchidanand Vaidya <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Mesos QA Testplan Initial-Draft

Hi,

Please find the detailed QA testplan for Mesos R5.1 based on blueprint doc and as few of clarificiations pending which is on chat-channel, so leaving those LB tests not included and this testplan has all other planned testcases detailed.

Please review and provide your comments.

Note: As per our Team process insisted by Jeba, we need to have “Unit test results” and “Integration results” for acceptance to QA. As talked to Venkat, he has most of provisioning procedure ready and also verified unit tests of most sections.

Thanks
*Sarath

[Sachchidanand Vaidya (vaidyasd)]

What is Red here ?

Thanks,
Sachin
From: Sarathbabu Narasimhan <email address hidden>
Date: Friday, November 30, 2018 at 9:28 AM
To: Venkatraman Venkatapathy <email address hidden>, Aniket Gawade <email address hidden>, Aniket Daptari <email address hidden>
Cc: Sachchidanand Vaidya <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Re: Mesos QA Testplan Initial-Draft

As we not yet heard for queries sent which also keep QA testplan incomplete with only ~85% document for review and so marking the Tracking sheet
Status Red for this tracking for Mesos/Contrail integration,
https://docs.google.com/spreadsheets/d/12UjwumTPmvceqaYVr2Ri9TaITKB9skX_ZRQquI62YyY/edit#gid=1827649949

Also, per R5.1 process highlighted, we need to have CI sanity running for qualifiying dev builds and opened below bug for tracking this,
https://docs.google.com/spreadsheets/d/12UjwumTPmvceqaYVr2Ri9TaITKB9skX_ZRQquI62YyY/edit#gid=1827649949

Thanks
*Sarath

From: Sarathbabu Narasimhan <email address hidden>
Date: Thursday, November 29, 2018 at 12:09 AM
To: Venkatraman Venkatapathy <email address hidden>, Aniket Gawade <email address hidden>, Aniket Daptari <email address hidden>
Cc: Sachchidanand Vaidya <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Mesos QA Testplan Initial-Draft

Hi,

Please find the detailed QA testplan for Mesos R5.1 based on blueprint doc and as few of clarificiations pending which is on chat-channel, so leaving those LB tests not included and this testplan has all other planned testcases detailed.

Please review and provide your comments.

Note: As per our Team process insisted by Jeba, we need to have “Unit test results” and “Integration results” for acceptance to QA. As talked to Venkat, he has most of provisioning procedure ready and also verified unit tests of most sections.

Thanks
*Sarath

[Sarath (nsarath)]

If you find the Mesos status of Dev complete, it is marked Red, as we need to have all clarifications completing QA testplan which is now pending with few testcases.

Thanks
*Sarath

From: Sachchidanand Vaidya <email address hidden>
Date: Friday, November 30, 2018 at 10:56 AM
To: Sarathbabu Narasimhan <email address hidden>, Venkatraman Venkatapathy <email address hidden>, Aniket Gawade <email address hidden>, Aniket Daptari <email address hidden>
Cc: Jeba Paulaiyan <email address hidden>
Subject: Re: Mesos QA Testplan Initial-Draft

What is Red here ?

Thanks,
Sachin
From: Sarathbabu Narasimhan <email address hidden>
Date: Friday, November 30, 2018 at 9:28 AM
To: Venkatraman Venkatapathy <email address hidden>, Aniket Gawade <email address hidden>, Aniket Daptari <email address hidden>
Cc: Sachchidanand Vaidya <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Re: Mesos QA Testplan Initial-Draft

As we not yet heard for queries sent which also keep QA testplan incomplete with only ~85% document for review and so marking the Tracking sheet
Status Red for this tracking for Mesos/Contrail integration,
https://docs.google.com/spreadsheets/d/12UjwumTPmvceqaYVr2Ri9TaITKB9skX_ZRQquI62YyY/edit#gid=1827649949

Also, per R5.1 process highlighted, we need to have CI sanity running for qualifiying dev builds and opened below bug for tracking this,
https://docs.google.com/spreadsheets/d/12UjwumTPmvceqaYVr2Ri9TaITKB9skX_ZRQquI62YyY/edit#gid=1827649949

Thanks
*Sarath

From: Sarathbabu Narasimhan <email address hidden>
Date: Thursday, November 29, 2018 at 12:09 AM
To: Venkatraman Venkatapathy <email address hidden>, Aniket Gawade <email address hidden>, Aniket Daptari <email address hidden>
Cc: Sachchidanand Vaidya <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Mesos QA Testplan Initial-Draft

Hi,

Please find the detailed QA testplan for Mesos R5.1 based on blueprint doc and as few of clarificiations pending which is on chat-channel, so leaving those LB tests not included and this testplan has all other planned testcases detailed.

Please review and provide your comments.

Note: As per our Team process insisted by Jeba, we need to have “Unit test results” and “Integration results” for acceptance to QA. As talked to Venkat, he has most of provisioning procedure ready and also verified unit tests of most sections.

Thanks
*Sarath

[Sarath (nsarath)]

Thanks for the inputs and below is the meeting-minutes,

## Discussed status of provisioning attempts ongoing for Mesos/Contrail bringup with Venkat given procedure on QA testbed.
## QA Testplan reviewed for comments
## All queries sent on chat-channel discussed as below of responses,

Q-1
currently messos load-balancing only we have and we don’t support contrail-load-balancing on R5.1
is my understanding based on Blueprints

Dev confirmed <No>

Q-2
and also as Sachin Vaidya mentioned whether dev checked with Mesos for API tool for automation
No defined structure. Use JSON to POST to API server.

QA has to find out the way for automation tool/client and Dev not automate Unit tests as of R5.1

Q-3
we need also unit-tests report and integration report which we insist for all the projects from QA insisted by Jeba

>> detailed steps/expected results of Unit tests and available integration report will be provided by Dev
    And also CI for Provisioning verify on dev-builds

Q-4
Ipv6 support

Dev confirmed <No>

Q-5
whether do we have “Service” concept and i see blueprint talks Service with refernce to Minuteman(DC/OS) and i heard there is no such from Venky and remember hearing same from Aniket during demo… If there is no Service how this Marathon internal LB and external LB scenarios work
 >>> There is no talk of service and how the App to client talk each other…

>> Dev will verify and give detailed comments.

Q-6
how the multi-tenancy supported and I don’t see any reference of namespace (or) project in the requirement blueprints

>> Dev confirmed it is network only isolation that is supported

Q-7
from team internal review, do we support Command UI provisioning (or) VN management for this Mesos/Contrail integration solution

Mid-december, there will be Command UI support only for contrail management and No-provisioning support.

######################

From: Sarathbabu Narasimhan <email address hidden>
Date: Monday, December 3, 2018 at 11:28 AM
To: Aniket Gawade <email address hidden>, Venkatraman Venkatapathy <email address hidden>, Aniket Daptari <email address hidden>, Sachchidanand Vaidya <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Clarifications on Mesos Blueprint/requirements for QA testplan