SYMC: Allow custom configs with LBaaS

[Varun Lodaya]

LBaaS today spawns an HAProxy process with default HAProxy configs. Different applications require different set of configurations for optimal performance. We want to give our customers more power over controlling their HAProxy configs via Openstack APIs.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/12811
Submitter: Varun Lodaya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/12811
Committed: http://github.org/Juniper/contrail-controller/commit/8d756ab3dbc8226db7e0c68d72175ad17615a01e
Submitter: Zuul
Branch: master

commit 8d756ab3dbc8226db7e0c68d72175ad17615a01e
Author: Varun Lodaya <email address hidden>
Date: Mon Aug 3 09:44:41 2015 -0700

Allow custom configs with LBaaS

This fix enables a new field "custom-attr" in loadbalancer_pool
properties in the schema.

Change-Id: I17eecc2fedea4d1d3889b7e114e99732ac2eecc9
Closes-Bug: #1475393

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/12980
Submitter: Varun Lodaya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/12980
Committed: http://github.org/Juniper/contrail-neutron-plugin/commit/6588ebfea9e19325756f8ac16697c2563c1b434b
Submitter: Zuul
Branch: master

commit 6588ebfea9e19325756f8ac16697c2563c1b434b
Author: Varun Lodaya <email address hidden>
Date: Tue Aug 11 10:27:58 2015 -0700

Allow custom configs with LBaaS

Added key_value pair as a list of dict. Modified
the behavior of pool-update, only support bulk-update
for kvps. Incorporated Prakash's comments.

Closes Bug: #1475393
Change-Id: Ib7381b30eba3fffd03e9a11e57510b04653e4ac3

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13165
Submitter: Varun Lodaya (<email address hidden>)

[eon (eon-5)]

While making some benchs around lbaas I get tracebacks caused by this commit:

2015-08-19 13:49:04.060 ERROR neutron.api.v2.resource [req-62cafad8-e71c-45a2-ae60-3d7e7d22678c ctx_rally_ef53fb6cd05c49af955a3fd992ab7f25_user_0 ef53fb6cd05c49af955a3fd992ab7f25] create failed
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource Traceback (most recent call last):
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 87, in resource
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource result = method(request=request, **args)
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 448, in create
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource obj = obj_creator(request.context, **kwargs)
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource File "/opt/stack/contrail/openstack/neutron_plugin/neutron_plugin_contrail/plugins/opencontrail/loadbalancer/plugin.py", line 22, in create_pool
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource return super(LoadBalancerPlugin, self).create_pool(context, pool)
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource File "/opt/stack/contrail/openstack/neutron_plugin/neutron_plugin_contrail/plugins/opencontrail/loadbalancer/loadbalancer_db.py", line 121, in create_pool
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource return self._pool_manager.create(context, pool)
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource File "/opt/stack/contrail/openstack/neutron_plugin/neutron_plugin_contrail/plugins/opencontrail/loadbalancer/loadbalancer_pool.py", line 178, in create
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource self.create_update_custom_attributes(p['custom_attributes'], custom_attributes)
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource File "/opt/stack/contrail/openstack/neutron_plugin/neutron_plugin_contrail/plugins/opencontrail/loadbalancer/loadbalancer_pool.py", line 45, in create_update_custom_attributes
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource for custom_attribute in custom_attributes or []:
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource TypeError: 'object' object is not iterable
2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13181
Submitter: Varun Lodaya (<email address hidden>)

[Varun Lodaya (varun-lodaya)]

Found the regression, submitted a review with the fix
https://review.opencontrail.org/#/c/13181/

Thanks,
Varun

On 8/19/15, 6:50 AM, "eon" <email address hidden> wrote:

>While making some benchs around lbaas I get tracebacks caused by this
>commit:
>
>2015-08-19 13:49:04.060 ERROR neutron.api.v2.resource
>[req-62cafad8-e71c-45a2-ae60-3d7e7d22678c
>ctx_rally_ef53fb6cd05c49af955a3fd992ab7f25_user_0
>ef53fb6cd05c49af955a3fd992ab7f25] create failed
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource Traceback (most
>recent call last):
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource File
>"/opt/stack/neutron/neutron/api/v2/resource.py", line 87, in resource
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource result =
>method(request=request, **args)
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource File
>"/opt/stack/neutron/neutron/api/v2/base.py", line 448, in create
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource obj =
>obj_creator(request.context, **kwargs)
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource File
>"/opt/stack/contrail/openstack/neutron_plugin/neutron_plugin_contrail/plug
>ins/opencontrail/loadbalancer/plugin.py", line 22, in create_pool
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource return
>super(LoadBalancerPlugin, self).create_pool(context, pool)
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource File
>"/opt/stack/contrail/openstack/neutron_plugin/neutron_plugin_contrail/plug
>ins/opencontrail/loadbalancer/loadbalancer_db.py", line 121, in
>create_pool
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource return
>self._pool_manager.create(context, pool)
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource File
>"/opt/stack/contrail/openstack/neutron_plugin/neutron_plugin_contrail/plug
>ins/opencontrail/loadbalancer/loadbalancer_pool.py", line 178, in create
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource
>self.create_update_custom_attributes(p['custom_attributes'],
>custom_attributes)
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource File
>"/opt/stack/contrail/openstack/neutron_plugin/neutron_plugin_contrail/plug
>ins/opencontrail/loadbalancer/loadbalancer_pool.py", line 45, in
>create_update_custom_attributes
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource for
>custom_attribute in custom_attributes or []:
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource TypeError: 'object'
>object is not iterable
>2015-08-19 13:49:04.060 TRACE neutron.api.v2.resource
>
>--
>You received this bug notification because you are subscribed to the bug
>report.
>https://bugs.launchpad.net/bugs/1475393
>
>Title:
> Allow custom configs with LBaaS
>
>Status in Juniper Openstack:
> In Progress
>Status in Juniper Openstack trunk series:
> In Progress
>Status in OpenContrail:
> New
>
>Bug description:
> LBaaS today spawns an HAProxy process with default HAProxy configs.
> Different applications require different set of configurations for
> optimal performance. We want to give our customers more power over
> controlling their HAProxy configs via Openstack APIs.
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/ju...

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13165
Submitter: Varun Lodaya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13181
Committed: http://github.org/Juniper/contrail-neutron-plugin/commit/b2ffb8762ace193fbdb07abd95af2a173b7102e6
Submitter: Zuul
Branch: master

commit b2ffb8762ace193fbdb07abd95af2a173b7102e6
Author: Varun Lodaya <email address hidden>
Date: Wed Aug 19 08:45:41 2015 -0700

Allow custom configs with LBaaS

This fix checks if custom_attributes is
not specified or not in the create config.
If not specified, it skips

Closes Bug: #1475393

Change-Id: I55d524586fb906e801922af137309bfcdedf9a89

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13165
Submitter: Varun Lodaya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13165
Committed: http://github.org/Juniper/contrail-controller/commit/53cc6cd9e2abd87ce691b4c4f43be68cd208cbff
Submitter: Zuul
Branch: master

commit 53cc6cd9e2abd87ce691b4c4f43be68cd208cbff
Author: Varun Lodaya <email address hidden>
Date: Tue Aug 18 22:32:04 2015 -0700

Allow custom configs with LBaaS

This fix commits the vrouter agent code to read
the custom_attributes from ifmap node and copy it
to config.json file which the haproxy parser
would read. Added missing '}'. Incorporating the
comments

Closes-Bug: #1475393
Change-Id: I6f22f4f537c97c48b2283971b2959c9be5931361

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13554
Submitter: Varun Lodaya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13554
Committed: http://github.org/Juniper/contrail-controller/commit/cfbd9b801847275bb51a209a134f1b09ad628e05
Submitter: Zuul
Branch: master

commit cfbd9b801847275bb51a209a134f1b09ad628e05
Author: Varun Lodaya <email address hidden>
Date: Thu Sep 3 11:24:59 2015 -0700

Allow custom configs with LBaaS

This fix takes care of haproxy parsing and
validation changes on vrouter agent. Removing
extra white spaces

Closes-Bug: #1475393
Change-Id: I822e27792f78168a178d555db5703fa1e73d0cc9

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13722
Submitter: Varun Lodaya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13722
Committed: http://github.org/Juniper/contrail-controller/commit/c634286866fb695f350e0546af31bf88b5d9c51c
Submitter: Zuul
Branch: R2.20

commit c634286866fb695f350e0546af31bf88b5d9c51c
Author: Varun Lodaya <email address hidden>
Date: Mon Aug 3 09:44:41 2015 -0700

Allow custom configs with LBaaS

This fix enables a new field "custom-attr" in loadbalancer_pool
properties in the schema.

Change-Id: I17eecc2fedea4d1d3889b7e114e99732ac2eecc9
Closes-Bug: #1475393

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13850
Submitter: Varun Lodaya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13871
Submitter: Rudra Rugge (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13871
Committed: http://github.org/Juniper/contrail-controller/commit/245f6d85c7d28aedb5bb28f4b09b8c2a766b5c56
Submitter: Zuul
Branch: R2.22-dev

commit 245f6d85c7d28aedb5bb28f4b09b8c2a766b5c56
Author: Rudra Rugge <email address hidden>
Date: Thu May 14 13:41:45 2015 -0700

LBAAS haproxy process manager

Manage haproxy daemon for lbaas. Two options avaialable:
- Manage through supervisor. This will run on non-daemon mode
as the process cannot be managed by supervisord if it runs in
background. Process monitoring provided by supervisor.
- Start/stop the daemon as we do today. Need additional changes
to ensure monitoring/restarting of the process.

Additional commit needed to enable this code from vrouter_netns.

Change-Id: I05c13d7c96c86bee2fcddc73342ba28c6010c8e6
Partial-Bug: #1452928

Enable haproxy config translation

Enable haproxy config translation from json format
Also enable haproxy daemon handling by supervisord

Change-Id: If3489ea66430ec0ac50bb6198093a0689fa16219
Closes-Bug: #1452928

Conflicts:

 src/nodemgr/haproxy_stats.py

Generate mac from instance ip for service VMs

Generate the same mac-address for all interfaces sharing the same
IP. In addition a change to daemonize the haproxy process instead
of managing through supervisor.

Change-Id: I2394f29c4a11bffeee4b0184ce6cd6867b01e0e9
Closes-Bug: #1461882

Haproxy config generation fixes for HTTPS protocol

Change-Id: I140361ad4785be2a87d23a04181e73ca999e8e2b
Closes-bug: #1466318

Fix for poodle vulnerability; ChangeId: I9432d035eb59b1ff53cb5d33350cd5f8063e077c; Closes-Bug: #1475392

Change-Id: I390a77261bc0d3257108c06951c79f1d2c3dadaa

Fix for FREAK SSL vulnerability

This fix pushes selected set of secure ciphers into
haproxy config file

Change-Id: Idfc11ce0411024e7154d3b2c46a095fb4f80337d
Closes-Bug: #1477400

HAProxy Performance Tuning

HAProxy's default config is non-performant.
This fix updates following config in HAProxy:
1) Increase TCP client/server timeouts.
2) Increase ulimit globally per HAProxy process.
3) Increase maxconn globally per HAProxy process.

Change-Id: I28be29d5ab3dcb2a35fcbe9168300edf18b2c23c
Closes-Bug: #1477781

Allow custom configs with LBaaS

This fix takes care of haproxy parsing and
validation changes on vrouter agent. Removing
extra white spaces

Closes-Bug: #1475393
Change-Id: I822e27792f78168a178d555db5703fa1e73d0cc9

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13880
Submitter: Varun Lodaya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13881
Submitter: Varun Lodaya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13881
Committed: http://github.org/Juniper/contrail-neutron-plugin/commit/6e0c9860c3c10b46139be1ba598ecf1cb460f0c1
Submitter: Zuul
Branch: R2.22-dev

commit 6e0c9860c3c10b46139be1ba598ecf1cb460f0c1
Author: Varun Lodaya <email address hidden>
Date: Tue Aug 11 10:27:58 2015 -0700

Allow custom configs with LBaaS

Added key_value pair as a list of dict. Modified
the behavior of pool-update, only support bulk-update
for kvps. Incorporated Prakash's comments.

Closes Bug: #1475393
Change-Id: Ib7381b30eba3fffd03e9a11e57510b04653e4ac3

Allow custom configs with LBaaS

This fix checks if custom_attributes is
not specified or not in the create config.
If not specified, it skips

Closes Bug: #1475393
Change-Id: I55d524586fb906e801922af137309bfcdedf9a89

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13880
Committed: http://github.org/Juniper/contrail-controller/commit/1841342649a32fd945001474458dd43596721438
Submitter: Zuul
Branch: R2.22-dev

commit 1841342649a32fd945001474458dd43596721438
Author: Varun Lodaya <email address hidden>
Date: Mon Aug 3 09:44:41 2015 -0700

Allow custom configs with LBaaS

This fix enables a new field "custom-attr" in loadbalancer_pool
properties in the schema.

Change-Id: I17eecc2fedea4d1d3889b7e114e99732ac2eecc9
Closes-Bug: #1475393

Allow custom configs with LBaaS

This fix commits the vrouter agent code to read
the custom_attributes from ifmap node and copy it
to config.json file which the haproxy parser
would read. Added missing '}'. Incorporating the
comments

Closes-Bug: #1475393
Change-Id: I6f22f4f537c97c48b2283971b2959c9be5931361

Conflicts:
 src/vnsw/agent/oper/loadbalancer.cc
 src/vnsw/agent/oper/loadbalancer_config.cc
 src/vnsw/agent/oper/loadbalancer_config.h

Change-Id: Iea0aff5589a21e3c802e4e63633a1d74f22cdeaf

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14371
Submitter: Varun Lodaya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14576
Submitter: Varun Lodaya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14576
Committed: http://github.org/Juniper/contrail-controller/commit/888049f626fbd7d6ad349ffb2270bcc3886958f1
Submitter: Zuul
Branch: R2.20

commit 888049f626fbd7d6ad349ffb2270bcc3886958f1
Author: Rudra Rugge <email address hidden>
Date: Fri May 8 10:54:27 2015 -0700

Generate loadbalancer config in json format

Currently the agent generates loadbalancer configuration in
haproxy specific format. Going forward agent will generate
a generic json based loadbalancer config. This config will
be handled by driver specific configuration parser. Currently
only haproxy parsing is supported.

Closes-Bug: #1452928
Change-Id: I2d198aff0a569615ac5c331e4b6c582b93d9d3a3

Conflicts:
 src/vnsw/agent/oper/loadbalancer_haproxy.cc

LBAAS haproxy process manager

Manage haproxy daemon for lbaas. Two options avaialable:
- Manage through supervisor. This will run on non-daemon mode
as the process cannot be managed by supervisord if it runs in
background. Process monitoring provided by supervisor.
- Start/stop the daemon as we do today. Need additional changes
to ensure monitoring/restarting of the process.

Additional commit needed to enable this code from vrouter_netns.

Change-Id: I05c13d7c96c86bee2fcddc73342ba28c6010c8e6
Partial-Bug: #1452928

Enable haproxy config translation

Enable haproxy config translation from json format
Also enable haproxy daemon handling by supervisord

Change-Id: If3489ea66430ec0ac50bb6198093a0689fa16219
Closes-Bug: #1452928

Conflicts:

 src/nodemgr/haproxy_stats.py

Generate mac from instance ip for service VMs

Generate the same mac-address for all interfaces sharing the same
IP. In addition a change to daemonize the haproxy process instead
of managing through supervisor.

Change-Id: I2394f29c4a11bffeee4b0184ce6cd6867b01e0e9
Closes-Bug: #1461882

Haproxy config generation fixes for HTTPS protocol

Change-Id: I140361ad4785be2a87d23a04181e73ca999e8e2b
Closes-bug: #1466318

Fix for poodle vulnerability; ChangeId: I9432d035eb59b1ff53cb5d33350cd5f8063e077c; Closes-Bug: #1475392

Change-Id: I390a77261bc0d3257108c06951c79f1d2c3dadaa

Fix for FREAK SSL vulnerability

This fix pushes selected set of secure ciphers into
haproxy config file

Change-Id: Idfc11ce0411024e7154d3b2c46a095fb4f80337d
Closes-Bug: #1477400

HAProxy Performance Tuning

HAProxy's default config is non-performant.
This fix updates following config in HAProxy:
1) Increase TCP client/server timeouts.
2) Increase ulimit globally per HAProxy process.
3) Increase maxconn globally per HAProxy process.

Change-Id: I28be29d5ab3dcb2a35fcbe9168300edf18b2c23c
Closes-Bug: #1477781

Allow custom configs with LBaaS

This fix takes care of haproxy parsing and
validation changes on vrouter agent. Removing
extra white spaces

Closes-Bug: #1475393
Change-Id: I822e27792f78168a178d555db5703fa1e73d0cc9

Allow custom configs with LBaaS

This fix enables a new field "custom-attr" in loadbalancer_pool
properties in the schema.

Change-Id: I17eecc2fedea4d1d3889b7e114e99732ac2eecc9
Closes-Bug: #1475393

Allow custom configs with LBaaS

This fix commits the vrouter agent code to read
the custom_attributes from ifmap node and copy it
to config.json file...

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14919
Submitter: Varun Lodaya (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14919
Committed: http://github.org/Juniper/contrail-neutron-plugin/commit/3b9d389cbbffb91b10826e63cabbcab27faa710a
Submitter: Zuul
Branch: R2.20

commit 3b9d389cbbffb91b10826e63cabbcab27faa710a
Author: Varun Lodaya <email address hidden>
Date: Tue Aug 11 10:27:58 2015 -0700

Allow custom configs with LBaaS

Added key_value pair as a list of dict. Modified
the behavior of pool-update, only support bulk-update
for kvps. Incorporated Prakash's comments.

Closes Bug: #1475393
Change-Id: Ib7381b30eba3fffd03e9a11e57510b04653e4ac3

Allow custom configs with LBaaS

This fix checks if custom_attributes is
not specified or not in the create config.
If not specified, it skips

Closes Bug: #1475393

Change-Id: I55d524586fb906e801922af137309bfcdedf9a89