Cannot update session persistence for pool

[Elena Ezhova]

Scenario test for session persistence [1] fails when it tries to update pool with new session persistence values.
Neutron-server logs show that the pool remained in PENDING_UPDATE status:

2015-10-09 10:30:40.761 ERROR neutron.api.v2.resource [req-2ce65314-e91c-4b03-82a5-c7d5cee1cd04 TestSessionPersistence-932377504 TestSessionPersistence-1760116977] delete failed
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource Traceback (most recent call last):
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource result = method(request=request, **args)
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 146, in wrapper
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 136, in wrapper
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 546, in delete
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource obj_deleter(request.context, id, **kwargs)
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron-lbaas/neutron_lbaas/services/loadbalancer/plugin.py", line 786, in delete_pool
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource constants.PENDING_DELETE)
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron-lbaas/neutron_lbaas/db/loadbalancer/loadbalancer_dbv2.py", line 166, in test_and_set_status
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource self.assert_modification_allowed(db_lb)
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron-lbaas/neutron_lbaas/db/loadbalancer/loadbalancer_dbv2.py", line 153, in assert_modification_allowed
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource raise loadbalancerv2.StateInvalid(id=id, state=status)
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource StateInvalid: Invalid state PENDING_UPDATE of loadbalancer resource 4149f63f-4c17-4eab-a14c-b7b159e13275
2015-10-09 10:30:40.761 28826 ERROR neutron.api.v2.resource

And in o-cw logs it can be seen that while building a config for jinja pool.session_persistence contained a dict instead of a SessionPersistence data model. [3]. A bit of research showed that when pool is updated [4] the order of the tasks in the flow is so that we firstly update the pool data model object with a dictionary with updates and now listener.default_pool.session_persistence points to a dictionary. [5] Then, when we try to update listener, haproxy amphorae driver attempts to build config and fails. [6][7]

So perhaps either pool_updates should be transformed to data models or the flow itself should be updated.

[1] https://review.openstack.org/#/c/207945/
[2] http://logs.openstack.org/45/207945/7/check/gate-neutron-lbaasv2-dsvm-scenario/2d38889/logs/screen-q-svc.txt.gz?#_2015-10-09_10_30_40_761
[3] http://logs.openstack.org/45/207945/7/check/gate-neutron-lbaasv2-dsvm-scenario/2d38889/logs/screen-o-cw.txt.gz#_2015-10-09_10_20_39_662
[4] https://github.com/openstack/octavia/blob/master/octavia/controller/worker/controller_worker.py#L457
[5] https://github.com/openstack/octavia/blob/master/octavia/controller/worker/flows/pool_flows.py#L56
[6] https://github.com/openstack/octavia/blob/master/octavia/amphorae/drivers/haproxy/rest_api_driver.py#L67
[7] https://github.com/openstack/octavia/blob/master/octavia/amphorae/drivers/haproxy/jinja/jinja_cfg.py#L208

[Elena Ezhova (eezhova)]

It is a blocker for scenario test for session persistence: https://review.openstack.org/#/c/207945/

[Elena Ezhova (eezhova)]

A WIP patch: https://review.openstack.org/#/c/267010/

[Elena Ezhova (eezhova)]

The error that is thrown by o-cw has changed a bit with time [1] - now it is triggered by an attempt to incorrectly update a pool in db.

[1] http://logs.openstack.org/45/207945/8/check/gate-neutron-lbaasv2-dsvm-scenario/4a7ef0b/logs/screen-o-cw.txt.gz#_2016-01-12_23_54_36_038

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267010
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=caecc4f9986891d9954b43e17d207cc34e514e80
Submitter: Jenkins
Branch: master

commit caecc4f9986891d9954b43e17d207cc34e514e80
Author: Elena Ezhova <email address hidden>
Date: Wed Jan 13 19:20:46 2016 +0300

    Fix updating session persistence of a pool in DB

    In case when update data for a pool contains session persistence
    it is required to update session persistence table first and
    then remove 'session_persistence' key from the update dict.
    Otherwise, update will fail and a pool will get stuck in
    PENDING_UPDATE status.

    This change reuses update_pool_on_listener method of
    db.repositories.Repositories class in UpdatePoolInDB task instead
    of PoolRepository update method.

    This regression was discovered using a scenario test:
    https://review.openstack.org/#/c/207945/

    Change-Id: I53d6b7962c0baa466db7e053157cdd302edae473
    Closes-Bug: #1505125