AttributeError: 'unicode' object has no attribute 'extend' during TLS barbican scenario testing

[Madhusudhan Kandadai]

When I create a listener with protocol=TERMINATED_HTTPS, protocol_port=443, default_tls_container_ref and sni_container_refs, the lb status never goes ONLINE and its in PENDING_UPDATE after 600 sec. In o-cw logs, I could see the detailed error as shown here: http://paste.openstack.org/show/lTLIMiEakT59NJEXnV4H/

Steps to reproduce:

1. Perform necessary steps to Create SSL Cert and Private Key and upload them into barbican
2. Create a loadbalancer and wait for 2-3 mins to become ACTIVE
3. Create listener with attributes: loadbalancer_id, protocol=TERMINATED_HTTPS, protocol_port=443, default_tls_container_ref, sni_container_refs
4. loadalancer status is in PENDING_UPDATE and it is not go back to ACTIVE state after 600 sec.

devstack@ubuntu:/opt/stack/neutron-lbaas/neutron_lbaas/tests/tempest/v2/scenario$ neutron lbaas-loadbalancer-list
+--------------------------------------+------+-------------+---------------------+----------+
| id | name | vip_address | provisioning_status | provider |
+--------------------------------------+------+-------------+---------------------+----------+
| de36a41f-6c52-41c4-a2eb-ae1c581648e3 | | 10.100.0.5 | PENDING_UPDATE | octavia |
+--------------------------------------+------+-------------+---------------------+----------+
devstack@ubuntu:/opt/stack/neutron-lbaas/neutron_lbaas/tests/tempest/v2/scenario$
devstack@ubuntu:/opt/stack/neutron-lbaas/neutron_lbaas/tests/tempest/v2/scenario$
devstack@ubuntu:/opt/stack/neutron-lbaas/neutron_lbaas/tests/tempest/v2/scenario$ neutron lbaas-listener-list
+--------------------------------------+--------------------------------------+------+------------------+---------------+----------------+
| id | default_pool_id | name | protocol | protocol_port | admin_state_up |
+--------------------------------------+--------------------------------------+------+------------------+---------------+----------------+
| 2bd06702-0fb0-4a15-9182-45efa86ab83c | 3a511758-3941-473d-9913-90f4a8b51cc7 | | TERMINATED_HTTPS | 443 | True |
+--------------------------------------+--------------------------------------+------+------------------+---------------+----------------+
devstack@ubuntu:/opt/stack/neutron-lbaas/neutron_lbaas/tests/tempest/v2/scenario$
devstack@ubuntu:/opt/stack/neutron-lbaas/neutron_lbaas/tests/tempest/v2/scenario$
devstack@ubuntu:/opt/stack/neutron-lbaas/neutron_lbaas/tests/tempest/v2/scenario$
devstack@ubuntu:/opt/stack/neutron-lbaas/neutron_lbaas/tests/tempest/v2/scenario$
devstack@ubuntu:/opt/stack/neutron-lbaas/neutron_lbaas/tests/tempest/v2/scenario$
devstack@ubuntu:/opt/stack/neutron-lbaas/neutron_lbaas/tests/tempest/v2/scenario$
devstack@ubuntu:/opt/stack/neutron-lbaas/neutron_lbaas/tests/tempest/v2/scenario$ neutron lbaas-listener-show 2bd06702-0fb0-4a15-9182-45efa86ab83c
+---------------------------+--------------------------------------------------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------------------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| default_pool_id | 3a511758-3941-473d-9913-90f4a8b51cc7 |
| default_tls_container_ref | http://192.168.109.129:9311/v1/containers/4def6268-2dd8-4764-8cfc-fcb95e7676a3 |
| description | |
| id | 2bd06702-0fb0-4a15-9182-45efa86ab83c |
| loadbalancers | {"id": "de36a41f-6c52-41c4-a2eb-ae1c581648e3"} |
| name | |
| protocol | TERMINATED_HTTPS |
| protocol_port | 443 |
| sni_container_refs | http://192.168.109.129:9311/v1/containers/9c8886c0-9f7f-44b3-883a-60c3d8669dca |
| | http://192.168.109.129:9311/v1/containers/e01dc0ad-1c40-43f6-a6b4-f15cd4d8ca49 |
| tenant_id | 8d7ecd10d71844139655b8414e86b712 |
+---------------------------+--------------------------------------------------------------------------------+

A copy of octavia.conf is for your reference:

http://paste.openstack.org/show/488435/

[Michael Johnson (johnsom)]

2016-02-26 13:10:18.109 20503 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last):
2016-02-26 13:10:18.109 20503 ERROR octavia.controller.worker.controller_worker File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
2016-02-26 13:10:18.109 20503 ERROR octavia.controller.worker.controller_worker result = task.execute(**arguments)
2016-02-26 13:10:18.109 20503 ERROR octavia.controller.worker.controller_worker File "/opt/stack/octavia/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 55, in execute
2016-02-26 13:10:18.109 20503 ERROR octavia.controller.worker.controller_worker self.amphora_driver.update(listener, loadbalancer.vip)
2016-02-26 13:10:18.109 20503 ERROR octavia.controller.worker.controller_worker File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 68, in update
2016-02-26 13:10:18.109 20503 ERROR octavia.controller.worker.controller_worker certs = self._process_tls_certificates(listener)
2016-02-26 13:10:18.109 20503 ERROR octavia.controller.worker.controller_worker File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 161, in _process_tls_certificates
2016-02-26 13:10:18.109 20503 ERROR octavia.controller.worker.controller_worker pem = cert_parser.build_pem(cert)
2016-02-26 13:10:18.109 20503 ERROR octavia.controller.worker.controller_worker File "/opt/stack/octavia/octavia/common/tls_utils/cert_parser.py", line 178, in build_pem
2016-02-26 13:10:18.109 20503 ERROR octavia.controller.worker.controller_worker pem.extend([tls_container.certificate, tls_container.private_key])
2016-02-26 13:10:18.109 20503 ERROR octavia.controller.worker.controller_worker AttributeError: 'unicode' object has no attribute 'extend'

[Adam Harwell (adam-harwell)]

This ... shouldn't be possible.

[Adam Harwell (adam-harwell)]

Ok, confirmed why and how this is happening, fix is up or soon to be up.

[Adam Harwell (adam-harwell)]

Nevermind, thought the bot wasn't picking it up, guess it's actually not merged yet (still in gate checks).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/285574
Committed: https://git.openstack.org/cgit/openstack/octavia/commit/?id=06f50c07efb14d199e26778a1546a963db18fdc1
Submitter: Jenkins
Branch: master

commit 06f50c07efb14d199e26778a1546a963db18fdc1
Author: Adam Harwell <email address hidden>
Date: Fri Feb 26 16:37:33 2016 -0600

    Barbican Cert Manager fails to correctly parse intermediates

    This may be a candidate for stable/liberty backport.

    Change-Id: Ia61bbc7099630fa33ca0f1ee4825c2b01244c7f3
    Closes-Bug: 1550535