octavia

loadbalancer status is stuck in PENDING_UPDATE for about 50-55 mins after creating listener with TLS no SNI

Bug #1555316 reported by Madhusudhan Kandadai on 2016-03-09

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	octavia	Invalid	Critical	Unassigned

Bug Description

1. Create necessary barbican secrets - private key, certificate (with or without Intermediates)
2. Create barbican container by referencing the above secrets
3. Create a loadbalancer
4. Create a listener with TLS no SNI.

neutron lbaas-listener-create --loadbalancer lb1 --protocol-port 443 --protocol TERMINATED_HTTPS --name listener1 --default-tls-container=<barbican container href>

Watch out the errors in o-cw logs:

2016-03-08 17:12:26.010 33399 INFO octavia.network.drivers.neutron.allowed_address_pairs [-] Port 32af3668-59db-40d4-ae7e-fdfaf580fc82 already exists. Nothing to be done.
2016-03-08 17:13:11.495 33399 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ACTIVE in DB for load balancer id: f64fb84d-8722-4b41-ac2a-097454a7ecc2
2016-03-08 17:13:23.913 33399 INFO octavia.controller.queue.endpoint [-] Creating listener '40248d06-d848-438a-9b63-fa38ca36962d'...
2016-03-08 17:13:24.451 33399 INFO octavia.certificates.manager.barbican [-] Loading certificate container http://192.168.109.129:9311/v1/containers/ac38454b-7755-492c-ad1a-b81b279e87a0 from Barbican.
2016-03-08 17:13:25.493 33399 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.ListenersUpdate' (5b799207-e5eb-49ac-a028-9ab908b8859d) transitioned into state 'FAILURE' from state 'RUNNING'
1 predecessors (most recent first):
Flow 'octavia-create-listener_flow'
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last):
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker result = task.execute(**arguments)
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker File "/opt/stack/octavia/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 56, in execute
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker self.amphora_driver.update(listener, loadbalancer.vip)
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 75, in update
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker self.client.upload_config(amp, listener.id, config)
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 257, in upload_config
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker return exc.check_exception(r)
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/exceptions.py", line 32, in check_exception
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker raise responses[status_code]()
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker InvalidRequest: Invalid request
2016-03-08 17:13:25.493 33399 ERROR octavia.controller.worker.controller_worker
2016-03-08 17:13:25.504 33399 WARNING octavia.controller.worker.tasks.amphora_driver_tasks [-] Reverting listeners updates.
2016-03-08 17:13:25.521 33399 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.ListenersUpdate' (5b799207-e5eb-49ac-a028-9ab908b8859d) transitioned into state 'REVERTED' from state 'REVERTING'
2016-03-08 17:13:25.526 33399 WARNING octavia.controller.worker.controller_worker [-] Flow 'octavia-create-listener_flow' (bf9bd083-f872-42db-a1e0-5a5819e9fb86) transitioned into state 'REVERTED' from state 'RUNNING'
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher [-] Exception during message handling: Invalid request
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/controller/queue/endpoint.py", line 59, in create_listener
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher self.worker.create_listener(listener_id)
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/controller/worker/controller_worker.py", line 203, in create_listener
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher create_listener_tf.run()
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 230, in run
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher for _state in self.run_iter(timeout=timeout):
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 308, in run_iter
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher failure.Failure.reraise_if_any(fails)
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/taskflow/types/failure.py", line 336, in reraise_if_any
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher failures[0].reraise()
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/taskflow/types/failure.py", line 343, in reraise
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher six.reraise(*self._exc_info)
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher result = task.execute(**arguments)
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 56, in execute
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher self.amphora_driver.update(listener, loadbalancer.vip)
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 75, in update
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher self.client.upload_config(amp, listener.id, config)
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 257, in upload_config
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher return exc.check_exception(r)
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/octavia/octavia/amphorae/drivers/haproxy/exceptions.py", line 32, in check_exception
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher raise responses[status_code]()
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher InvalidRequest: Invalid request
2016-03-08 17:13:25.532 33399 ERROR oslo_messaging.rpc.dispatcher

In q-svc logs:

2016-03-08 17:13:15.008 ^[[00;32mDEBUG neutron_lbaas.drivers.driver_mixins [^[[01;36mreq-e5116426-0c70-488e-8923-a143e7263efe ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mStarting successful_completion method after a successful driver action.^[[00m ^[[00;33mfrom (pid=28278) successful_completion /opt/stack/neutron-lbaas/neutron_lbaas/drivers/driver_mixins.py:65^[[00m
2016-03-08 17:13:15.040 ^[[00;32mDEBUG neutron_lbaas.drivers.driver_mixins [^[[01;36mreq-e5116426-0c70-488e-8923-a143e7263efe ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mUpdating object of type <class 'neutron_lbaas.services.loadbalancer.data_models.LoadBalancer'> with id of f64fb84d-8722-4b41-ac2a-097454a7ecc2 to provisioning_status = ACTIVE, operating_status = ONLINE^[[00m ^[[00;33mfrom (pid=28278) successful_completion /opt/stack/neutron-lbaas/neutron_lbaas/drivers/driver_mixins.py:117^[[00m
2016-03-08 17:13:15.579 ^[[00;36mINFO neutron.wsgi [^[[01;36mreq-c7296cd1-2381-42ab-9779-a37fea65a5a7 ^[[00;36mnova 3addd5788cba430386dc2692c60eed4f^[[00;36m] ^[[01;35m^[[00;36m192.168.109.129 - - [08/Mar/2016 17:13:15] "GET /v2.0/ports.json?network_id=7a22396c-805e-46ba-99d6-d610f8e507ba&device_owner=network%3Adhcp HTTP/1.1" 200 1091 0.690414^[[00m
2016-03-08 17:13:15.927 ^[[00;32mDEBUG neutron_lbaas.drivers.octavia.driver [^[[01;36mreq-f5a3028e-c91e-4557-90a8-08c9ca5c56f4 ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32murl = http://127.0.0.1:9876/v1/loadbalancers/d4915905-a22a-4d32-beab-3778a77ac734^[[00m ^[[00;33mfrom (pid=28279) request /opt/stack/neutron-lbaas/neutron_lbaas/drivers/octavia/driver.py:137^[[00m
2016-03-08 17:13:15.941 ^[[00;32mDEBUG neutron_lbaas.drivers.octavia.driver [^[[01;36mreq-f5a3028e-c91e-4557-90a8-08c9ca5c56f4 ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32margs = None^[[00m ^[[00;33mfrom (pid=28279) request /opt/stack/neutron-lbaas/neutron_lbaas/drivers/octavia/driver.py:138^[[00m
2016-03-08 17:13:16.041 ^[[00;32mDEBUG neutron_lbaas.drivers.octavia.driver [^[[01;36mreq-f5a3028e-c91e-4557-90a8-08c9ca5c56f4 ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mOctavia Response Code: 404^[[00m ^[[00;33mfrom (pid=28279) request /opt/stack/neutron-lbaas/neutron_lbaas/drivers/octavia/driver.py:143^[[00m
2016-03-08 17:13:16.047 ^[[00;32mDEBUG neutron_lbaas.drivers.octavia.driver [^[[01;36mreq-f5a3028e-c91e-4557-90a8-08c9ca5c56f4 ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mOctavia Response Body: {"debuginfo": null, "faultcode": "Client", "faultstring": "Load Balancer d4915905-a22a-4d32-beab-3778a77ac734 not found."}^[[00m ^[[00;33mfrom (pid=28279) request /opt/stack/neutron-lbaas/neutron_lbaas/drivers/octavia/driver.py:144^[[00m
2016-03-08 17:13:16.048 ^[[00;32mDEBUG neutron_lbaas.drivers.octavia.driver [^[[01;36mreq-f5a3028e-c91e-4557-90a8-08c9ca5c56f4 ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mOctavia Response Headers: {'Date': 'Wed, 09 Mar 2016 01:13:15 GMT', 'Content-Length': '122', 'Content-Type': 'application/json; charset=UTF-8', 'Server': 'WSGIServer/0.1 Python/2.7.6'}^[[00m ^[[00;33mfrom (pid=28279) request /opt/stack/neutron-lbaas/neutron_lbaas/drivers/octavia/driver.py:145^[[00m
2016-03-08 17:13:16.067 ^[[00;32mDEBUG neutron_lbaas.drivers.octavia.driver [^[[01;36mreq-f5a3028e-c91e-4557-90a8-08c9ca5c56f4 ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mOctavia reports load balancer d4915905-a22a-4d32-beab-3778a77ac734 has provisioning status of None^[[00m ^[[00;33mfrom (pid=28279) thread_op /opt/stack/neutron-lbaas/neutron_lbaas/drivers/octavia/driver.py:75^[[00m
2016-03-08 17:13:16.084 ^[[00;36mINFO neutron.wsgi [^[[01;36mreq-9e9340e5-ac5e-4bf6-9fb1-6d90dbeee653 ^[[00;36mTestLoadBalancerTLS-701805341 aceddd6e1b404eb488bfc958386a468f^[[00;36m] ^[[01;35m^[[00;36m192.168.109.129 - - [08/Mar/2016 17:13:16] "GET /v2.0/lbaas/loadbalancers/f64fb84d-8722-4b41-ac2a-097454a7ecc2 HTTP/1.1" 200 643 3.078938^[[00m
2016-03-08 17:13:17.094 ^[[00;32mDEBUG neutron.wsgi [^[[00;36m-^[[00;32m] ^[[01;35m^[[00;32m(28278) accepted ('192.168.109.129', 37521)^[[00m ^[[00;33mfrom (pid=28278) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:867^[[00m
2016-03-08 17:13:18.880 ^[[00;36mINFO neutron.wsgi [^[[01;36mreq-232bbf0e-465a-4e77-b6ab-10bf46d2062c ^[[00;36mTestLoadBalancerTLS-701805341 aceddd6e1b404eb488bfc958386a468f^[[00;36m] ^[[01;35m^[[00;36m192.168.109.129 - - [08/Mar/2016 17:13:18] "GET /v2.0/lbaas/loadbalancers/f64fb84d-8722-4b41-ac2a-097454a7ecc2 HTTP/1.1" 200 634 1.776131^[[00m
2016-03-08 17:13:18.915 ^[[00;32mDEBUG neutron.wsgi [^[[00;36m-^[[00;32m] ^[[01;35m^[[00;32m(28279) accepted ('192.168.109.129', 37561)^[[00m ^[[00;33mfrom (pid=28279) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:867^[[00m
2016-03-08 17:13:19.068 ^[[00;32mDEBUG neutron_lbaas.drivers.octavia.driver [^[[01;36mreq-f5a3028e-c91e-4557-90a8-08c9ca5c56f4 ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32murl = http://127.0.0.1:9876/v1/loadbalancers/d4915905-a22a-4d32-beab-3778a77ac734^[[00m ^[[00;33mfrom (pid=28279) request /opt/stack/neutron-lbaas/neutron_lbaas/drivers/octavia/driver.py:137^[[00m
2016-03-08 17:13:19.069 ^[[00;32mDEBUG neutron_lbaas.drivers.octavia.driver [^[[01;36mreq-f5a3028e-c91e-4557-90a8-08c9ca5c56f4 ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32margs = None^[[00m ^[[00;33mfrom (pid=28279) request /opt/stack/neutron-lbaas/neutron_lbaas/drivers/octavia/driver.py:138^[[00m
2016-03-08 17:13:19.218 ^[[00;32mDEBUG neutron_lbaas.drivers.octavia.driver [^[[01;36mreq-f5a3028e-c91e-4557-90a8-08c9ca5c56f4 ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mOctavia Response Code: 404^[[00m ^[[00;33mfrom (pid=28279) request /opt/stack/neutron-lbaas/neutron_lbaas/drivers/octavia/driver.py:143^[[00m
2016-03-08 17:13:19.231 ^[[00;32mDEBUG neutron_lbaas.drivers.octavia.driver [^[[01;36mreq-f5a3028e-c91e-4557-90a8-08c9ca5c56f4 ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mOctavia Response Body: {"debuginfo": null, "faultcode": "Client", "faultstring": "Load Balancer d4915905-a22a-4d32-beab-3778a77ac734 not found."}^[[00m ^[[00;33mfrom (pid=28279) request /opt/stack/neutron-lbaas/neutron_lbaas/drivers/octavia/driver.py:144^[[00m
2016-03-08 17:13:19.243 ^[[00;32mDEBUG neutron_lbaas.drivers.octavia.driver [^[[01;36mreq-f5a3028e-c91e-4557-90a8-08c9ca5c56f4 ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mOctavia Response Headers: {'Date': 'Wed, 09 Mar 2016 01:13:19 GMT', 'Content-Length': '122', 'Content-Type': 'application/json; charset=UTF-8', 'Server': 'WSGIServer/0.1 Python/2.7.6'}^[[00m ^[[00;33mfrom (pid=28279) request /opt/stack/neutron-lbaas/neutron_lbaas/drivers/octavia/driver.py:145^[[00m
2016-03-08 17:13:19.249 ^[[00;32mDEBUG neutron_lbaas.drivers.octavia.driver [^[[01;36mreq-f5a3028e-c91e-4557-90a8-08c9ca5c56f4 ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mOctavia reports load balancer d4915905-a22a-4d32-beab-3778a77ac734 has provisioning status of None^[[00m ^[[00;33mfrom (pid=28279) thread_op /opt/stack/neutron-lbaas/neutron_lbaas/drivers/octavia/driver.py:75^[[00m
2016-03-08 17:13:19.511 ^[[00;32mDEBUG neutron.api.v2.base [^[[01;36mreq-8c08c73d-cbaf-4e42-8665-007a03bcbafd ^[[00;36mTestLoadBalancerTLS-701805341 aceddd6e1b404eb488bfc958386a468f^[[00;32m] ^[[01;35m^[[00;32mRequest body: {u'listener': {u'protocol_port': 443, u'protocol': u'TERMINATED_HTTPS', u'loadbalancer_id': u'f64fb84d-8722-4b41-ac2a-097454a7ecc2', u'default_tls_container_ref': u'http://192.168.109.129:9311/v1/containers/ac38454b-7755-492c-ad1a-b81b279e87a0'}}^[[00m ^[[00;33mfrom (pid=28279) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:656^[[00m
2016-03-08 17:13:19.519 ^[[00;32mDEBUG neutron.db.quota.driver [^[[01;36mreq-8c08c73d-cbaf-4e42-8665-007a03bcbafd ^[[00;36mTestLoadBalancerTLS-701805341 aceddd6e1b404eb488bfc958386a468f^[[00;32m] ^[[01;35m^[[00;32mResources listener,l7policy,subnetpool,healthmonitor have unlimited quota limit. It is not required to calculated headroom ^[[00m ^[[00;33mfrom (pid=28279) make_reservation /opt/stack/neutron/neutron/db/quota/driver.py:170^[[00m
2016-03-08 17:13:19.573 ^[[00;36mINFO neutron_lbaas.common.cert_manager.barbican_cert_manager [^[[01;36mreq-8c08c73d-cbaf-4e42-8665-007a03bcbafd ^[[00;36mTestLoadBalancerTLS-701805341 aceddd6e1b404eb488bfc958386a468f^[[00;36m] ^[[01;35m^[[00;36mLoading certificate container http://192.168.109.129:9311/v1/containers/ac38454b-7755-492c-ad1a-b81b279e87a0 from Barbican.^[[00m
2016-03-08 17:13:19.585 ^[[00;32mDEBUG barbicanclient.containers [^[[01;36mreq-8c08c73d-cbaf-4e42-8665-007a03bcbafd ^[[00;36mTestLoadBalancerTLS-701805341 aceddd6e1b404eb488bfc958386a468f^[[00;32m] ^[[01;35m^[[00;32mCreating consumer registration for container http://192.168.109.129:9311/v1/containers/ac38454b-7755-492c-ad1a-b81b279e87a0 as lbaas: lbaas://RegionOne/loadbalancer/f64fb84d-8722-4b41-ac2a-097454a7ecc2^[[00m ^[[00;33mfrom (pid=28279) register_consumer /opt/stack/python-barbicanclient/barbicanclient/containers.py:745^[[00m
2016-03-08 17:13:20.002 ^[[00;32mDEBUG oslo_messaging._drivers.amqpdriver [^[[00;36m-^[[00;32m] ^[[01;35m^[[00;32mreceived message msg_id: a58b04c83b63486fa08f796b8ad86369 reply to reply_c861469e4c20471bb68c40794d45dd6
...
...
2016-03-08 18:03:24.448 ^[[00;32mDEBUG neutron_lbaas.drivers.driver_mixins [^[[01;36mreq-52b63e93-b2b3-48e1-873c-15a9cdf68edf ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mStarting failed_completion method after a failed driver action.^[[00m ^[[00;33mfrom (pid=28279) failed_completion /opt/stack/neutron-lbaas/neutron_lbaas/drivers/driver_mixins.py:142^[[00m
2016-03-08 18:03:24.449 ^[[00;32mDEBUG neutron_lbaas.drivers.driver_mixins [^[[01;36mreq-52b63e93-b2b3-48e1-873c-15a9cdf68edf ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mUpdating object of type <class 'neutron_lbaas.services.loadbalancer.data_models.Listener'> with id of 40248d06-d848-438a-9b63-fa38ca36962d to provisioning_status = ERROR, operating_status = OFFLINE^[[00m ^[[00;33mfrom (pid=28279) failed_completion /opt/stack/neutron-lbaas/neutron_lbaas/drivers/driver_mixins.py:158^[[00m
2016-03-08 18:03:24.576 ^[[00;32mDEBUG neutron_lbaas.drivers.driver_mixins [^[[01;36mreq-52b63e93-b2b3-48e1-873c-15a9cdf68edf ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mUpdating load balancer f64fb84d-8722-4b41-ac2a-097454a7ecc2 to provisioning_status = ACTIVE^[[00m ^[[00;33mfrom (pid=28279) failed_completion /opt/stack/neutron-lbaas/neutron_lbaas/drivers/driver_mixins.py:165^[[00m
2016-03-08 18:03:26.759 ^[[00;32mDEBUG oslo_messaging._drivers.amqpdriver [^[[00;36m-^[[00;32m] ^[[01;35m^[[00;32mreceived message msg_id: 1b43560b2243436abb1c80eb2363e1f2 reply to reply_f101da61675942d3bfba2d254a88380a^[[00m ^[[00;33mfrom (pid=28280) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:201^[[00m

devstack@ubuntu:~/devstack$ neutron lbaas-loadbalancer-list
+--------------------------------------+------+-------------+---------------------+----------+
| id | name | vip_address | provisioning_status | provider |
+--------------------------------------+------+-------------+---------------------+----------+
| f64fb84d-8722-4b41-ac2a-097454a7ecc2 | | 10.100.0.5 | ACTIVE | octavia |
+--------------------------------------+------+-------------+---------------------+----------+
devstack@ubuntu:~/devstack$
devstack@ubuntu:~/devstack$
devstack@ubuntu:~/devstack$
devstack@ubuntu:~/devstack$ neutron lbaas-listener-list
+--------------------------------------+-----------------+------+------------------+---------------+----------------+
| id | default_pool_id | name | protocol | protocol_port | admin_state_up |
+--------------------------------------+-----------------+------+------------------+---------------+----------------+
| 40248d06-d848-438a-9b63-fa38ca36962d | | | TERMINATED_HTTPS | 443 | True |
+--------------------------------------+-----------------+------+------------------+---------------+----------------+
devstack@ubuntu:~/devstack$ neutron lbaas-pool-list

devstack@ubuntu:~/devstack$
devstack@ubuntu:~/devstack$
devstack@ubuntu:~/devstack$ . openrc admin ADMIN
WARNING: setting legacy OS_TENANT_NAME to support cli tools.
devstack@ubuntu:~/devstack$ . openrc admin admin
WARNING: setting legacy OS_TENANT_NAME to support cli tools.
devstack@ubuntu:~/devstack$ barbican secret list
barbicStarting new HTTP connection (1): 192.168.109.129
aStarting new HTTP connection (1): 192.168.109.129
Resetting dropped connection: 192.168.109.129
n +-----------------------------------------------------------------------------+---------------+---------------------------+--------+-----------------------------+-----------+------------+-------------+------+------------+
| Secret href | Name | Created | Status | Content types | Algorithm | Bit length | Secret type | Mode | Expiration |
+-----------------------------------------------------------------------------+---------------+---------------------------+--------+-----------------------------+-----------+------------+-------------+------+------------+
| http://192.168.109.129:9311/v1/secrets/a5812e02-f220-4652-baae-0e5e981f46fe | passphrase | 2016-03-09 01:11:59+00:00 | ACTIVE | {u'default': u'text/plain'} | aes | 256 | opaque | cbc | None |
| http://192.168.109.129:9311/v1/secrets/d8053d45-e06e-4251-965d-0b56af275a08 | key | 2016-03-09 01:11:59+00:00 | ACTIVE | {u'default': u'text/plain'} | aes | 256 | opaque | cbc | None |
| http://192.168.109.129:9311/v1/secrets/d97a43d7-01c6-4bf6-a4e8-dee212316c52 | cert | 2016-03-09 01:11:59+00:00 | ACTIVE | {u'default': u'text/plain'} | aes | 256 | opaque | cbc | None |
| http://192.168.109.129:9311/v1/secrets/df8c63b2-b625-4521-9460-d2ee2fb845d3 | intermediates | 2016-03-09 01:11:59+00:00 | ACTIVE | {u'default': u'text/plain'} | None | None | opaque | None | None |
+-----------------------------------------------------------------------------+---------------+---------------------------+--------+-----------------------------+-----------+------------+-------------+------+------------+
sdevstack@ubuntu:~/devstack$ barbican secret container list
Starting new HTTP connection (1): 192.168.109.129
Starting new HTTP connection (1): 192.168.109.129
Resetting dropped connection: 192.168.109.129
+-----------------------------------------------------------+------------------+---------------------------+--------+-------------+-----------------------------------------------------------+-------------------------------------------------------------+
| Container href | Name | Created | Status | Type | Secrets | Consumers |
+-----------------------------------------------------------+------------------+---------------------------+--------+-------------+-----------------------------------------------------------+-------------------------------------------------------------+
| http://192.168.109.129:9311/v1/containers/ac38454b-7755 | tls-lb-container | 2016-03-09 01:11:59+00:00 | ACTIVE | certificate | private_key=http://192.168.109.129:9311/v1/secrets/d8053d | {u'URL': |
| -492c-ad1a-b81b279e87a0 | | | | | 45-e06e-4251-965d-0b56af275a08 | u'lbaas://RegionOne/loadbalancer/f64fb84d-8722-4b41-ac2a- |
| | | | | | intermediates=http://192.168.109.129:9311/v1/secrets/df8c | 097454a7ecc2', u'name': u'lbaas'} |
| | | | | | 63b2-b625-4521-9460-d2ee2fb845d3 | |
| | | | | | private_key_passphrase=http://192.168.109.129:9311/v1/sec | |
| | | | | | rets/a5812e02-f220-4652-baae-0e5e981f46fe | |
| | | | | | certificate=http://192.168.109.129:9311/v1/secrets/d97a43 | |
| | | | | | d7-01c6-4bf6-a4e8-dee212316c52 | |
+-----------------------------------------------------------+------------------+---------------------------+--------+-------------+-----------------------------------------------------------+-------------------------------------------------------------+
devstack@ubuntu:~/devstack$

Tags:

Revision history for this message

Michael Johnson (johnsom) wrote on 2016-03-09:

Please collect the amphora-agent.log from the amphora.

Changed in octavia:
status:	New → Incomplete
importance:	Undecided → Critical
tags:	added: target-mitaka

Revision history for this message

Michael Johnson (johnsom) wrote on 2016-03-10:

This appears to have been an issue with Nova on the devstack host. There were 4-5 instances running, but nova was not reporting any instances. We are unable to reproduce after a restack.

Changed in octavia:
status:	Incomplete → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.