octavia

Enabling lbaas through Octavia in Devstack Fails

Bug #1506027 reported by praburm on 2015-10-14

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	octavia	Expired	High	Unassigned

Bug Description

Hi,

I have Openstack setup where controller, compute and network(neutron) running in single physical machine through devstack(git branch: master). Enabling lbaas through Octavia plugin in KILO version of devstack keeps provision status as "PENDING_CREATE" on lb creation for long time. Could find the below error message from /opt/stack/logs/q-lbaasv2.log

ERROR neutron_lbaas.agent.agent_manager [-] Unable to retrieve ready devices

Lbaas Options added in [local.conf] of devstack setup

Followed the below link and did the below config

https://wiki.openstack.org/wiki/Neutron/LBaaS/HowToRun

I would like to confirm whether the way I enabled lbaas in devstack is correct. Please throw some light on this. I have added the logs neutron & lbaas config too here.

[NEUTRON CONFIG]
Q_USE_SECGROUP=True
FLOATING_RANGE=172.30.5.0/24
FIXED_RANGE=10.0.0.0/24
Q_FLOATING_ALLOCATION_POOL=start=172.30.5.250,end=172.30.5.254
PUBLIC_NETWORK_GATEWAY="172.30.5.1"
Q_L3_ENABLED=True
PUBLIC_INTERFACE=em1
Q_USE_PROVIDERNET_FOR_PUBLIC=True
OVS_PHYSICAL_BRIDGE=br-ex
PUBLIC_BRIDGE=br-ex
OVS_BRIDGE_MAPPINGS=public:br-ex

disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-meta
enable_service q-l3

enable_plugin neutron-lbaas http://git.openstack.org/openstack/neutron-lbaas
enable_plugin octavia http://github.com/openstack/octavia.git
ENABLED_SERVICES+=,q-lbaasv2
ENABLED_SERVICES+=,octavia,o-cw,o-hk,o-hm,o-api

Below are debug info for lb creation & error output

>$:~/Openstack/devstack$ neutron --debug lbaas-loadbalancer-create --name lb2 private-subnet
DEBUG: keystoneclient.session REQ: curl -g -i -X GET http://172.30.5.21:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
DEBUG: keystoneclient.session RESP: [200] content-length: 337 vary: X-Auth-Token keep-alive: timeout=5, max=100 server: Apache/2.4.7 (Ubuntu) connection: Keep-Alive date: Wed, 14 Oct 2015 19:12:21 GMT content-type: application/json x-openstack-request-id: req-832927f6-b9b4-4e35-a157-5ede404a690f
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://172.30.5.21:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG: stevedore.extension found extension EntryPoint.parse('table = cliff.formatters.table:TableFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('json = cliff.formatters.json_format:JSONFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('shell = cliff.formatters.shell:ShellFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('value = cliff.formatters.value:ValueFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('yaml = cliff.formatters.yaml_format:YAMLFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('yaml = clifftablib.formatters:YamlFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('json = clifftablib.formatters:JsonFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('html = clifftablib.formatters:HtmlFormatter')
DEBUG: neutronclient.neutron.v2_0.lb.v2.loadbalancer.CreateLoadBalancer get_data(Namespace(admin_state=True, columns=[], description=None, formatter='table', max_width=0, name=u'lb2', noindent=False, prefix='', provider=None, request_format='json', tenant_id=None, variables=[], vip_address=None, vip_subnet=u'private-subnet'))
DEBUG: keystoneclient.auth.identity.v2 Making authentication request to http://172.30.5.21:5000/v2.0/tokens
DEBUG: keystoneclient.session REQ: curl -g -i -X GET http://172.30.5.21:9696/v2.0/subnets.json?fields=id&name=private-subnet -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}5fe5f7ed8ce3a71e119ba3dfbca42657283726f8"
DEBUG: keystoneclient.session RESP: [200] date: Wed, 14 Oct 2015 19:12:21 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 61 x-openstack-request-id: req-ef4a0578-0c62-40fd-9593-058526e9d2fc
RESP BODY: {"subnets": [{"id": "fbc4b629-69f7-4c38-bc51-0d90ba376355"}]}

DEBUG: keystoneclient.session REQ: curl -g -i -X POST http://172.30.5.21:9696/v2.0/lbaas/loadbalancers.json -H "User-Agent: python-neutronclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}5fe5f7ed8ce3a71e119ba3dfbca42657283726f8" -d '{"loadbalancer": {"vip_subnet_id": "fbc4b629-69f7-4c38-bc51-0d90ba376355", "name": "lb2", "admin_state_up": true}}'
DEBUG: keystoneclient.session RESP: [201] date: Wed, 14 Oct 2015 19:12:22 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 422 x-openstack-request-id: req-13ff2b40-6d29-43db-bf7a-b802038399eb
RESP BODY: {"loadbalancer": {"description": "", "admin_state_up": true, "tenant_id": "eb9edd97154e46d78f3e1fdb38b0c62b", "provisioning_status": "PENDING_CREATE", "listeners": [], "vip_address": "10.0.0.7", "vip_port_id": "ff827bda-92e5-4a46-a95f-747536e81cc1", "provider": "octavia", "vip_subnet_id": "fbc4b629-69f7-4c38-bc51-0d90ba376355", "id": "1cfd8721-2e60-4dd9-beca-56deaf74a3a3", "operating_status": "OFFLINE", "name": "lb2"}}

/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392
2015-10-15 00:46:07.285 ERROR neutron_lbaas.agent.agent_manager [-] Unable to retrieve ready devices
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager Traceback (most recent call last):
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager File "/opt/stack/neutron-lbaas/neutron_lbaas/agent/agent_manager.py", line 151, in sync_state
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager ready_instances = set(self.plugin_rpc.get_ready_devices())
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager File "/opt/stack/neutron-lbaas/neutron_lbaas/agent/agent_api.py", line 34, in get_ready_devices
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager return cctxt.call(self.context, 'get_ready_devices', host=self.host)
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager retry=self.retry)
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager timeout=timeout, retry=retry)
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager retry=retry)
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 420, in _send
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager result = self._waiter.wait(msg_id, timeout)
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 318, in wait
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager message = self.waiters.get(msg_id, timeout=timeout)
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 223, in get
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager 'to message ID %s' % msg_id)
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID 6eca73cef71442afbb5f2226864df3bf
2015-10-15 00:46:07.285 TRACE neutron_lbaas.agent.agent_manager
2015-10-15 00:46:07.286 WARNING oslo.service.loopingcall [-] Function 'neutron_lbaas.agent.agent_manager.LbaasAgentManager.run_periodic_tasks' run outlasted interval by 50.00 sec
2015-10-15 00:46:07.286 DEBUG oslo_service.periodic_task [-] Running periodic task LbaasAgentManager.collect_stats from (pid=663) run_periodic_tasks /usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-10-15 00:46:07.287 DEBUG oslo_service.periodic_task [-] Running periodic task LbaasAgentManager.periodic_resync from (pid=663) run_periodic_tasks /usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213
2015-10-15 00:46:07.287 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 7c6683b8e1e7478ab038dc1d372d8928 from (pid=663) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392

Thanks Advance!

-Prabu

Revision history for this message

Pankaj Khandar (pankaj-khandar) wrote on 2015-10-15:

Prabhu,
Any update on this. I am getting the same on Liberty release.
thanks
-Pan

Revision history for this message

Adolfo Duarte (adolfo-duarte) wrote on 2015-10-23:

Download full text (5.8 KiB)

I am also seeing something similar with master branch:
I know rabbit is working correclty becuase other services don't have a problem.

2015-10-22 22:10:42.636 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 351b422c33fd42ffb67326c061d36787 from (pid=5522) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392
2015-10-22 22:11:42.640 ERROR neutron_lbaas.agent.agent_manager [-] Unable to retrieve ready devices
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager Traceback (most recent call last):
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager File "/opt/stack/neutron-lbaas/neutron_lbaas/agent/agent_manager.py", line 151, in sync_state
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager ready_instances = set(self.plugin_rpc.get_ready_devices())
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager File "/opt/stack/neutron-lbaas/neutron_lbaas/agent/agent_api.py", line 34, in get_ready_devices
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager return cctxt.call(self.context, 'get_ready_devices', host=self.host)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager retry=self.retry)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager timeout=timeout, retry=retry)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager retry=retry)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 420, in _send
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager result = self._waiter.wait(msg_id, timeout)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 318, in wait
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager message = self.waiters.get(msg_id, timeout=timeout)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 223, in get
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager 'to message ID %s' % msg_id)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID 351b422c33fd42ffb67326c061d36787
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager
2015-10-22 22:11:42.641 WARNING oslo.service.loopingcall [-] Function 'neutron_lbaas.agent.agent_manager.LbaasAgentManager.run_periodic_tasks' run outlasted interval by 50.01 sec
2015-10-22 22:11:42.641 DEBUG oslo_service.periodic_task [-] Running ...

I am also seeing something similar with master branch: 
I know rabbit is working correclty becuase other services don't have a problem.

2015-10-22 22:10:42.636 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 351b422c33fd42ffb67326c061d36787 from (pid=5522) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:392
2015-10-22 22:11:42.640 ERROR neutron_lbaas.agent.agent_manager [-] Unable to retrieve ready devices
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager Traceback (most recent call last):
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager   File "/opt/stack/neutron-lbaas/neutron_lbaas/agent/agent_manager.py", line 151, in sync_state
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager     ready_instances = set(self.plugin_rpc.get_ready_devices())
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager   File "/opt/stack/neutron-lbaas/neutron_lbaas/agent/agent_api.py", line 34, in get_ready_devices
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager     return cctxt.call(self.context, 'get_ready_devices', host=self.host)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager     retry=self.retry)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager     timeout=timeout, retry=retry)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager     retry=retry)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 420, in _send
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager     result = self._waiter.wait(msg_id, timeout)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 318, in wait
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager     message = self.waiters.get(msg_id, timeout=timeout)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 223, in get
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager     'to message ID %s' % msg_id)
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager MessagingTimeout: Timed out waiting for a reply to message ID 351b422c33fd42ffb67326c061d36787
2015-10-22 22:11:42.640 TRACE neutron_lbaas.agent.agent_manager 
2015-10-22 22:11:42.641 WARNING oslo.service.loopingcall [-] Function 'neutron_lbaas.agent.agent_manager.LbaasAgentManager.run_periodic_tasks' run outlasted interval by 50.01 sec
2015-10-22 22:11:42.641 DEBUG oslo_service.periodic_task [-] Running periodic task LbaasAgentManager.collect_stats from (pid=5522) run_periodic_tasks /usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py:213

Here is my local.conf 
[[local|localrc]]
GIT_BASE=https://git.openstack.org
HOST_IP=192.168.214.178
DATA_IP=192.168.214.178
SERVICE_HOST=192.168.214.178
#OFFLINE=True
Q_USE_DEBUG_COMMAND=True
#Q_DVR_MODE=dvr_snat
#USE_SCREEN=False
#NEUTRON_CREATE_INITIAL_NETWORKS=False
DEST=/opt/stack
# move DATA_DIR outside of DEST to keep DEST a bit cleaner
DATA_DIR=/opt/stack/data
ACTIVE_TIMEOUT=90
BOOT_TIMEOUT=90
ASSOCIATE_TIMEOUT=60
TERMINATE_TIMEOUT=60

ADMIN_PASSWORD=password
MYSQL_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_PASSWORD=password
SERVICE_TOKEN=password

DATABASE_PASSWORD=password

enable_plugin neutron-lbaas https://git.openstack.org/openstack/neutron-lbaas
enable_plugin octavia https://git.openstack.org/openstack/octavia

LIBS_FROM_GIT+=python-neutronclient
# Enable Logging
LOGFILE=$DEST/logs/stack.sh.log
VERBOSE=True
LOG_COLOR=True
SCREEN_LOGDIR=$DEST/logs

# Pre-requisites
enable_service rabbit
enable_service mysql
enable_service key

# Horizon
enable_service horizon

# Nova
enable_service n-api
enable_service n-crt
enable_service n-obj
enable_service n-cpu
enable_service n-cond
enable_service n-sch

# Glance
enable_service g-api
enable_service g-reg

# Neutron
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta

# Cinder
enable_service c-api
enable_service c-vol
enable_service c-sch

# LBaaS V2 and Octavia
enable_service q-lbaasv2
enable_service octavia
enable_service o-cw
enable_service o-hm
enable_service o-hk
enable_service o-api

# enable DVR

Q_PLUGIN=ml2
Q_ML2_TENANT_NETWORK_TYPE=vxlan
#Q_DVR_MODE=dvr_snat

IMAGE_URLS+=",http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img"

LOGFILE=$DEST/logs/stack.sh.log

# Old log files are automatically removed after 7 days to keep things neat.  Change
# the number of days by setting ``LOGDAYS``.
LOGDAYS=2

# Swift
# -----

# Swift is now used as the back-end for the S3-like object store. If Nova's
# objectstore (``n-obj`` in ``ENABLED_SERVICES``) is enabled, it will NOT
# run if Swift is enabled. Setting the hash value is required and you will
# be prompted for it if Swift is enabled so just set it to something already:
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5

# For development purposes the default of 3 replicas is usually not required.
# Set this to 1 to save some resources:
SWIFT_REPLICAS=1

# The data for Swift is stored by default in (``$DEST/data/swift``),
# or (``$DATA_DIR/swift``) if ``DATA_DIR`` has been set, and can be
# moved by setting ``SWIFT_DATA_DIR``. The directory will be created
# if it does not exist.
SWIFT_DATA_DIR=$DEST/data

Revision history for this message

P Ingle (pingle) wrote on 2016-02-19:

any update this one? I am trying to get loadbalancer to work with master and after create it just stays in PENDING_CREATE status, subsequent attempt to create listener on that loadbalancer fails.

Tom Fifield (fifieldt) on 2016-05-30

affects:

openstack-community → octavia

Revision history for this message

Michael Johnson (johnsom) wrote on 2016-06-22:

Hello.

Octavia was not released for Kilo so I would expect you would run into trouble attempting to use it with a Kilo environment.

However, it should work with Liberty. For a liberty install that is failing, can you please attach your localrc, octavia.conf, o-api, and o-cw log files?

Thanks

Changed in octavia:
importance:	Undecided → High
status:	New → Incomplete

Revision history for this message

Launchpad Janitor (janitor) wrote on 2016-08-21:

[Expired for octavia because there has been no activity for 60 days.]

Changed in octavia:
status:	Incomplete → Expired

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.