blocked by Pending to Create

[Cindia-blue]

I am using the latest master branch devstack. but keep get failure with loadbalancer create:

Revised the code to get below log in o-wc:

2016-03-11 17:50:40.823 3785 ERROR octavia.amphorae.drivers.haproxy.rest_api_driver [-] {'url': 'https://192.168.0.5:9443/0.5/plug/vip/10.0.0.5', 'verify': '/etc/octavia/certs/ca_01.pem', 'json': {'subnet_cidr': u'10.0.0.0/24', 'gateway': u'10.0.0.1', 'mac_address': u'fa:16:3e:94:b0:2c'}, 'timeout': (10.0, 60.0), 'headers': {'User-Agent': 'Octavia HaProxy Rest Client/0.5 (https://wiki.openstack.org/wiki/Octavia)'}}
2016-03-11 17:50:40.827 3785 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.

In nova, we can see:
$ nova show b18af3ca-5f5b-4e74-8382-7be0f5e0cc50
+--------------------------------------+------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | lixinhui-virtual-machine |
| OS-EXT-SRV-ATTR:hostname | amphora-1e73795f-6077-4d8b-82e7-73059f50c50e |
| OS-EXT-SRV-ATTR:hypervisor_hostname | lixinhui-virtual-machine |
| OS-EXT-SRV-ATTR:instance_name | instance-00000002 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-sxpghmtp |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-03-11T09:47:34.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | True |
| created | 2016-03-11T09:47:16Z |
| description | amphora-1e73795f-6077-4d8b-82e7-73059f50c50e |
| flavor | m1.amphora (10) |
| hostId | b0b62f895d7384e68b9696aa970b03d1280dca7ecc119d2c75283348 |
| host_status | UP |
| id | b18af3ca-5f5b-4e74-8382-7be0f5e0cc50 |
| image | amphora-x64-haproxy (541089b0-77b9-458d-b16a-c289ce87f151) |
| key_name | octavia_ssh_key |
| lb-mgmt-net network | 192.168.0.5 |
| locked | False |
| metadata | {} |
| name | amphora-1e73795f-6077-4d8b-82e7-73059f50c50e |
| os-extended-volumes:volumes_attached | [] |
| private network | fd3e:1361:3934:0:f816:3eff:fe94:b02c, 10.0.0.6 |
| progress | 0 |
| security_groups | lb-c08db3d4-4d93-4dfb-bc74-71d2662c4f75, lb-mgmt-sec-grp |
| status | ACTIVE |
| tenant_id | eb59c8ab580b40c586c5bda06f51c8f8 |
| updated | 2016-03-11T09:47:34Z |
| user_id | e90ca2384bb64bd5a4e208f449992db9 |
+--------------------------------------+------------------------------------------------------------+

[Cindia-blue (miaoxinhuili)]

Clean the whole machine by making empty of /opt/stack and dist-packages. Then revise the conf section under [haproxy_amphora] to set connection_max_retries as 15000(default is 1500)

The creation of loadbalancer is still pending despite below nova instance created:
$ nova show 466107b2-91db-46ca-9068-48baf64a7c01
+--------------------------------------+------------------------------------------------------------+
| Property | Value |
+--------------------------------------+------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | lixinhui-virtual-machine |
| OS-EXT-SRV-ATTR:hostname | amphora-0e6d3583-1a85-43cc-9425-4e0766888a36 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | lixinhui-virtual-machine |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-mh9f0z7c |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-03-12T01:01:03.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | True |
| created | 2016-03-12T00:59:54Z |
| description | amphora-0e6d3583-1a85-43cc-9425-4e0766888a36 |
| flavor | m1.amphora (10) |
| hostId | 7db1fbe65821bf982e2fff6ceb88b5d8bc41071f5f417f3ad2edc3f3 |
| host_status | UP |
| id ...

[Cindia-blue (miaoxinhuili)]

complete log of o-cw service.

[Michael Johnson (johnsom)]

Can you please collect the following logs for the amphora?

Under /opt/stack/data/nova/instances/<nova instance id>/ there is a console.log
Also, "sudo ssh -i /etc/octavia/.ssh/octavia_ssh_key ubuntu@<amphora mgmt ip>", collect the log /var/log/upstart/amphora-agent.log

This could be a networking issue. Please check that the o-cw environment does not have a proxy configured. If you have a system wide proxy server configured, the requests to the amphora from the controller may be getting directed to the proxy server and not the amphora.

[Cindia-blue (miaoxinhuili)]

The needed console.log for the nova instance

[Cindia-blue (miaoxinhuili)]

/var/log/upstart/amphora-agent.log

[Cindia-blue (miaoxinhuili)]

Attached is the local.conf of my devstack.

[Cindia-blue (miaoxinhuili)]

my etc/octavia/octavia.conf

[Cindia-blue (miaoxinhuili)]

After set " amp_active_wait_sec" as 100, the creation is still pending. more information comes below:

$ neutron net-list
+--------------------------------------+-------------+----------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+-------------+----------------------------------------------------------+
| b1ca2ccb-564a-4109-8131-e63aeee4e801 | public | 21e733d4-07b3-4088-a79a-95a478524fde 2001:db8::/64 |
| | | 6957a375-5041-4051-a70f-c32da85982f8 172.24.4.0/24 |
| 0db492fa-2757-451f-8933-800c168e1c33 | lb-mgmt-net | b34c57f7-00f8-4a4d-acd3-dd7f5374f913 192.168.0.0/24 |
| 2ec7cfd8-c329-4823-bdb5-65972470e8ae | private | 2ba147e0-b14a-4b8a-8682-e1172be940f1 10.0.0.0/24 |
| | | 72e3685b-6305-4d46-8958-c3a0a7bcb026 fdf4:8f50:c23b::/64 |
+--------------------------------------+-------------+----------------------------------------------------------+
$ neutron lbaas-loadbalancer-create --name lixinhui-lb2 2ba147e0-b14a-4b8a-8682-e1172be940f1
Created a new loadbalancer:
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| admin_state_up | True |
| description | |
| id | 1b1a099f-b471-4757-a382-ae1b20043cde |
| listeners | |
| name | lixinhui-lb2 |
| operating_status | OFFLINE |
| pools | |
| provider | octavia |
| provisioning_status | PENDING_CREATE |
| tenant_id | 0132dc728b4040ec89c4939dbb4f1deb |
| vip_address | 10.0.0.7 |
| vip_port_id | 2b5246b7-16ce-4223-b22f-e208a8eba9ca |
| vip_subnet_id | 2ba147e0-b14a-4b8a-8682-e1172be940f1 |
+---------------------+--------------------------------------+

$ tail -f o-cw.log.2016-03-11-212812
2016-03-12 10:52:19.618 2516 INFO octavia.controller.queue.endpoint [-] Creating load balancer '1b1a099f-b471-4757-a382-ae1b20043cde'...
2016-03-12 10:52:21.213 2516 INFO octavia.controller.worker.tasks.database_tasks [-] Created Amphora in DB with id c65a6680-71dc-4536-83bf-b8c1e840d8bb
2016-03-12 10:52:21.780 2516 INFO octavia.certificates.generator.local [-] Signing a certificate request using OpenSSL locally.
2016-03-12 10:52:21.781 2516 INFO octavia.certificates.generator.local [-] Using CA Certificate from config.
2016-03-12 10:52:21.782 2516 INFO octavia.certificates.generator.local [-] Using CA Private Key from config.
2016-03-12 10:52:21.782 2516 INFO octavia.certificates.generator.local [-] Using CA Private Key Passphrase from config.
2016-03-12 10:54:11.129 2516 INFO octavia.controller.worker.task...

[Bharath (bharathm)]

I am unable to reproduce this issue.

Looking at your logs, it appears the POST request is reaching the amphora but the response is somehow getting terminated. As the error was triggered at self._sslobj.write(data), may be it's a certificate error ?

I tried it on a fresh devstack install with latest octavia bits with successful results.
FYI, my setup is below:

H/V: VMWare Workstation
Host: Ubuntu 15.10 running Devstack Master Branch

[Lingxian Kong (kong)]

I met problems similiar to this days ago, but solved it according to Michael's suggesion. You could try:

rest_request_conn_timeout = 60
rest_request_read_timeout = 300

The reason of mine is because I setup devstack inside vagrant vm on my laptop, nested virtualizaion. You can even set the value of the above options bigger.

[Cindia-blue (miaoxinhuili)]

I once tried the two settings in my env. Anyway, I will try with the latest release then return to this bug with update.

[Cindia-blue (miaoxinhuili)]

I have tried and this problem will not reproduce in the same experimental environment (VM, Guest OS, and Network) with Mitaka stable version. So close this bug.