Octavia loadbalancer pending create state

[BN]

**Bug Report**

What happened:

Following 2 similar guides on how to create octavia loadbalancer with amphora images are stuck at pending create state. Amphora instances are active and working without issues. However, for some reason octavia-worker shows in logs that:

Could not connect to instance. Retrying.: ConnectTimeout: HTTPSConnectionPool(host='172.16.103.35', port=9443): Max retries exceeded with url: /0.5/info (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1c2672d750>, 'Connection to 172.16.103.35 timed out. (connect timeout=10.0)'))

Thus, I have associated public ip to amphora image and it is pingable.

What you expected to happen:

It is expected loadbalancer to be active and working. Those 2 guides were followed:

https://shreddedbacon.com/post/openstack-kolla/
http://www.panticz.de/openstack-octavia-loadbalancer

How to reproduce it (minimal and precise):

Globals:

kolla_base_distro: "ubuntu"
kolla_install_type: "source"
openstack_release: "stein"
kolla_internal_vip_address: "10.0.225.254"
network_interface: "enp2s0f0"
neutron_external_interface: "enp2s0f1"
neutron_tenant_network_types: "vxlan,vlan"
enable_barbican: "yes"
enable_chrony: "yes"
enable_cinder: "yes"
enable_cinder_backup: "yes"
enable_fluentd: "yes"
enable_haproxy: "yes"
enable_heat: "yes"
enable_magnum: "yes"
enable_horizon: "yes"
enable_horizon_neutron_lbaas: "no"
enable_horizon_octavia: "{{ enable_octavia | bool }}"
enable_neutron_dvr: "yes"
enable_neutron_lbaas: "yes"
enable_nova_ssh: "yes"
enable_octavia: "yes"
enable_ceph: "yes"
enable_ceph_rgw: "yes"
glance_backend_file: "no"
glance_backend_ceph: "yes"
glance_enable_rolling_upgrade: "no"
barbican_crypto_plugin: "simple_crypto"
barbican_library_path: "/usr/lib/libCryptoki2_64.so"
ironic_dnsmasq_dhcp_range:
tempest_image_id:
tempest_flavor_ref_id:
tempest_public_network_id:
tempest_floating_network_name:
enable_central_logging: "yes"
elasticsearch_datadir_volume: "/opt/logging"
horizon_port: 48000

External and internal networks are created. Amphora instances are created within internal network. Load balancer is assigned with internal vip ip.

**Environment**:
* OS (e.g. from /etc/os-release): ubuntu
* Kernel (e.g. `uname -a`):
* Docker version if applicable (e.g. `docker version`): latest
* Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release): latest
* Docker image Install type (source/binary): source
* Docker image distribution: stein
* Are you using official images from Docker Hub or self built? official
* If self built - Kolla version and environment used to build: no
* Share your inventory file, globals.yml and other configuration files if relevant

[Dincer Celik (dincercelik)]

I'm facing the same issue, and the root cause is a code mismatch. octavia-worker is trying to access the amphora api via "/0.5/info" but it does not exist. The correct version is 1.0, however source code of octavia still points to 0.5. If you try running https://172.16.103.35:9443/1.0/info, you'll see it works.

My current workaround is using template-overrides for building octavia images. Example;

{% block octavia_base_footer %}
RUN sed -i "s/API_VERSION = '0.5'/API_VERSION = '1.0'/g" /usr/lib/python3/dist-packages/octavia/common/constants.py
RUN sed -i "s/VERSION = '0.5'/VERSION = '1.0'/g" /usr/lib/python3/dist-packages/octavia/amphorae/backends/agent/api_server/__init__.py
{% endblock %}

[BN (zatoichy)]

Hi Dincer,

Can you please specify in more detail how can I override template for building octavia images? Thank you

[Dincer Celik (dincercelik)]

Have a look at https://docs.openstack.org/kolla/latest/admin/index.html#building-container-images

[Michael Johnson (johnsom)]

Changing the version constant is not the right answer. You will get failures from doing that.

The issue is that your controller version of Octavia does not match your amphora image version of Octavia.

When building your amphora image, if you are building for a controller version other than "master", you need to set environment variables such that diskimage-builder will install the correct version of Octavia in the image.

Those variables are: DIB_REPOLOCATION_amphora_agent and DIB_REPOREF_amphora_agent.

See the Octavia image building guide for more information: https://docs.openstack.org/octavia/latest/admin/amphora-image-build.html#environment-variables

[Dincer Celik (dincercelik)]

@Michael, you're right but setting DIB variables will not fix the issue completely. You'll also hit https://storyboard.openstack.org/#!/story/2006591

[Dincer Celik (dincercelik)]

My current workaround for this issue is switching to use source images for octavia. Ubuntu binaries are still using 4.0, but we need 4.1, and kolla source images use 4.1.

[Dincer Celik (dincercelik)]

I'm marking this bug as invalid because this is not a kolla related issue.