Description
===========
TripleO 16.1 Beta Centos7 Train/Current deploys Octavia on overcloud using the supplied template
-e /usr/share/openstack-tripleo-heat-templates/environment/service/octavia.yaml to the deployment command.
The configuration is 3 Controllers,3 Compute,3 Ceph-Storage with SSL Public Endpoints and OVN (onv-ha-dvr).
However, once deployed, all loadbalancer commands result in a HTTP 503 Service unavailable response. Here are the logs from driver-agent.log and octavia.log from any of the three controllers.
/var/log/containers/octavia/octavia.log
2020-07-15 11:00:47.730 24 INFO octavia.common.config [-] Logging enabled!
2020-07-15 11:00:47.730 24 INFO octavia.common.config [-] mod_wsgi version 5.0.2
2020-07-15 11:00:47.774 25 INFO octavia.common.config [-] Logging enabled!
2020-07-15 11:00:47.774 25 INFO octavia.common.config [-] mod_wsgi version 5.0.2
2020-07-15 11:03:12.962 25 ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver ovn due to: Unable to open the driver agent socket: /var/run/octavia/status.sock: DriverAgentNotFound: Unable to open the driver agent socket: /var/run/octavia/status.sock
2020-07-15 11:03:12.977 24 ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver ovn due to: Unable to open the driver agent socket: /var/run/octavia/status.sock: DriverAgentNotFound: Unable to open the driver agent socket: /var/run/octavia/status.sock
2020-07-15 11:03:13.100 24 INFO octavia.common.config [-] Logging enabled!
2020-07-15 11:03:13.100 24 INFO octavia.common.config [-] mod_wsgi version 5.0.2
2020-07-15 11:03:13.106 25 INFO octavia.common.config [-] Logging enabled!
2020-07-15 11:03:13.107 25 INFO octavia.common.config [-] mod_wsgi version 5.0.2
2020-07-15 11:05:30.297 24 ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver ovn due to: Unable to open the driver agent socket: /var/run/octavia/status.sock: DriverAgentNotFound: Unable to open the driver agent socket: /var/run/octavia/status.sock
2020-07-15 11:05:30.298 25 ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver ovn due to: Unable to open the driver agent socket: /var/run/octavia/status.sock: DriverAgentNotFound: Unable to open the driver agent socket: /var/run/octavia/status.sock
/var/log/containers/octavia/driver-agent.log
2020-07-15 11:59:34.673 32 ERROR octavia.cmd.driver_agent [-] stats_listener raised exception: [Errno 13] Permission denied. Restarting stats_listener.: error: [Errno 13] Permission denied
2020-07-15 11:59:34.673 32 ERROR octavia.cmd.driver_agent Traceback (most recent call last):
2020-07-15 11:59:34.673 32 ERROR octavia.cmd.driver_agent File "/usr/lib/python2.7/site-packages/octavia/cmd/driver_agent.py", line 65, in _process_wrapper
2020-07-15 11:59:34.673 32 ERROR octavia.cmd.driver_agent function(exit_event)
2020-07-15 11:59:34.673 32 ERROR octavia.cmd.driver_agent File "/usr/lib/python2.7/site-packages/octavia/api/drivers/driver_agent/driver_listener.py", line 138, in stats_listener
2020-07-15 11:59:34.673 32 ERROR octavia.cmd.driver_agent StatsRequestHandler)
2020-07-15 11:59:34.673 32 ERROR octavia.cmd.driver_agent File "/usr/lib64/python2.7/SocketServer.py", line 419, in __init__
2020-07-15 11:59:34.673 32 ERROR octavia.cmd.driver_agent self.server_bind()
2020-07-15 11:59:34.673 32 ERROR octavia.cmd.driver_agent File "/usr/lib64/python2.7/SocketServer.py", line 430, in server_bind
2020-07-15 11:59:34.673 32 ERROR octavia.cmd.driver_agent self.socket.bind(self.server_address)
2020-07-15 11:59:34.673 32 ERROR octavia.cmd.driver_agent File "/usr/lib64/python2.7/socket.py", line 224, in meth
2020-07-15 11:59:34.673 32 ERROR octavia.cmd.driver_agent return getattr(self._sock,name)(*args)
Steps to reproduce
==================
1) Deploy Undercloud configuration of Tripleo 16.1 Beta (upstream current) for Train on CentOS7.
2) Deploy Overcloud with Octavia by adding
-e /usr/share/openstack-tripleo-heat-templates/environment/service/octavia.yaml
3) Deployment has 3 controllers, 3 compute, 3 ceph-storage nodes; manila, barbican, octavia, ironic and sahara projects.
THT=/usr/share/openstack-tripleo-heat-templates
openstack overcloud deploy --templates $THT \
-e $THT/environments/network-environment.yaml \
-e /home/stack/templates/network-environment.yaml -e /home/stack/templates_generated/environments/network-isolation.yaml \
-e $THT/environments/services/octavia.yaml \
-e $THT/environments/docker.yaml \
-e $THT/environments/docker-ha.yaml \
-e $THT/environments/ceph-ansible/ceph-ansible.yaml --ntp-server 10.2.2.14 \
-e $THT/environments/ceph-ansible/ceph-mds.yaml \
-e /home/stack/templates/my-ceph-settings.yaml \
-e $THT/environments/docker.yaml \
-r /home/stack/templates/roles_data.yaml \
-n /home/stack/templates/network_data.yaml \
-e /home/stack/ssl-heat-templates/environments/ssl/enable-tls.yaml \
-e /home/stack/ssl-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e /home/stack/ssl-heat-templates/environments/ssl/inject-trust-anchor.yaml \
-e /home/stack/ssl-heat-templates/environments/ssl/inject-trust-anchor-hiera.yaml \
-e /home/stack/templates/public_vip.yaml \
-e /home/stack/containers-prepare-parameter.yaml \
-e $THT/environments/services/sahara.yaml \
-e $THT/environments/manila-cephfsnative-config.yaml \
-e $THT/environments/services/ironic.yaml \
-e $THT/environments/services/ironic-overcloud.yaml \
-e $THT/environments/services/neutron-ovn-dvr-ha.yaml \
-e /home/stack/ironic-config.yaml \
-e /home/stack/templates/node-info.yaml \
-e $THT/environments/services/barbican.yaml \
-e $THT/environments/barbican-backend-simple-crypto.yaml \
-e /home/stack/barbican-configure.yaml \
-e /home/stack/templates/fencing.yaml \
-e $THT/environments/services/mistral.yaml \
--libvirt-type kvm \
--timeout 210
4) From Undercloud source overcloudrc then - "openstack loadbalancer create lb1 ..."
5) HTTP 503 Service unavailable returned.
Expected result
===============
The command should result in the creation of loadbalance lb1 with default settings.
Actual result
=============
Service Unavailable HTTP 503.
ADDITIONAL INFORMATION
- I have tracked the bug to the octavia_driver_agent docker container which has Ownership root.root for /var/run/octavia where the octavia socket is created.
- The docker container USER is "octavia". So hence the "[Errno 13] Permission Denied".
- The docker image is centos-binary-octavia-api. Which probably needs updating.
Okay, I have workarounds for all the bugs that stopped Octavia (Openstack Loadbalancer) coming up for TripleO 16.1 Beta - Train on CentOS7 production capable installation. I hate having to answer all my own questions.
OCTAVIA BUGS OR MISCONFIGURATIONS
1. /var/run/octavia has incorrect ownership of root.root. It needs to be oactavia.octavia it prevents the new driver_agent from running, so nothing runs then.
WORKAROUND
Go to the Docker Container octavia_
You have to execute these on all controllers.
-------
sudo docker exec -ti --user root octavia_
chown octavi.octavia /var/run/octavia
exit
sudo docker restart octavia_
sudo docker restart octavia_worker
sudo docker restart octavia_
sudo docker restart octavia_
------------------
I did try changing the docker image in the container-
2. No Octavia certs get generated automatically.
The logic within the deployment jinga2 script /usr/share/
WORKAROUND
So it only works if your TripleO deploys completely the first time round. If you are looking at multiple Controllers and Computes the share number of moving parts makes this highly unlikely. I think the logic should be changed so that generate_certs is true when OctaviaGenerate
Is there a problem with regenerating the certs each time you deploy??? (it's better than not creating them at all)
My TripleO never deploys completely each time, I always have at least one restart even just owing to docker.io pull bandwidth issues. So I changed Stack - CREATE to UPDATE. I'll change it to CREATE or UPDATE (the best option) on my next redeployment.
-------
generate_certs:
and:
- get_param: OctaviaGenerate
- equals:
- get_param: StackAction
# - CREATE
- UPDATE
3. No Octavia roles are generated by deployment scripts. This is a strange one. The Policy file is generated and in place but no roles are set. Octavia cannot work without the 'load-balancer_
This was an abandoned update (shown below possible fix).
WORKAROUND
-------
source overcloudrc
openstack role create load-balancer_
openstack role create load-balancer_admin
openstack role create load-balancer_
openstack role create load-balancer_
openstack role create load-balancer_
For USER and...