This is a follow-up to bug #1806113
With https://review.openstack.org/#/c/628957 applied on current master, deployment passes with Octavia enabled (adding -e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml )
Basic operations (image/network/server) work fine
However the Octavia containers show up as "Restarting (2) 11 minutes ago" and API calls do not work:
# openstack loadbalancer list
Unable to establish connection to http://192.168.122.237:9876/v2.0/lbaas/loadbalancers: HTTPConnectionPool(host='192.168.122.237', port=9876): Max retries exceeded with url: /v2.0/lbaas/loadbalancers (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbad27e31d0>: Failed to establish a new connection: [Errno 111] Connexion refus\xc3\xa9e',))
# docker ps|grep octa
21e1272c6c8e docker.io/tripleomaster/centos-binary-octavia-health-manager:b32ba15b0e053d6660c35713dec47ff941c11c2e_01511b06 "dumb-init --singl..." 2 days ago Restarting (2) 6 hours ago octavia_health_manager
881f081b6243 docker.io/tripleomaster/centos-binary-octavia-api:b32ba15b0e053d6660c35713dec47ff941c11c2e_01511b06 "dumb-init --singl..." 2 days ago Restarting (2) 6 hours ago octavia_api
58f19b672475 docker.io/tripleomaster/centos-binary-octavia-housekeeping:b32ba15b0e053d6660c35713dec47ff941c11c2e_01511b06 "dumb-init --singl..." 2 days ago Restarting (2) 6 hours ago octavia_housekeeping
a82ca5a66ab8 docker.io/tripleomaster/centos-binary-octavia-worker:b32ba15b0e053d6660c35713dec47ff941c11c2e_01511b06 "dumb-init --singl..." 2 days ago Restarting (2) 6 hours ago octavia_worker
All 4 affected containers show similar output with "docker logs":
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/my.cnf.d/tripleo.cnf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/tripleo.cnf to /etc/my.cnf.d/tripleo.cnf
INFO:__main__:Deleting /etc/octavia/octavia.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/octavia/octavia.conf to /etc/octavia/octavia.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/octavia/conf.d/octavia-worker/worker-post-deploy.conf to /etc/octavia/conf.d/octavia-worker/worker-post-deploy.conf
ERROR:__main__:Unexpected error:
Traceback (most recent call last):
File "/usr/local/bin/kolla_set_configs", line 411, in main
execute_config_strategy(config)
File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy
copy_config(config)
File "/usr/local/bin/kolla_set_configs", line 306, in copy_config
config_file.copy()
File "/usr/local/bin/kolla_set_configs", line 150, in copy
self._merge_directories(source, dest)
File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
os.path.join(dest, to_copy))
File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
os.path.join(dest, to_copy))
File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
os.path.join(dest, to_copy))
File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
os.path.join(dest, to_copy))
File "/usr/local/bin/kolla_set_configs", line 99, in _merge_directories
self._copy_file(source, dest)
File "/usr/local/bin/kolla_set_configs", line 82, in _copy_file
shutil.copy(source, dest)
File "/usr/lib64/python2.7/shutil.py", line 119, in copy
copyfile(src, dst)
File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile
with open(src, 'rb') as fsrc:
IOError: [Errno 13] Permission denied: u'/var/lib/kolla/config_files/src/etc/octavia/conf.d/octavia-worker/worker-post-deploy.conf'
As said, other containers are healthy so something specific to Octavia deployment probably triggers this permission issue?
I ran into this myself with the standalone. I started the container in interactive mode with paunch debug and found it was selinux denying access to the file. setenforce 0 and the container works fine.