Redis container fails to start:
Jul 24 06:59:21 standalone-1.localdomain systemd[103511]: tripleo_redis.service: Failed to set up special execution directory in /run: Permission denied
Jul 24 06:59:21 standalone-1.localdomain systemd[103511]: tripleo_redis.service: Failed at step RUNTIME_DIRECTORY spawning /bin/chcon: Permission denied
[vagrant@standalone-1 yum.repos.d]$ rpm -qa | grep redis
puppet-redis-6.0.1-0.20200528180336.101cb49.el8.noarch
[vagrant@standalone-1 yum.repos.d]$ rpm -qa | grep tripleo
openstack-tripleo-puppet-elements-12.4.0-0.20200710233941.96199e9.el8.noarch
openstack-tripleo-image-elements-12.1.0-0.20200609053734.dafd3c7.el8.noarch
tripleo-ansible-1.5.0-0.20200714015159.4e0293e.el8.noarch
python3-tripleo-repos-0.1.1-0.20200702223938.f18928a.el8.noarch
openstack-tripleo-common-12.5.0-0.20200710115415.1bab8cc.el8.noarch
openstack-tripleo-common-containers-12.5.0-0.20200710115415.1bab8cc.el8.noarch
python3-tripleo-common-12.5.0-0.20200710115415.1bab8cc.el8.noarch
openstack-tripleo-validations-12.4.0-0.20200714014423.a3fbf34.el8.noarch
puppet-tripleo-13.0.0-0.20200707113950.4cca6ed.el8.noarch
openstack-tripleo-heat-templates-12.4.0-0.20200714172342.a6fd3bd.el8.noarch
python3-tripleoclient-13.4.0-0.20200713220947.2c6dc2f.el8.noarch
ansible-role-tripleo-modify-image-1.2.1-0.20200706163521.affe2c3.el8.noarch
ansible-tripleo-ipa-0.2.1-0.20200609201504.c22fc8d.el8.noarch
ansible-tripleo-ipsec-9.2.1-0.20200513070458.0c8693c.el8.noarch
Detailed logs:
2020-07-23 17:33:02.261630 | 5254009a-08f8-1d4e-5d10-000000001457 | CHANGED | Check podman create status | standalone-1 | item={'started': 1, 'finished': 0, 'ansible_job_id': '732746662347.24654', 'results_file': '/tmp/.ansible_async/7
32746662347.24654', 'changed': True, 'failed': False, 'container_data': {'redis': {'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'healthcheck': {'test': '/openstack/healthcheck'}, 'image': 'docker.io/tripleomaster/centos-binary
-redis:current-tripleo', 'net': 'host', 'privileged': False, 'restart': 'always', 'start_order': 1, 'stop_grace_period': 60, 'systemd_exec_flags': {'ExecStartPre': '/bin/chcon -t container_file_t /run/redis', 'RuntimeDirectory': 'redis'},
'volumes': ['/var/lib/kolla/config_files/redis.json:/var/lib/kolla/config_files/config.json:ro', '/var/lib/config-data/puppet-generated/redis:/var/lib/kolla/config_files/src:ro', '/etc/localtime:/etc/localtime:ro', '/var/log/containers/r
edis:/var/log/redis:z', '/run/redis:/run/redis:z']}}, 'ansible_loop_var': 'container_data'}
2020-07-23 17:33:02.378509 | 5254009a-08f8-1d4e-5d10-000000001459 | TASK | Check containers status
2020-07-23 17:33:02.379042 | 5254009a-08f8-1d4e-5d10-000000001457 | TIMING | tripleo_container_manage : Check podman create status | 0:05:00.640 | 10.74s
These containers are supposed to terminate with a valid exit code and will be checked: ['mysql_bootstrap']
podman run --name redis --conmon-pidfile /var/run/redis.pid --detach=True --env KOLLA_CONFIG_STRATEGY=COPY_ALWAYS --healthcheck-command /openstack/healthcheck --healthcheck-interval 60s --interactive=False --label config_id=tripleo_step1
--label container_name=redis --label managed_by=tripleo_ansible --label config_data={'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'healthcheck': {'test': '/openstack/healthcheck'}, 'image': 'docker.io/tripleomaster/centos-bina
ry-redis:current-tripleo', 'net': 'host', 'privileged': False, 'restart': 'always', 'start_order': 1, 'stop_grace_period': 60, 'systemd_exec_flags': {'ExecStartPre': '/bin/chcon -t container_file_t /run/redis', 'RuntimeDirectory': 'redis'
}, 'volumes': ['/var/lib/kolla/config_files/redis.json:/var/lib/kolla/config_files/config.json:ro', '/var/lib/config-data/puppet-generated/redis:/var/lib/kolla/config_files/src:ro', '/etc/localtime:/etc/localtime:ro', '/var/log/containers
/redis:/var/log/redis:z', '/run/redis:/run/redis:z']} --log-driver k8s-file --log-level debug --log-opt path=/var/log/containers/stdouts/redis.log --privileged=False --stop-timeout 60 --tty=False --volume /var/lib/kolla/config_files/redis
.json:/var/lib/kolla/config_files/config.json:ro --volume /var/lib/config-data/puppet-generated/redis:/var/lib/kolla/config_files/src:ro --volume /etc/localtime:/etc/localtime:ro --volume /var/log/containers/redis:/var/log/redis:z --volum
e /run/redis:/run/redis:z --network host --rootfs=False docker.io/tripleomaster/centos-binary-redis:current-tripleo
2020-07-23 17:33:02.993687 | 5254009a-08f8-1d4e-5d10-000000001459 | CHANGED | Check containers status | standalone-1
2020-07-23 17:33:03.118864 | 5254009a-08f8-1d4e-5d10-00000000145a | TASK | Create fact for containers which changed
2020-07-23 17:33:03.119367 | 5254009a-08f8-1d4e-5d10-000000001459 | TIMING | tripleo_container_manage : Check containers status | 0:05:01.380 | 0.74s
2020-07-23 17:33:03.174219 | 5254009a-08f8-1d4e-5d10-00000000145a | OK | Create fact for containers which changed | standalone-1
2020-07-23 17:33:03.294890 | 5254009a-08f8-1d4e-5d10-00000000145b | TASK | Append the list of all podman commands that are run for containers with changes
2020-07-23 17:33:03.295410 | 5254009a-08f8-1d4e-5d10-00000000145a | TIMING | tripleo_container_manage : Create fact for containers which changed | 0:05:01.556 | 0.18s
2020-07-23 17:33:03.350121 | 5254009a-08f8-1d4e-5d10-00000000145b | SKIPPED | Append the list of all podman commands that are run for containers with changes | standalone-1
2020-07-23 17:33:03.471512 | 5254009a-08f8-1d4e-5d10-000000001239 | TASK | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_1
2020-07-23 17:33:03.472007 | 5254009a-08f8-1d4e-5d10-00000000145b | TIMING | tripleo_container_manage : Append the list of all podman commands that are run for containers with changes | 0:05:01.733 | 0.18s
Removing /etc/systemd/system/tripleo_memcached.requires file
Removing /etc/systemd/system/tripleo_redis.requires file
Systemd unit files were created or updated for: ['memcached', 'redis']
Running systemd daemon reload
Restarting systemd service for memcached
Restarting systemd service for redis
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'status'
2020-07-23 17:33:26.814937 | 5254009a-08f8-1d4e-5d10-000000001239 | FATAL | Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_1 | standalone-1 | error={"m
sg": "Unexpected failure during module execution.", "stdout": ""}
PLAY RECAP *********************************************************************
standalone-1 : ok=290 changed=73 unreachable=0 failed=1 skipped=107 rescued=0 ignored=0
undercloud : ok=20 changed=10 unreachable=0 failed=0 skipped=19 rescued=0 ignored=0
2020-07-23 17:33:26.979819 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-07-23 17:33:26.981629 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:05:25.242 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-07-23 17:33:26.983414 | UUID | Info | Task Name | Run Time
2020-07-23 17:33:26.985200 | 5254009a-08f8-1d4e-5d10-000000001239 | SUMMARY | tripleo_container_manage : Manage container systemd services and cleanup old systemd healthchecks for /var/lib/tripleo-config/container-startup-config/step_1
| 23.51s
2020-07-23 17:33:26.986952 | 5254009a-08f8-1d4e-5d10-00000000054a | SUMMARY | tripleo_firewall : Manage firewall rules | 13.82s
2020-07-23 17:33:26.988854 | 5254009a-08f8-1d4e-5d10-000000000fe7 | SUMMARY | tripleo_container_manage : Check containers status | 11.75s
2020-07-23 17:33:26.990643 | 5254009a-08f8-1d4e-5d10-00000000104d | SUMMARY | tripleo_container_manage : Check containers status | 11.71s
020-07-23 17:33:26.992346 | 5254009a-08f8-1d4e-5d10-000000000f81 | SUMMARY | tripleo_container_manage : Check containers status | 11.24s
2020-07-23 17:33:26.994147 | 5254009a-08f8-1d4e-5d10-0000000010b3 | SUMMARY | tripleo_container_manage : Check containers status | 11.13s
2020-07-23 17:33:26.995857 | 5254009a-08f8-1d4e-5d10-000000001457 | SUMMARY | tripleo_container_manage : Check podman create status | 10.74s
2020-07-23 17:33:26.997243 | 5254009a-08f8-1d4e-5d10-000000000c44 | SUMMARY | Write kolla config json files | 9.94s
2020-07-23 17:33:26.998468 | 5254009a-08f8-1d4e-5d10-000000000cdf | SUMMARY | Wait for puppet host configuration to finish | 6.69s
2020-07-23 17:33:26.999802 | 5254009a-08f8-1d4e-5d10-000000000f7f | SUMMARY | tripleo_container_manage : Check podman create status | 6.65s
2020-07-23 17:33:27.001020 | 5254009a-08f8-1d4e-5d10-000000000fe5 | SUMMARY | tripleo_container_manage : Check podman create status | 6.63s
2020-07-23 17:33:27.001996 | 5254009a-08f8-1d4e-5d10-00000000104b | SUMMARY | tripleo_container_manage : Check podman create status | 6.59s
2020-07-23 17:33:27.002893 | 5254009a-08f8-1d4e-5d10-000000000b66 | SUMMARY | tripleo_container_tag : Pull docker.io/tripleomaster/centos-binary-cinder-volume:current-tripleo image | 6.06s
2020-07-23 17:33:27.003756 | 5254009a-08f8-1d4e-5d10-0000000012f2 | SUMMARY | tripleo_container_manage : Check podman create status | 5.95s
2020-07-23 17:33:27.004652 | 5254009a-08f8-1d4e-5d10-000000000109 | SUMMARY | Run tripleo_container_image_prepare logged to: /var/log/tripleo-container-image-prepare.log | 5.62s
2020-07-23 17:33:27.005557 | 5254009a-08f8-1d4e-5d10-0000000010b1 | SUMMARY | tripleo_container_manage : Check podman create status | 5.57s
2020-07-23 17:33:27.006436 | 5254009a-08f8-1d4e-5d10-000000000bb3 | SUMMARY | Pre-fetch all the containers | 5.49s
2020-07-23 17:33:27.007135 | 5254009a-08f8-1d4e-5d10-000000000095 | SUMMARY | Run puppet on the host to apply IPtables rules | 4.86s
2020-07-23 17:33:27.007894 | 5254009a-08f8-1d4e-5d10-00000000005b | SUMMARY | Run NetworkConfig script | 3.61s
2020-07-23 17:33:27.008611 | 5254009a-08f8-1d4e-5d10-000000000ba9 | SUMMARY | tripleo_container_tag : Pull docker.io/tripleomaster/centos-binary-rabbitmq:current-tripleo image | 3.59s
2020-07-23 17:33:27.009293 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-07-23 17:33:27.010007 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-07-23 17:33:27.010686 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~
2020-07-23 17:33:27.011367 | This or these node(s) failed to deploy: standalone-1
2020-07-23 17:33:27.011924 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Ansible execution failed. playbook: deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2
Not cleaning working directory /home/vagrant/tripleo-heat-installer-templates
Not cleaning ansible directory /home/vagrant/standalone-ansible-cxk7uzfl
Install artifact is located at /home/vagrant/standalone-install-20200723173327.tar.bzip2
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Deployment Failed!
ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-9_dspze3
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Deployment failed.
[root@standalone-1 ~]# systemctl start tripleo_redis.service
Job for tripleo_redis.service failed because the control process exited with error code.
See "systemctl status tripleo_redis.service" and "journalctl -xe" for details.
Jul 24 06:59:21 standalone-1.localdomain systemd[1]: Starting redis container...
Jul 24 06:59:21 standalone-1.localdomain systemd[103511]: tripleo_redis.service: Failed to set up special execution directory in /run: Permission denied
Jul 24 06:59:21 standalone-1.localdomain systemd[103511]: tripleo_redis.service: Failed at step RUNTIME_DIRECTORY spawning /bin/chcon: Permission denied
Jul 24 06:59:21 standalone-1.localdomain systemd[1]: tripleo_redis.service: Control process exited, code=exited status=233
Jul 24 06:59:21 standalone-1.localdomain systemd[103513]: tripleo_redis.service: Failed to set up special execution directory in /run: Permission denied
Jul 24 06:59:21 standalone-1.localdomain systemd[103513]: tripleo_redis.service: Failed at step RUNTIME_DIRECTORY spawning /usr/bin/podman: Permission denied
Jul 24 06:59:21 standalone-1.localdomain systemd[1]: tripleo_redis.service: Control process exited, code=exited status=233
Jul 24 06:59:21 standalone-1.localdomain systemd[1]: tripleo_redis.service: Failed with result 'exit-code'.
Jul 24 06:59:21 standalone-1.localdomain systemd[1]: Failed to start redis container.
Jul 24 06:59:21 standalone-1.localdomain systemd[1]: tripleo_redis.service: Service RestartSec=100ms expired, scheduling restart.
Jul 24 06:59:21 standalone-1.localdomain systemd[1]: tripleo_redis.service: Scheduled restart job, restart counter is at 1.
Jul 24 06:59:21 standalone-1.localdomain systemd[1]: Stopped redis container.
SElinux:
type=AVC msg=audit(