podman: redis fails to start on reboot

[Dan Prince]

In t-h-t 1d9629ec0b3320bcbc5a4150c8be19c6eb4096eb we added a bind mount to fix non-pacemaker Docker container redis reboot issues by mounting /var/run/redis.

Podman is more sensitive to bind mounts and requires it to exist before starting the container. As such today we get the following error when a non-pacemaker machine is rebooting using podman redis:

podman: unable to start container b623c85da9421b9be6e5c6ad03181fa860dbde0302b49fd4a3d8ba3358fc75d3: container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused \"rootfs_linux.go:58: mounting \\\"/var/run/redis\\\" to rootfs \\\"/var/lib/containers/storage/overlay/ac98382bfe096365bf320b3bfaf35af57a43cf16b7e57a439ef32bb80188df1d/merged\\\" at \\\"/var/run/redis\\\" caused \\\"stat /var/run/redis: no such file or directory\\\"\""

[Dan Prince (dan-prince)]

The best way to fix this is to use systemd to recreate the required /var/run directories and setup permissions there accordingly. Something like this works:

[Unit]
Description=redis container
After=paunch-container-shutdown.service
Wants=
[Service]
Restart=always
RuntimeDirectory=redis
ExecStartPre=/bin/chcon -t svirt_sandbox_file_t /var/run/redis
ExecStart=/usr/bin/podman start -a redis
ExecStop=/usr/bin/podman stop -t 10 redis
KillMode=process

[Install]
WantedBy=multi-user.target

----

NOTE: the addition of RuntimeDirectory and ExecStartPre sections above. I think the best solution here it to make paunch's systemd abstraction "leaky" with regards to systemd.unit creation such that we can inject ad-hoc systemd settings for each services.

[Dan Prince (dan-prince)]

systemd_exec_flags should do it...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/637624

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637624
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=276743b5fe2261a3350586186ae6eda8fc0e0eba
Submitter: Zuul
Branch: master

commit 276743b5fe2261a3350586186ae6eda8fc0e0eba
Author: Dan Prince <email address hidden>
Date: Mon Feb 18 15:38:01 2019 -0500

    Redis: fix podman start on reboot

    This patch updates the tripleo_redis.service systemd unit
    so that it recreates the /var/run/redis directory and sets
    the SELinux context accordingly after a reboot. Previously
    it would fail on reboot as these files are created via Ansible
    only during deployment.

    Change-Id: I4df04007098b610936d172ecfc5a92dcb1fb3fdc
    Closes-bug: #1816444

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.4.0 release.