swift_rsync_container: failed to create pid file

[Dan Prince]

When restarting or updating the swift rsync container I'm seeing the following error in the logs:

+ CMD='/usr/bin/rsync --daemon --no-detach --config=/etc/rsyncd.conf'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
+ echo 'Running command: '\''/usr/bin/rsync --daemon --no-detach --config=/etc/rsyncd.conf'\'''
+ exec /usr/bin/rsync --daemon --no-detach --config=/etc/rsyncd.conf
Running command: '/usr/bin/rsync --daemon --no-detach --config=/etc/rsyncd.conf'
failed to create pid file /var/run/rsyncd.pid: File exists

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/513020

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/513020
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f5754bfe53e90f5559a2bce487607a3a2491db6e
Submitter: Zuul
Branch: master

commit f5754bfe53e90f5559a2bce487607a3a2491db6e
Author: Dan Prince <email address hidden>
Date: Wed Oct 18 08:39:17 2017 -0400

    swift_rsync: don't bind mount /run

    This resolves an issue where the pid file exists from a previous
    run of the container.

    Change-Id: Id051172407f0e879d3edf18c8b2ec13734794ed2
    Closes-bug: #1724559

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/545864

[Michele Baldessari (michele)]

Downstream reference: https://bugzilla.redhat.com/show_bug.cgi?id=1546807

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545864
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6587b6e8a00bac9f0861d0cd311eb8b4866b814c
Submitter: Zuul
Branch: stable/pike

commit 6587b6e8a00bac9f0861d0cd311eb8b4866b814c
Author: Dan Prince <email address hidden>
Date: Wed Oct 18 08:39:17 2017 -0400

    swift_rsync: don't bind mount /run

    This resolves an issue where the pid file exists from a previous
    run of the container.

    Change-Id: Id051172407f0e879d3edf18c8b2ec13734794ed2
    Closes-bug: #1724559
    (cherry picked from commit f5754bfe53e90f5559a2bce487607a3a2491db6e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.10 release.

[Bogdan Dobrelya (bogdando)]

I reproduced that with

Name : openstack-tripleo-heat-templates
Version : 8.0.0
Release : 0.20180305111919.910653c.el7.centos

Even though /var/run is not mounted from host anymore, the issue retains with the internal /var/run state it seems

[Bogdan Dobrelya (bogdando)]

Here is a config snippet for the container (a fragment of /var/lib/tripleo-config/hashed-docker-container-startup-config-step_4.json):

  "swift_rsync": {
    "image": "docker.io/tripleomaster/centos-binary-swift-object:current-tripleo",
    "environment": [
      "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS",
      "TRIPLEO_CONFIG_HASH=39bb96726bef0af93b7d29bff71bfdc7"
    ],
    "user": "root",
    "volumes": [
      "/etc/hosts:/etc/hosts:ro",
      "/etc/localtime:/etc/localtime:ro",
      "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
      "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
      "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
      "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
      "/dev/log:/dev/log",
      "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro",
      "/etc/puppet:/etc/puppet:ro",
      "/var/lib/kolla/config_files/swift_rsync.json:/var/lib/kolla/config_files/config.json:ro",
      "/var/lib/config-data/puppet-generated/swift/:/var/lib/kolla/config_files/src:ro",
      "/srv/node:/srv/node",
      "/dev:/dev"
    ],
    "net": "host",
    "privileged": true,
    "restart": "always"

[Bogdan Dobrelya (bogdando)]

Related bug: https://bugs.launchpad.net/tripleo/+bug/1734674

The fix for docker/services is not complete though, we need to make sure it works for a container restarts as well

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/577126

[Christian Schwede (cschwede)]

I was able to reproduce this too.

It happens if the restart of the container hits the default 10 second timeout; in that case the rsync process is killed, leaving the pidfile - which prevents a restart.

Actually we don't need the pid file at all when using containers, so we should get rid of it. However, it's currently hard-coded into the template in puppetlabs-rsync [1], so we can't simply change that setting. I proposed a patch for this [2], but we need a workaround until that PR gets merged.

I proposed one that only relies on a t-h-t change: https://review.openstack.org/577126

[1] https://github.com/puppetlabs/puppetlabs-rsync/blob/master/templates/header.erb#L4
[2] https://github.com/puppetlabs/puppetlabs-rsync/pull/120

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/577403

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/577403
Reason: The gate is suffering of timeouts, we need to clear it. Please do not restore or recheck this patch, I'll take care of it when gate is stable again.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/577403
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=448e04029e5d8bc982cbc3fbd4409304488437ec
Submitter: Zuul
Branch: master

commit 448e04029e5d8bc982cbc3fbd4409304488437ec
Author: Christian Schwede <email address hidden>
Date: Thu Jun 21 11:12:30 2018 +0000

    Disable pid file usage in the swift_rsync container

    The pidfile is useless within containers, but prevents a restart if the
    process has been stopped unclean. This happens for example when a
    restart takes longer than the default timeout of 10 seconds.

    This is mainly a workaround because the pid file setting is hardcoded in
    the used Puppet module template. It should be removed later once the
    setting can be disabled cleanly with an updated Puppet rsync module.

    Related-Bug: 1724559
    Change-Id: Iecf855a785bc1f787234e4e54430c929cb9cb906

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/577126
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6499f09f8b1206cfb3697ebd7005b5308359c016
Submitter: Zuul
Branch: stable/queens

commit 6499f09f8b1206cfb3697ebd7005b5308359c016
Author: Christian Schwede <email address hidden>
Date: Thu Jun 21 11:12:30 2018 +0000

    Disable pid file usage in the swift_rsync container

    The pidfile is useless within containers, but prevents a restart if the
    process has been stopped unclean. This happens for example when a
    restart takes longer than the default timeout of 10 seconds.

    This is mainly a workaround because the pid file setting is hardcoded in
    the used Puppet module template. It should be removed later once the
    setting can be disabled cleanly with an updated Puppet rsync module.

    Related-Bug: 1724559
    Change-Id: Iecf855a785bc1f787234e4e54430c929cb9cb906
    (cherry picked from commit 448e04029e5d8bc982cbc3fbd4409304488437ec)