VPNaaS:pluto.pid already exists

[Shogo3]

1.Edit existing IPSec site connections
2.Save changes

2014-06-11 08:12:36.174 28251 TRACE neutron.services.vpn.device_drivers.ipsec Stderr: 'shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory\nadjusting ipsec.d to /var/lib/neutron/ipsec/df322631-2560-4caf-81ef-e33a58acbb99/etc\npluto: lock file "/var/lib/neutron/ipsec/df322631-2560-4caf-81ef-e33a58acbb99/var/run/pluto.pid" already exists\n'

This case is remaining pluto.pid when submit "ipsec whack --shutdown". I hope it will remove when puluto.pid remain.

OS: CentOS release 6.5 (Final)
OpenSwan:openswan-2.6.32-27.4.el6_5.x86_64

python-neutron-2014.1-18.el6.noarch

[Shogo3 (shogosato)]

I did the following change for workaround.

$ diff /usr/lib/python2.6/site-packages/neutron/services/vpn/device_drivers/ipsec.py /usr/lib/python2.6/site-packages/neutron/services/vpn/device_drivers/ipsec.py.org
379,384d378
<
< #remove pluto.pid, if it remain.
< if ( os.path.isfile(self.pid_path +".pid")):
< LOG.info("PID file is exists. remove :" + self.pid_path + ".pid")
< os.remove(self.pid_path + ".pid")
<

[Andrey Epifanov (aepifanov)]

It seems, that is related bug
https://bugs.launchpad.net/fuel/+bug/1397284

[Numan Siddique (numansiddique)]

Similar bug raised - https://bugs.launchpad.net/neutron/+bug/1394890.
I have marked that as a duplicate

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/147852

[Andrey Epifanov (aepifanov)]

Basically this fix https://review.openstack.org/147852 is workaround.
Pluto daemon doesn't delete his pid file
on CentOS while owner of parent dir is not root for some reason.
It looks like some tricky nuance with unlink function in pluto or specific
CentOS behaviour.

Current situation is the following:
[root@node-4 run]# ll /var/lib/neutron/ipsec/0c9b0941-940f-4b11-8dc6-2f406ce80247/var/run/ -a
total 16
drwxr-xr-x 3 neutron neutron 4096 Jan 16 13:41 .
drwxr-xr-x 3 neutron neutron 4096 Jan 16 09:34 ..
drwxr-xr-x 2 root root 4096 Jan 16 13:40 pluto
srwx------ 1 root root 0 Jan 16 13:41 pluto.ctl
srwxrwxrwx 1 root root 0 Jan 16 13:41 pluto.info
-r--r--r-- 1 root root 6 Jan 16 13:41 pluto.pid

This fix add slash to the end of the path for pluto daemon and we get:
[root@node-4 run]# ll /var/lib/neutron/ipsec/0c9b0941-940f-4b11-8dc6-2f406ce80247/var/run/pluto -a
total 8
drwxr-xr-x 2 root root 4096 Jan 16 13:40 .
drwxr-xr-x 3 neutron neutron 4096 Jan 16 13:41 ..
srwxrwxrwx 1 root root 0 Jan 16 09:45 .ctl
srwxrwxrwx 1 root root 0 Jan 16 09:45 .info
srwxrwxrwx 1 root root 6 Jan 16 09:45 .pid

For now imho it the simplest fix...
If anybody knows how to do it more elegant, please welcome...

[Salvatore Orlando (salvatore-orlando)]

The workaround however produces files like '.ctl' or '.pid' which are at least weird... I think this can be fixed in the workaround as well.

It is also important to understand whether this occurs only in deployment based on devstack or even when deploying from packages. In the former case I would look for a fix in devstack. It is also interesting that the only file which ended up being readonly is the pid file while the others aren't.

The bug was reported with
OS: CentOS release 6.5 (Final)
OpenSwan:openswan-2.6.32-27.4.el6_5.x86_64

Has it been confirmed with versions different from the above ones?

[Andrey Epifanov (aepifanov)]

this happens on CentOS 6.5 and doesn't depens how it was deployed (devstack or packages)

[Erik McCormick (emccormickva)]

This also affects Centos 7 using openstack-neutron-vpn-agent-2014.2.1-1.el7.centos.noarch

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kyle Mestery (<email address hidden>) on branch: master
Review: https://review.openstack.org/147852
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Paul Michali (pcm)]

Is CentOS using OpenSwan or LibreSwan? OpenStack supports only OpenSwan, and we are seeing some differences in operation with LibreSwan (on Fedora). If this is LibreSwan, it may be good to create a LibreSwan driver to address this and other issues.

[Wei Hu (huwei-xtu)]

I'm using rhel7.1 with libreswan 3.12-5.el7 also hit this same issue.

My environment is:
[root@test100 ~]# rpm -qa | grep libreswan
libreswan-3.12-5.el7.x86_64
[root@test100 ~]# uname -r
3.10.0-229.el7.x86_64

[Brent Eagles (beagles)]

This or a similar issue is happening with libreswan. Some relevant details are included in https://bugzilla.redhat.com/show_bug.cgi?id=1245842

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/212673

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/212673
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=00b633d284f0f21aa380fa47a270c612ebef0795
Submitter: Jenkins
Branch: master

commit 00b633d284f0f21aa380fa47a270c612ebef0795
Author: Brent Eagles <email address hidden>
Date: Thu Aug 13 13:35:36 2015 -0230

    Manage cleanup of .ctl/.pid files for LibreSwan

    LibreSwan checks for the presence of pid/ctl files when starting up and
    will error out if they already exist. However, LibreSwan's usage of the
    capabilities library removes the access required to cleanup .ctl and
    .pid files on shutdown if any of the directories in the path are missing
    explicit permissions for root. This is not considered a bug by the
    LibreSwan maintainers, so the LibreSwan driver must work around it by
    checking if it is okay to remove the files and removing them on
    startup. It must also wait for shutdown to complete before restarting
    the daemon on LibreSwanProcess.restart().

    Introduces new configuration for retrying check for process shutdown.

    DocImpact

    Change-Id: I5c215d70c348524979b740f882029f74e400e6d7
    Closes-Bug: #1331502

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/223512

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/223512
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=5996aab16e776dd25c2d3fa3a48d7806f9c0f86f
Submitter: Jenkins
Branch: stable/kilo

commit 5996aab16e776dd25c2d3fa3a48d7806f9c0f86f
Author: Brent Eagles <email address hidden>
Date: Thu Aug 13 13:35:36 2015 -0230

    Manage cleanup of .ctl/.pid files for LibreSwan

    LibreSwan checks for the presence of pid/ctl files when starting up and
    will error out if they already exist. However, LibreSwan's usage of the
    capabilities library removes the access required to cleanup .ctl and
    .pid files on shutdown if any of the directories in the path are missing
    explicit permissions for root. This is not considered a bug by the
    LibreSwan maintainers, so the LibreSwan driver must work around it by
    checking if it is okay to remove the files and removing them on
    startup. It must also wait for shutdown to complete before restarting
    the daemon on LibreSwanProcess.restart().

    Introduces new configuration for retrying check for process shutdown.

    Conflicts:
     neutron_vpnaas/tests/unit/services/vpn/device_drivers/test_ipsec.py

    Closes-Bug: #1331502
    (cherry picked from commit 00b633d284f0f21aa380fa47a270c612ebef0795)
    Change-Id: I5c215d70c348524979b740f882029f74e400e6d7

[Tyler Bishop (tyler-bishop)]

This bug is still present.

pluto: FATAL: lock file "/var/lib/neutron/ipsec/61a5231d-7525-4eb0-8844-09b72f2481b5/var/run/pluto.pid" already exists

2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec [req-61d3a46f-b391-4a6c-a2df-49961bddb562 ffc148881b554cc49670863241cfea43 66e8be2338fe4cc28d815883b021bc61 - - -] Failed to enable vpn process on router 61a5231d-7525-4eb0-8844-09b72f2481b5
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec Traceback (most recent call last):
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec File "/usr/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 258, in enable
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec self.restart()
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec File "/usr/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 370, in restart
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec self.start()
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec File "/usr/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 436, in start
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec self._execute(cmd)
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec File "/usr/lib/python2.7/site-packages/neutron_vpnaas/services/vpn/device_drivers/ipsec.py", line 341, in _execute
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec extra_ok_codes=extra_ok_codes)
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 861, in execute
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec log_fail_as_error=log_fail_as_error, **kwargs)
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 159, in execute
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec raise RuntimeError(m)
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec RuntimeError:
2016-09-10 10:24:38.736 1384161 ERROR neutron_vpnaas.services.vpn.device_drivers.ipsec Command: ['ip', 'netns', 'exec', u'snat-61a5231d-7525-4eb0-8844-09b72f2481b5', 'ipsec', 'pluto', '--ctlbase', u'/var/lib/neutron/ipsec/61a5231d-7525-4eb0-8844-09b72f2481b5/var/run/pluto', '--ipsecdir', u'/var/lib/neutron/ipsec/61a5231d-7525-4eb0-8844-09b72f2481b5/etc', '--use-netkey', '--uniqueids', '--nat_traversal', '--secretsfile', u'/var/lib/neutron/ipsec/61a5231d-7525-4eb0-8844-09b72f2481b5/etc/ipsec.secrets', '--virtual_private', u'%v4:REDACTED/28,%v4:172.16.2.0/24', '--perpeerlogbase', u'/var/lib/neutron/ipsec/61a5231d-7525-4eb0-8844-09b72f2481b5/logs']
2016-09-10 10:24:38.736 1384161 E...

[Vladimir Grujic (hyperbaba)]

pluto process needs to be killed upon service restart