Incorrect remote logs symlinks on master node lead to deployment failure after several redeployments

[Dmitry Belyaninov]

The same problem was fixed here https://bugs.launchpad.net/fuel/+bug/1506112

    Scenario:
        1. Revert snapshot
        2. Select the primary controller
        3. Reinstall the controller
        4. Run network verification
        5. Run OSTF
        6. Verify that the hostname is not changed on reinstallation
           of the node
        7. Verify that the primary-controller role is not migrated on
           reinstallation of the node

Failed job:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ready_node_reinstallation/157/testReport/(root)/reinstall_single_primary_controller_node/reinstall_single_primary_controller_node/

app.log
2016-12-26 23:58:52.520 DEBUG [7f42cb323880] (logs) Preparing syslog directories for node: node-1.test.domain.local
2016-12-26 23:58:52.520 DEBUG [7f42cb323880] (logs) prepare_syslog_dir prefix=/var/log/remote/
2016-12-26 23:58:52.523 DEBUG [7f42cb323880] (logs) prepare_syslog_dir old=/var/log/remote/10.109.0.3
2016-12-26 23:58:52.524 DEBUG [7f42cb323880] (logs) prepare_syslog_dir new=/var/log/remote/node-1.test.domain.local
2016-12-26 23:58:52.524 DEBUG [7f42cb323880] (logs) prepare_syslog_dir bak=/var/log/remote/node-1.test.domain.local.bak
2016-12-26 23:58:52.524 DEBUG [7f42cb323880] (logs) prepare_syslog_dir links=['/var/log/remote/10.109.0.3']
2016-12-26 23:58:52.524 DEBUG [7f42cb323880] (logs) New /var/log/remote/node-1.test.domain.local already exists. Trying to backup
2016-12-26 23:58:52.525 DEBUG [7f42cb323880] (logs) Old /var/log/remote/10.109.0.3 exists and it is link. Trying to unlink
2016-12-26 23:58:52.525 DEBUG [7f42cb323880] (logs) Creating /var/log/remote/node-1.test.domain.local
2016-12-26 23:58:52.525 DEBUG [7f42cb323880] (logs) Creating symlink /var/log/remote/10.109.0.3 -> /var/log/remote/node-1.test.domain.local
2016-12-26 23:58:52.552 ERROR [7f42cb323880] (manager) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/task/manager.py", line 61, in _call_silently
    to_return = method(task, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nailgun/task/task.py", line 655, in message
    logs_utils.prepare_syslog_dir(node)
  File "/usr/lib/python2.7/site-packages/nailgun/utils/logs.py", line 90, in prepare_syslog_dir
    os.symlink(objects.Node.get_node_fqdn(node), l)
OSError: [Errno 17] File exists

full log:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ready_node_reinstallation/157/artifact/logs/fail_error_reinstall_single_primary_controller_node-fuel-snapshot-2016-12-26_23-58-57.tar

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/418991

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/419467

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/419468

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418991
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=46c859acc0a7a2d6229de4454d6ecc0addc34655
Submitter: Jenkins
Branch: master

commit 46c859acc0a7a2d6229de4454d6ecc0addc34655
Author: Alexander Kislitsky <email address hidden>
Date: Wed Jan 11 17:53:10 2017 +0300

    Logs symlinks creation fixed

    We had a race on the old symlink recreation. After old symlink
    removing rsyslog creates a dir with old name and after that
    symlink creation is failed due to 'File exists' error. As fix
    we are stopping the rsyslog for removing old link as well as
    it was done for removing old dir.

    Change-Id: I7fa969c3495f7e15a3b9f44ee9eaccca4106d5c1
    Closes-Bug: #1652771

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419468
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=32641ffabfda61f55ef43f81753f356e5b115b23
Submitter: Jenkins
Branch: stable/newton

commit 32641ffabfda61f55ef43f81753f356e5b115b23
Author: Alexander Kislitsky <email address hidden>
Date: Wed Jan 11 17:53:10 2017 +0300

    Logs symlinks creation fixed

    We had a race on the old symlink recreation. After old symlink
    removing rsyslog creates a dir with old name and after that
    symlink creation is failed due to 'File exists' error. As fix
    we are stopping the rsyslog for removing old link as well as
    it was done for removing old dir.

    Change-Id: I7fa969c3495f7e15a3b9f44ee9eaccca4106d5c1
    Closes-Bug: #1652771

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419467
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=550e355e0d254b02658147f01d5b639037afee1b
Submitter: Jenkins
Branch: stable/mitaka

commit 550e355e0d254b02658147f01d5b639037afee1b
Author: Alexander Kislitsky <email address hidden>
Date: Wed Jan 11 17:53:10 2017 +0300

    Logs symlinks creation fixed

    We had a race on the old symlink recreation. After old symlink
    removing rsyslog creates a dir with old name and after that
    symlink creation is failed due to 'File exists' error. As fix
    we are stopping the rsyslog for removing old link as well as
    it was done for removing old dir.

    Change-Id: I7fa969c3495f7e15a3b9f44ee9eaccca4106d5c1
    Closes-Bug: #1652771

[Dmitry Belyaninov (dbelyaninov)]

Passed on few 9.x runs. State changed to Fix Released

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-web 11.0.0.0rc1 release candidate.

[Ilya Bumarskov (ibumarskov)]

Verified on Fuel 10 build #1575
https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.ready_node_reinstallation/233/testReport/(root)/node_reinstallation_env/