Endless restart of swift-object-reconstructor and swift-container-reconciler

[Ilya Chukhnakov]

Detailed bug description:
The swift-object-reconstructor and swift-container-reconciler services fail to start and restart endlessly after rebooting the cluster.

Steps to reproduce:
1. deploy a new environment from the Fuel WebUI
2. after all nodes are up and services have settled, shut down the environment and the fuel master and then start it again (https://docs.mirantis.com/openstack/fuel/fuel-8.0/operations.html#howto-shut-down-the-whole-cluster)
3. check dmesg for the following repeating messages on the controller nodes:
   swift-object-reconstructor main process (5143) terminated with status 1
   swift-object-reconstructor main process ended, respawning
   swift-container-reconciler main process (8570) terminated with status 1
   swift-container-reconciler main process ended, respawning

Expected results:
container-reconciler and object-reconstructor either start successfully
or do not start at all.

Actual result:
The swift-object-reconstructor and swift-container-reconciler services fail to start and are being restarted indefinitely.

When started manually, produce the following errors:
swift@node-1$ /usr/bin/python /usr/bin/swift-container-reconciler /etc/swift/container-server.conf
Unable to find container-reconciler config section in /etc/swift/container-server.conf

swift@node-1$ /usr/bin/python /usr/bin/swift-object-reconstructor /etc/swift/object-server.conf
Unable to find object-reconstructor config section in /etc/swift/object-server.conf

Workaround:
echo '[container-reconciler]' >> /etc/swift/container-server.conf
echo '[object-reconstructor]' >> /etc/swift/object-server.conf

Impact:
1. container-reconciler and object-reconstructor are not working
2. increased CPU consumption due to OS trying to restart services

Description of the environment:
MOS 9.0 build 372
or
MOS 9.0-mos iso 475

OpenStack Release: Mitaka on Ubuntu 14.04
Compute: QEMU
Network: Neutron with tunneling segmentation
Storage Backends: Cinder LVM over iSCSI for volumes
No additional projects/plugins

Total Nodes: 5
Controller + Cinder: 3
Compute: 2

[Oleksiy Molchanov (omolchanov)]

I was not able to reproduce your case. These services are enabled on boot, but actually are stopped after boot. Seems that the situation you described happens only if you run 'service $service_name start'

[Ilya Chukhnakov (ichukhnakov)]

I've successfully reproduced this issue on a more recent fuel-9.0-mos-475-2016-06-12_06-18-00.iso with a simple 1 controller + 1 compute environment.

These services indeed do not start on a freshly deployed environment, but they do appear once you 'poweroff' the nodes and start them again.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/332152

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/332152
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=cd059f1cc27e18fb04058567f7499f130d84de14
Submitter: Jenkins
Branch: master

commit cd059f1cc27e18fb04058567f7499f130d84de14
Author: Oleksiy Molchanov <email address hidden>
Date: Tue Jun 21 16:49:04 2016 +0300

    Disable unneeded swift services

    This patch totally disable swift-object-reconstructor and
    swift-container-reconciler as they are not used, but are
    enabled by default. After reboot they are trying to start
    in a loop, consuming resources, but fail due to missing
    config-files.

    Change-Id: I7eaddec8d8e5e10dabbf3f44c042517f70a4cee3
    Closes-Bug: 1587047

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/340261

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/340261
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0fc960be9dd8595f72b99aaaa7693567386feffa
Submitter: Jenkins
Branch: stable/mitaka

commit 0fc960be9dd8595f72b99aaaa7693567386feffa
Author: Oleksiy Molchanov <email address hidden>
Date: Tue Jun 21 16:49:04 2016 +0300

    Disable unneeded swift services

    This patch totally disable swift-object-reconstructor and
    swift-container-reconciler as they are not used, but are
    enabled by default. After reboot they are trying to start
    in a loop, consuming resources, but fail due to missing
    config-files.

    Change-Id: I7eaddec8d8e5e10dabbf3f44c042517f70a4cee3
    Closes-Bug: 1587047
    (cherry picked from commit cd059f1cc27e18fb04058567f7499f130d84de14)

[Tatyana Kuterina (tkuterina)]

Verified on 9.1 snapshot #76

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-library9.0-9.0.0-1.mos8495.noarch
 rubygem-astute-9.0.0-1.mos753.noarch
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8748.noarch
 shotgun-9.0.0-1.mos90.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-openstack-metadata-9.0.0-1.mos8748.noarch
 python-packetary-9.0.0-1.mos142.noarch
 nailgun-mcagents-9.0.0-1.mos753.noarch
 fuel-utils-9.0.0-1.mos8495.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-misc-9.0.0-1.mos8495.noarch
 fuel-ostf-9.0.0-1.mos938.noarch
 fuel-notify-9.0.0-1.mos8495.noarch
 fuel-nailgun-9.0.0-1.mos8748.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-mirror-9.0.0-1.mos142.noarch
 fuel-migrate-9.0.0-1.mos8495.noarch

FUEL_QA_COMMIT=bfb750898b0f5ef196eb0c8a295cc29279487ade
UBUNTU_MIRROR_ID=ubuntu-2016-07-30-170657
CENTOS_MIRROR_ID=centos-7.2.1511-2016-05-31-083834
MOS_UBUNTU_MIRROR_ID=9.0-2016-07-29-200321
MOS_CENTOS_OS_MIRROR_ID=os-2016-06-23-135731
MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2016-07-29-200321
MOS_CENTOS_UPDATES_MIRROR_ID=updates-2016-06-23-135916
MOS_CENTOS_HOLDBACK_MIRROR_ID=holdback-2016-06-23-140047
MOS_CENTOS_HOTFIX_MIRROR_ID=hotfix-2016-07-18-162958
MOS_CENTOS_SECURITY_MIRROR_ID=security-2016-06-23-140002

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/371945

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/371983

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/371983
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c126e6bbae508a1acf776afff80cf2de5297f442
Submitter: Jenkins
Branch: master

commit c126e6bbae508a1acf776afff80cf2de5297f442
Author: Maksim Malchuk <email address hidden>
Date: Sat Sep 17 18:09:57 2016 +0300

    Don't stop non-existent services for UCA in Trusty

    Workaround the fix added by LP#1587047 [1] because UCA version of the
    'swift-object' and 'swift-container' packages in Trusty doen't have
    the corresponding services.

    [1] I7eaddec8d8e5e10dabbf3f44c042517f70a4cee3

    Change-Id: I37745c752b432d00d89c3068873dfe6513ddd2bc
    Closes-Bug: #1610947
    Related-Bug: #1587047
    Signed-off-by: Maksim Malchuk <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/371945
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=26e41ea93f880e4a3d565a3e9557299cc7daab70
Submitter: Jenkins
Branch: stable/mitaka

commit 26e41ea93f880e4a3d565a3e9557299cc7daab70
Author: Maksim Malchuk <email address hidden>
Date: Sat Sep 17 18:09:57 2016 +0300

    Don't stop non-existent services for UCA in Trusty

    Workaround the fix added by LP#1587047 [1] because UCA version of the
    'swift-object' and 'swift-container' packages in Trusty doen't have
    the corresponding services.

    [1] I7eaddec8d8e5e10dabbf3f44c042517f70a4cee3

    Change-Id: I37745c752b432d00d89c3068873dfe6513ddd2bc
    Closes-Bug: #1610947
    Related-Bug: #1587047
    Signed-off-by: Maksim Malchuk <email address hidden>