[pacemaker provider] Deployment of additional/replace controller fails

[Andrey Kirilochkin]

We have to check that we can redeploy controller after fail.

What we do:
1. Removing of failed controller node from fuel (was non-primary controller).
2. Push deploy button
3. Add the additional controller node(the same node).
4. Deploy changes

What we have:
1. Deployment fails on rabbit manifest: http://paste.gnu.su/show/53/

What we expect:
1. Controller would be deployed successful.

Api-version:

{"ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "auth_required": true, "api": "1.0", "nailgun_sha": "9e8d4487f26b0e285e903df754040cf2d952db42", "production": "docker", "fuelmain_sha": "5123de0953d402cf3edc471f831e6b22610f8b6c", "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de", "feature_groups": ["experimental"], "release": "5.1.1", "release_versions": {"2014.1.1-5.1.1": {"VERSION": {"ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "api": "1.0", "nailgun_sha": "9e8d4487f26b0e285e903df754040cf2d952db42", "production": "docker", "fuelmain_sha": "5123de0953d402cf3edc471f831e6b22610f8b6c", "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de", "feature_groups": ["experimental"], "release": "5.1.1", "fuellib_sha": "a52053b8ec79e7e79eb1e6703821697939470889"}}}, "fuellib_sha": "a52053b8ec79e7e79eb1e6703821697939470889"}

[Andrey Kirilochkin (andreika-mail)]

Puppet log:
http://paste.gnu.su/show/54/

[Bogdan Dobrelya (bogdando)]

ISO used for this deployment misses some important fixes for HA intermittent failures:
https://review.openstack.org/#/c/134920/
https://review.openstack.org/#/c/135030/
https://review.openstack.org#/c/135613/

Please try to reproduce again with aforementioned patches

[Andrey Kirilochkin (andreika-mail)]

Snapshot:
https://docs.google.com/document/d/1HCFN8LW2poVF60fNgBALB8iOPuqUsXgQQ33TD8U4WBM/edit?usp=sharing

[Bogdan Dobrelya (bogdando)]

Cannot see deployment issues in logs, but these records looks really bad:
http://paste.openstack.org/show/134822/

[Andrey Kirilochkin (andreika-mail)]

Just replaced the folder /etc/puppet/modules/rabbitmq/lib/ on fuel-master nodes with this https://github.com/dmitryilyin/puppetlabs-rabbitmq/tree/add_waits/lib from branch add_waits.

Everything now OK!

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/136015

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/136036

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/5.1
Review: https://review.openstack.org/136037

[Bogdan Dobrelya (bogdando)]

We have a patch for rabbit retries and a feedback from the reporter, so looks like issue is resolved once the patch merged

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/136036
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4a44aee5d5dfd050c13650c46e708af0a13e6f3e
Submitter: Jenkins
Branch: master

commit 4a44aee5d5dfd050c13650c46e708af0a13e6f3e
Author: Dmitry Ilyin <email address hidden>
Date: Thu Nov 20 19:14:03 2014 +0300

    Backport rabbitmq waits and retries

    Change-Id: Idb4cb73e17bc5035f22e26a5194c5af7ef74414f
    Related-Bug: 1394188

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/136037
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b732ec3a079dafd113689b5593b47b3879b364e3
Submitter: Jenkins
Branch: stable/5.1

commit b732ec3a079dafd113689b5593b47b3879b364e3
Author: Dmitry Ilyin <email address hidden>
Date: Thu Nov 20 19:14:03 2014 +0300

    Backport rabbitmq waits and retries

    Change-Id: Idb4cb73e17bc5035f22e26a5194c5af7ef74414f
    Related-Bug: 1394188

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Ilyin (<email address hidden>) on branch: master
Review: https://review.openstack.org/136015

[Serg Lystopad (slystopad)]

This scenario fails also for me.

What we do:
1) deploy env. All nodes were ready. All OSTF tests were green.
2) check controller which i want to delete, push 'deploy changes'
3) add discovered node as controller, push 'deploy changes'

after this pass we got
1) primary controller was ready, other two were in error state

second try:
1) just push 'deploy changes'

result was all 3 controllers in error state.

What we expect:
1) Controller would be deployed successful.

Diagnostic snapshot:

https://drive.google.com/a/mirantis.com/file/d/0BzqvkqZNKRGdYXJydS0zRkV6dUU/view?usp=sharing

Api-version:

{"ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "auth_required": true, "api": "1.0", "nailgun_sha": "9e8d4487f26b0e285e903df754040cf2d952db42", "production": "docker", "fuelmain_sha": "0fe3db7475b9f8b287c5b59cba94c9a40a8d8101", "astute_sha": "dade74af41d4972fe05a1c16ae1db2a2e60c6715", "feature_groups": ["experimental"], "release": "5.1.1", "release_versions": {"2014.1.1-5.1.1": {"VERSION": {"ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "api": "1.0", "nailgun_sha": "9e8d4487f26b0e285e903df754040cf2d952db42", "production": "docker", "fuelmain_sha": "0fe3db7475b9f8b287c5b59cba94c9a40a8d8101", "astute_sha": "dade74af41d4972fe05a1c16ae1db2a2e60c6715", "feature_groups": ["experimental"], "release": "5.1.1", "fuellib_sha": "a628048a6443e444fe05bf5620a56626f958a1ac"}}}, "fuellib_sha": "a628048a6443e444fe05bf5620a56626f958a1ac"}

Fuel-library:
https://github.com/slystopad/fuel-lib/tree/23-11-2014

rabbitmq/lib already in branch above (took from https://github.com/dmitryilyin/puppetlabs-rabbitmq/tree/add_waits/lib)

[Andrey Kirilochkin (andreika-mail)]

Please reopen this bug

[Miroslav Anashkin (manashkin)]

Dmitry, Please confirm, was this bug fixed or not?
Reopened it back to avoid missing bugs for 5.1.1

[Andrey Kirilochkin (andreika-mail)]

Solution for corosync cluster rebuild : https://review.openstack.org/#/c/137812

Final library could be found here: https://github.com/slystopad/fuel-lib/tree/01-12-2014

[Stanislav Makar (smakar)]

checking 5.1.1

[Stanislav Makar (smakar)]

verifying on 6.0

[Stanislav Makar (smakar)]

doesn't work
{"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "auth_required": true, "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"}}}, "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"}

[Stanislav Makar (smakar)]

before the deleting of one of controllers name was node-2
after the adding again - node-8

as we see the file /etc/astute.yaml isn't present on node-8

/var/log/docker-logs/astute/astute.log:

2014-12-11T11:13:09 debug: [408] 75607a3e-0ea0-4f2c-a097-160e863db79f: MC agent 'execute_shell_command', method 'execute', results: {:sender=>"
8", :statuscode=>0, :statusmsg=>"OK", :data=>{:stdout=>"", :exit_code=>1, :stderr=>"/usr/lib/ruby/1.8/yaml.rb:143:in `initialize': No such file
 or directory - /etc/astute.yaml (Errno::ENOENT)\n\tfrom /usr/lib/ruby/1.8/yaml.rb:143:in `open'\n\tfrom /usr/lib/ruby/1.8/yaml.rb:143:in `load
_file'\n\tfrom -e:1\n"}}
2014-12-11T11:13:09 debug: [408] 75607a3e-0ea0-4f2c-a097-160e863db79f: cmd: ruby -r 'yaml' -e 'y = YAML.load_file("/etc/astute.yaml"); y["nodes"] = YAML.load_file("/tmp/astute.yaml"); File.open("/etc/astute.yaml", "w") { |f| f.write y.to_yaml }'; puppet apply --logdest syslog --debug -e '$settings=parseyaml($::astute_settings_yaml) $nodes_hash=$settings["nodes"] class {"l23network::hosts_file": nodes => $nodes_hash }'
                                               stdout:
                                               stderr: /usr/lib/ruby/1.8/yaml.rb:143:in `initialize': No such file or directory - /etc/astute.yaml (Errno::ENOENT)
        from /usr/lib/ruby/1.8/yaml.rb:143:in `open'
        from /usr/lib/ruby/1.8/yaml.rb:143:in `load_file'
        from -e:1

                                               exit code: 1
2014-12-11T11:13:09 warning: [408] 75607a3e-0ea0-4f2c-a097-160e863db79f: Fail to update /etc/hosts, check the debugging output for node 8 for details

[Dmitry Borodaenko (angdraug)]

So what's the status of this bug in 6.0? Is it Won't Fix or Invalid?

[Dmitry Borodaenko (angdraug)]

And is the fix confirmed for 5.1.1?

[Irina Povolotskaya (ipovolotskaya)]

The same question about 6.0: is it fixed? If not, would you provide any workaround?

[Andrey Kirilochkin (andreika-mail)]

I have not tested this action on 6.0. Vova Kuklin said that this should not be reproduced on 6.0.

[Vladimir Kuklin (vkuklin)]

Issue in the description is invalid for 6.0 and 6.1. Other comments are related to HA improvements, such as corosync upgrade, which will get to 6.1

[Oleksandr Liemieshko (oliemieshko)]

I checked this bug in 6.0. It not fixed

[Denis Klepikov (dklepikov)]

Remove-add controller doesn't work
Baremetal lab 5 nodes

[Denis Klepikov (dklepikov)]

Remove-add controller doesn't work
Baremetal lab 5 nodes

[Bogdan Dobrelya (bogdando)]

The status of this issue looks lost for 6.0 and 6.1 targets, QA team could you please assist with reproducing?

[Anastasia Palkina (apalkina)]

Bug is not reproduced on ISO #156

"build_id": "2015-03-02_22-54-44", "ostf_sha": "103d6cf6badd57b791cfaf4310ec8bd81c7a8a46", "build_number": "156", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-02_22-54-44", "ostf_sha": "103d6cf6badd57b791cfaf4310ec8bd81c7a8a46", "build_number": "156", "api": "1.0", "nailgun_sha": "5da9e633204fa0affb86208ea2989500881f8ec2", "production": "docker", "python-fuelclient_sha": "3ebfa9c14a192d0298ff787526bf990055a23694", "astute_sha": "1be5b9b827f512d740fe907c7ff72486d4030938", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "baf24424a4e056c6753913de5f8c94851903f718", "fuellib_sha": "92d7c3021d573952b65cd5705a6c690cd24d9de3"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "5da9e633204fa0affb86208ea2989500881f8ec2", "production": "docker", "python-fuelclient_sha": "3ebfa9c14a192d0298ff787526bf990055a23694", "astute_sha": "1be5b9b827f512d740fe907c7ff72486d4030938", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "baf24424a4e056c6753913de5f8c94851903f718", "fuellib_sha": "92d7c3021d573952b65cd5705a6c690cd24d9de3"

1. Deploy HA, Ubuntu with 3 controllers and 1 compute
2. After starting deployment on primary controller I nulled /etc/controller.yaml on second controller
3. Deployment has failed with error on second controller
4. I deleted this 'error' node
5. Click button "Deploy Changes"
6. Node has deleted successfully
7. Waited until node becomes bootstrap
8. Add this node again to environment as a controller
9. Click button "Deploy Changes"
10. Deployment was successful

[Maksym Strukov (unbelll)]

{"build_id": "2015-04-10_04-14-22", "ostf_sha": "4bda5bbf9ea033189f16518032c063d43e4d0e5c", "build_number": "301", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-10_04-14-22", "ostf_sha": "4bda5bbf9ea033189f16518032c063d43e4d0e5c", "build_number": "301", "api": "1.0", "nailgun_sha": "d6e351189666e8afa01003e643e63216ef7abd26", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c", "astute_sha": "5041b2fb508e6860c3cb96474ca31ec97e549e8b", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "2ca546b86e651d5638dbb1be9bae44b86c84a893", "fuellib_sha": "e9c3ba332b05120c967b20260c7b223afc1b4f1a"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "d6e351189666e8afa01003e643e63216ef7abd26", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c", "astute_sha": "5041b2fb508e6860c3cb96474ca31ec97e549e8b", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "2ca546b86e651d5638dbb1be9bae44b86c84a893", "fuellib_sha": "e9c3ba332b05120c967b20260c7b223afc1b4f1a"}

What we do:
1) deploy env. All nodes were ready.
2) Remove one of controllers push 'deploy changes'
3) add discovered node as controller, push 'deploy changes'

after this pass we got
1) primary and one of controller were ready, added controller was in error state

second try:
1) just push 'deploy changes'

result was 2 non-primary controllers in error state (re-added was fine but marked as failed with failed controlle)

[Aleksandr Didenko (adidenko)]

Tested on 6.1 #315 ISO:
    "api": "1.0",
    "astute_sha": "36a3d54baf86480326866279778b867086aa4b05",
    "auth_required": true,
    "build_id": "2015-04-16_03-32-26",
    "build_number": "315",
    "feature_groups": [
        "mirantis"
    ],
    "fuellib_sha": "ab6dbba825a5e54f3183ed8582921a0328b211c7",
    "fuelmain_sha": "d3aa72d814c3f18799b566363004a22591161cef",
    "nailgun_sha": "24b56283fb4caaba6b6414056dad30e503a7c30f",
    "openstack_version": "2014.2-6.1",
    "ostf_sha": "90f232a25d10f98db9d5531a3b133319529f57aa",
    "production": "docker",
    "python-fuelclient_sha": "ff186de358cbda489a320723de47a9e61ae23364",
    "release": "6.1",

New controller and all existing nodes are in 'ready' state, no errors in puppet log:
http://paste.openstack.org/show/204180/

OpenStack is working, OSTF passes. But I can see the following issues in corosync cluster:
http://paste.openstack.org/show/204181/

According to the snapshot provided above, there was also some problem with adding new node to corosync cluster. Investigating it further.

[Aleksandr Didenko (adidenko)]

Some further details on this issue:

1) When I remove node-4 from the environment, it still stays in corosync cluster and it's marked as OFFLINE.
2) Then, when I add it back, it gets new name (in my case node-6) but it gets exactly the same IP address as before.
3) When I deploy changes with newly added node-6, I get timeouts on this step in puppet on existing secondary controller:

2015-04-20 13:49:09 +0000 Exec[stop-pacemaker](provider=posix) (debug): Executing 'service pacemaker stop || true'
2015-04-20 13:49:09 +0000 Puppet (debug): Executing 'service pacemaker stop || true'

Here's what I see in pacemaker log:
http://paste.openstack.org/show/204883/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/176412

[Aleksandr Didenko (adidenko)]

OK. In order to fix this we need to correctly remove controllers from environment:
- remove it from corosync cluster in runtime without restarts of corosync/pacemaker, something like this should work for corosync-2:
  corosync-cmapctl -d nodelist.node.2.nodeid
  corosync-cmapctl -d nodelist.node.2.ring0_addr
  cibadmin --delete --obj_type nodes --crm_xml '<node uname="node-3.test.domain.local"/>'
  cibadmin --delete --obj_type status --crm_xml '<node_state uname="node-3.test.domain.local"/>'
- remove its IPs from memcache/rabbitmq hosts in configs

So solution here would be:
- develop a new resource "corosync_nodes" (or "cs_nodes") that will take care of runtime addition/removal of controllers from corosync cluster and CIB
- run deploy on all nodes whenever we remove controllers - with new resource it will take care of corosync cluster configuration and quorum policy

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/176799

[Bogdan Dobrelya (bogdando)]

@Aleksandr, you should not remove a DC node runtime w/o stopping the corosync service! Please see http://clusterlabs.org/doc/en-US/Pacemaker/1.0/html/Pacemaker_Explained/s-node-delete.html

[Bogdan Dobrelya (bogdando)]

Anyway, if you tested this and it works, just make sure new DC will be elected when you remove the current one by corosync-cmapctl -d & cibadmin --delete

[Bogdan Dobrelya (bogdando)]

Let me clarify the "remove it from corosync cluster in runtime without restarts of corosync/pacemaker"

A node will be removed from corosync cluster *only* after erased by Fuel orchestration, so there will be no corosync service running already. So that would allow us to not care about stopping the corosync service prior to do the node removal from the corosync cluster.

[Vladimir Kuklin (vkuklin)]

This issue happened due to change of cluster stack version to corosync 2.x invalid for pre-6.1 versions

[Bogdan Dobrelya (bogdando)]

Raised to critical as this bug have many duplicates

[Bogdan Dobrelya (bogdando)]

Addressed by https://review.openstack.org/#/c/176412

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/176412
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=cf204a9971825d280b9d8bd1ecc1f78433f48bb6
Submitter: Jenkins
Branch: master

commit cf204a9971825d280b9d8bd1ecc1f78433f48bb6
Author: Dmitry Ilyin <email address hidden>
Date: Wed Apr 22 20:56:37 2015 +0300

    Pacemaker fixes for controller add/remove

    * Location constraint should be present for
      a pacemaker service to be started even if
      this service is actually started.

    * pcmk_nodes resource to add and remove
      corosync nodes

    Change-Id: I97035e633c33f94519b66526268620f9cd6cc958
    Closes-Bug: 1394188

[Vladimir Kuklin (vkuklin)]

Nailgun fix is here:

https://review.openstack.org/176799

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/176799
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=b2a27681408350a48598a20cf6c5da6ead679d64
Submitter: Jenkins
Branch: master

commit b2a27681408350a48598a20cf6c5da6ead679d64
Author: Alexander Kislitsky <email address hidden>
Date: Wed May 6 18:48:32 2015 +0300

    Controllers deployment on controller removal

    On removing of controller(s) we should deploy other controller(s).
    Deployment of controllers added into ApplyChangesTaskManager
    and into NodeDeletionTaskManager.

    Change-Id: I74f07588b0d6baf4a5bd0599d2a0ee0fe7a6a765
    Partial-Bug: #1394188

[Alexander Kislitsky (akislitsky)]

Nailgun part is merged. Moving bug to Fix Commited

[Sergey Novikov (snovikov)]

I was trying to reproduce this bug on fuel-6.1-408-2015-05-12_17-03-01.iso .

I had virtual environment(CentOS) in HA mode (3 controllers + 2 compute) with VLAN segmentation deployed by fuel-devops tool.

I reset this environment and started to execute steps described in comment #31 (starting with step #2). Step #6 failed, node was not deleted from cluster. The following error was found in astute.log (for more details, please, see attached files):

2015-05-15 12:43:32 ERR [689] Error running RPC method remove_nodes: A JSON text must at least contain two octets!

Also, I tried to reset environment again, but i caught message "A server error has occurred. Please contact the administrators". Similar message is shown when I try to delete environment

[Sergey Novikov (snovikov)]

The behavior described above(see comment #49) is not reproduced on fuel-6.1-450-2015-05-22_03-56-41.iso

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/6.1
Review: https://review.openstack.org/194961

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/194961
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=0e26e7d7cc153d179ec34985645dd23cdd239ddb
Submitter: Jenkins
Branch: stable/6.1

commit 5cc5f0c643aebecaf3bf4580535a3ea7c3334a6c
Author: Mike Scherbakov <email address hidden>
Date: Tue Jun 23 13:43:35 2015 -0700

    Removed streamlined patching backend pieces

    Change-Id: I955e76ccdbd12a9145f4e9b689f80bdf9fcaf929

commit 563c4b5c78ebfcb1f4f91047c2919f6270f9a1d4
Author: Mike Scherbakov <email address hidden>
Date: Tue Jun 23 13:30:30 2015 -0700

    Removed outdated patching guide

    Change-Id: I76180c277789ade9c5ebedd19fe2092847c0b7d9

commit 8d120c14bec1ab41d448683ad146a3053a57c4ee
Author: Irina Povolotskaya <email address hidden>
Date: Tue Jun 23 19:59:11 2015 +0300

    Add dual hypervisor ref arch into 6.1 docs

    Change-Id: I900c24c9de878eafadbfc995aa879b7f55737fac

commit feebd1592d3305b64bbdfd0bc5fe108190aef120
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 18:38:17 2015 +0300

    [OPs guide] Running Ceilometer section edits

    1. conf file extract is updated
    2. note is updated

    Closes-bug: 1467817
    Change-Id: I0217e164108e0ba6c1397045a5e57d13ff429223

commit 44a93f9dead7511a3461ec35248dbb689c81eafd
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 18:04:40 2015 +0300

    [RN6_1] Final changes

    1. capitalization
    2. 2014.2 to 2014.2.2
    3. general improvements

    Change-Id: I45057e90c90550559f66bc67ccdf97a559fd9000

commit bb41389cae58084285688853281516b659686422
Author: evkonstantinov <email address hidden>
Date: Tue Jun 23 16:45:35 2015 +0300

    Update patching decription

    Update patching description with
    the standard Linux commands.

    Change-Id: Ia1a8346639c468fdfce15a11d2430bf3a4731244

commit bf3018fae3f2e564413d33aba6cdebf8868f0b4e
Author: OlgaGusarenko <email address hidden>
Date: Tue Jun 23 15:55:49 2015 +0300

    [RN6_1] Clean up

    1. Rearranges sections
    2. Improves RST
    3. Changes titles order

    Change-Id: I6110bf515667d3d6ba08ad35ff5d593dbc96641e

commit 1c7e4457808e8f2d6c56fdf31252170972e444b9
Author: Maria Zlatkova <email address hidden>
Date: Tue Jun 23 15:26:28 2015 +0300

    Replaces VBOX screenshots

    This patch:
    - replaces VBOX screenshots
    - changes the link for Download Mirantis VirtualBox scripts
     to https://docs.mirantis.com/openstack/fuel/fuel-master/#downloads

    Change-Id: I58dede960c5c3355d39b07ff44b757403f6af02c
    Closes-Bug: #1467872

commit 0a568bf53fc0e25d1d692d5d74b4a7b4d983bbcc
Author: evkonstantinov <email address hidden>
Date: Tue Jun 23 14:01:55 2015 +0300

    6.1 --separate repos

    change wording and add links to the
    separate repos feature.

    Change-Id: Ib5d0778a0d8f1534f79ed2f553574cb69a3150b0

commit 95a188b21cbdd064d92696b7920e6a0105fe0c56
Author: Maria Zlatkova <email address hidden>
Date: Tue Jun 23 12:07:28 2015 +0300

    Corrects the output 'pcs status'

    Changes the example outputs to appropriate ones.

    Change-Id: Ib6d83...

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 5.1.1-updates and 6.0-updates because of size of the fix (688 LOC) and no deliver channel for fuel-library fixes