cinder-api should run as active/standby

[Bogdan Dobrelya]

Cinder is no A/A ready, see http://lists.openstack.org/pipermail/openstack-dev/2015-August/071125.html
and http://gorka.eguileor.com/a-cinder-road-to-activeactive-ha ,
http://gorka.eguileor.com/cinders-api-races/

Fuel should configure Cinder API to be active standby, like Galera.
HAproxy should be reconfigured to make only one API node serving requests at any time.

Not sure how this bug affects cinder-scheduler.
Update: it affects the volume services as well

[Bogdan Dobrelya (bogdando)]

According to Jay Pipes comments: "I do not think you need to put nova-scheduler as a single resource for pacemaker, no. The state that each scheduler process keeps is kept up to date with calls to ComputeNode.get_all() during each beginning of scheduler select_destinations() loop call.
So there is no stale data anywhere... and when the compute node aborts a resource claim, the resource request is rescheduled to another host. so no, I don't think the scheduler needs to be a single resource for pacemaker :)
for cinder-api, there is no state, therefore it should be active/active as well."

[Bogdan Dobrelya (bogdando)]

Reopening as Cinder is no A/A ready, see http://lists.openstack.org/pipermail/openstack-dev/2015-August/071125.html
and http://gorka.eguileor.com/a-cinder-road-to-activeactive-ha

Removed Nova-scheduler from this bug. If you think Nova is also affected, please submit another bug for it.

[Bogdan Dobrelya (bogdando)]

Not sure if MOS Cinder could and should fix this bug ahead of upstream, so feel free to remove this projected from the list of affected ones.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/211059

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/211059
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2d7258eec7bbedbd5371ce2d10463a12361b4ba3
Submitter: Jenkins
Branch: master

commit 2d7258eec7bbedbd5371ce2d10463a12361b4ba3
Author: Michael Polenchuk <email address hidden>
Date: Mon Aug 10 12:04:10 2015 +0300

    Run cinder-api as active/standby

    Configure Cinder API to be active/standby in order to avoid its
    races (http://gorka.eguileor.com/cinders-api-races/).

    DocImpact: Reflect changes upon reference architecture
    Closes-Bug: #1393331
    Change-Id: Ia91c33eabc531fe366ec88eedd07b7ecfea72466

[Andrey Sledzinskiy (asledzinskiy)]

verified on
{

    "build_id": "299",
    "build_number": "299",
    "release_versions":

{

    "2015.1.0-7.0":

{

    "VERSION":

{

    "build_id": "299",
    "build_number": "299",
    "api": "1.0",
    "fuel-library_sha": "6efc6698dd3d4b07e90a97a49c702128ccc88de7",
    "nailgun_sha": "d590b26dbb09785b8a8b3651b0ef69746fcf9991",
    "feature_groups":

            [
                "mirantis"
            ],
            "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
            "openstack_version": "2015.1.0-7.0",
            "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67",
            "production": "docker",
            "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3",
            "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012",
            "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c",
            "release": "7.0",
            "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"
        }
    }

},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "6efc6698dd3d4b07e90a97a49c702128ccc88de7",
"nailgun_sha": "d590b26dbb09785b8a8b3651b0ef69746fcf9991",
"feature_groups":

    [
        "mirantis"
    ],
    "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
    "openstack_version": "2015.1.0-7.0",
    "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67",
    "production": "docker",
    "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3",
    "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012",
    "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c",
    "release": "7.0",
    "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"

}

[Bogdan Dobrelya (bogdando)]

Here is an update for the cinder-volume (so it seems affected as well):

There are currently concerns over the cinder-volume service ability
to run as a fully active-active service. During the Mitaka timeframe, this is being worked on, see:
   https://blueprints.launchpad.net/cinder/+spec/cinder-volume-active-active-support
   Thus, cinder-volume shall only be running on one of the controller nodes, even if it will be configured on all nodes

[Michael Polenchuk (mpolenchuk)]

For controllers cinder-volume could be Active/Passive under pacemaker control.
How about compute and standalone cinder nodes?

[Bogdan Dobrelya (bogdando)]

AFAIK, not possible w/o the cluster management

[Michael Polenchuk (mpolenchuk)]

Therefore there should be constraint to place cinder-volume service to controllers only?

[slava valyavskiy (slava-val-al)]

Bogdan, could you please elaborate on suitable solution? Is Michael proposal correct?

[Bogdan Dobrelya (bogdando)]

I suggest to fix it only for controllers. For other cases only provide comment to the docs as known issues

[slava valyavskiy (slava-val-al)]

Guys, @bogdando,

It seems that we can only partially cover this issue(to avoid issue on controller nodes using ocf resource). And issue remains for all non-controller nodes and we are going to cover this affected part in our documentation as known issue. But, to avoid this issue on controller nodes we are gonna to introduce new cinder-volume ocf script ( https://github.com/openstack/openstack-resource-agents/blob/master/ocf/cinder-volume ) what was not tested before by us and may introduce some degradation in cinder's functionality. Btw, appropriate fix for cinder service addressing this issue should be landed in Mitaka release.
I propose to completely cover part of bug related to cinder-volume by documentation only and not to introduce tech debt like fixes.

[slava valyavskiy (slava-val-al)]

Also, I propose to create separate bug for cinder-volume service.

[Denis Meltsaykin (dmeltsaykin)]

Closing as Won't Fix as this is clearly a feature not a bug.