API node incorrectly pins versions for RPC clients and serializers

Bug #1569350 reported by Gorka Eguileor
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Medium
Gorka Eguileor

Bug Description

The way the API node initializes it requires that all nodes are up and running for version pinning to work correctly, otherwise it will set version to None for the RPC clients and the serializers.

This is a problem, also on CI, because you cannot guarantee that the API will start last, and so versions will be incorrect and cause problems in some cases.

I believe this to be the cause for erros like this one: http://logs.openstack.org/21/303021/4/check/gate-grenade-dsvm/cfab219/logs/new/screen-c-api.txt.gz#_2016-04-11_12_39_36_177

Gorka Eguileor (gorka)
Changed in cinder:
assignee: nobody → Gorka Eguileor (gorka)
Changed in cinder:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/304686

Changed in cinder:
status: New → In Progress
Gorka Eguileor (gorka)
tags: added: mitaka-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/307105

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/304686
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=8ea9f67b038868ac4cec32040655498194e768bc
Submitter: Jenkins
Branch: master

commit 8ea9f67b038868ac4cec32040655498194e768bc
Author: Gorka Eguileor <email address hidden>
Date: Wed Apr 13 11:25:15 2016 +0200

    Fix RPC and Object version pinning on API

    The way the API node initializes it requires that all nodes are up and
    running for version pinning to work correctly, otherwise it will set
    version to None for the RPC clients and the serializers same thing
    happens to the scheduler node for the volume rpc.

    This is a problem, also on CI, because you cannot guarantee that the API
    will start last, and so versions will be incorrect and cause problems in
    some cases.

    This patch pins RPC client and serializer to the latest known versions
    when there are no services on the DB. This should only happen on CI,
    devstack, and similar cases where we start everything for the first time
    almost simultaneously.

    An alternative that would work for the API would be to make the RPC
    client lazily instantiated which solves the problem of load order as no
    API requests come in before the other nodes are up and running.

    But it would not solve the problem of the scheduler, as it fires a
    request to volumes to publish_service_capabilities on start and that
    would instantiate the rpc client.

    Closes-Bug: #1569350
    backport-potential: mitaka
    Change-Id: I301c60c4c75cebb4fa6392fe7c9c11cedd1c09c8

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/mitaka)

Reviewed: https://review.openstack.org/307105
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d7af059058ec890fc3b9745d01af349c2a3c2e5f
Submitter: Jenkins
Branch: stable/mitaka

commit d7af059058ec890fc3b9745d01af349c2a3c2e5f
Author: Gorka Eguileor <email address hidden>
Date: Wed Apr 13 11:25:15 2016 +0200

    Fix RPC and Object version pinning on API

    The way the API node initializes it requires that all nodes are up and
    running for version pinning to work correctly, otherwise it will set
    version to None for the RPC clients and the serializers same thing
    happens to the scheduler node for the volume rpc.

    This is a problem, also on CI, because you cannot guarantee that the API
    will start last, and so versions will be incorrect and cause problems in
    some cases.

    This patch pins RPC client and serializer to the latest known versions
    when there are no services on the DB. This should only happen on CI,
    devstack, and similar cases where we start everything for the first time
    almost simultaneously.

    An alternative that would work for the API would be to make the RPC
    client lazily instantiated which solves the problem of load order as no
    API requests come in before the other nodes are up and running.

    But it would not solve the problem of the scheduler, as it fires a
    request to volumes to publish_service_capabilities on start and that
    would instantiate the rpc client.

    Closes-Bug: #1569350
    Change-Id: I301c60c4c75cebb4fa6392fe7c9c11cedd1c09c8
    (cherry picked from commit 8ea9f67b038868ac4cec32040655498194e768bc)

tags: added: in-stable-mitaka
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/cinder 9.0.0.0b1

This issue was fixed in the openstack/cinder 9.0.0.0b1 development milestone.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/cinder 8.1.0

This issue was fixed in the openstack/cinder 8.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.