[baremetal] cinder-api is down on non-primary after deployment

Bug #1681686 reported by Vladimir Khlyunev
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Michael Polenchuk
Nominated for Ocata by Michael Polenchuk
Newton
In Progress
High
Michael Polenchuk

Bug Description

ISO 1575 (RC2)

On baremetal after any deployment with 3+ controllers cinder-api service on non-primary controllers could be down with error in cinder-api.log [0]

However, openstack is working and restarting this failed services fixes this issue (==workaround).
This reproduces only on baremetal deployment and can be related to specific hardware (or not).
Also this issue is floating, reproduces somewhere in 50-60% of deployments; if it was not reproduced at initial deployment - scale operations works fine.

Snapshot https://product-ci.infra.mirantis.net/view/10.0%20baremetal/job/10.0.baremetal.ubuntu.ceph_for_volumes_images_ephemeral_rados/16/artifact/logs/fail_error_ceph_for_volumes_images_ephemeral_rados-fuel-snapshot-2017-04-10_15-23-35.tar

[0]: ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'cinder.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.cluster_name AS services_cluster_name, services.host AS services_host, services.`binary` AS services_binary, services.updated_at AS services_updated_at, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version AS services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen \nFROM services \nWHERE services.deleted = false AND services.`binary` = %(binary_1)s'] [parameters: {u'binary_1': 'cinder-scheduler'}]

description: updated
tags: added: area-library
Changed in fuel:
milestone: 10.1 → 10.x-updates
status: New → Confirmed
assignee: Fuel Sustaining (fuel-sustaining-team) → Michael Polenchuk (mpolenchuk)
milestone: 10.x-updates → 12.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/455721

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/455721
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=402e66aca9e871e4c5b9c9d669c58db5bcd960f8
Submitter: Jenkins
Branch: master

commit 402e66aca9e871e4c5b9c9d669c58db5bcd960f8
Author: Michael Polenchuk <email address hidden>
Date: Tue Apr 11 17:57:46 2017 +0400

    Split out openstack-cinder task

    openstack-cinder task has been splitted out to avoid race condition
    when syncdb ain't completed (or even not started yet) on primary
    controller before a cinder-api service start up on secondary ones.

    Change-Id: Ic506ba6f5b5908a71eca52a077d8bd6084655dc6
    Closes-Bug: #1681686

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/459664

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/newton)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/newton
Review: https://review.opendev.org/459664
Reason: This repo is retired now, no further work will get merged.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.