Master node deployment failed with error from keystone

Bug #1430348 reported by Tatyanka
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Matthew Mosesohn

Bug Description

http://jenkins-product.srt.mirantis.net:8080/job/6.1.system_test.centos.bonding_ha_one_controller/37/

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "178"
  build_id: "2015-03-09_22-54-44"
  nailgun_sha: "a9a6578a649a2a006c4810b3d0aa6876ac6e8b83"
  python-fuelclient_sha: "4eb787f1ad969bd23c93d192865543dbd45a8626"
  astute_sha: "2d61ee42ec6dae3181d292c7769d32e40d463893"
  fuellib_sha: "62e68af896887ebe18944e6a0a9721e269119ad4"
  ostf_sha: "8df5f2fcdae3bc9ea7d700ffd64db820baf51914"
  fuelmain_sha: "c73b87f7cbc371307a21c368a45a65aa3f4b7a5d"

Deployment of master node failed with next error

http://paste.openstack.org/show/191292/

Keystone failed to start with follow errors:
http://paste.openstack.org/show/191298/

Tags: scale
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
description: updated
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Matthew Mosesohn (raytrac3r)
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

This looks like a duplicate of https://bugs.launchpad.net/fuel/+bug/1428060
What happens is dockerctl sees postgres is ready before it stops the daemon and starts it up again in the foreground. Then it starts keystone and runs db_sync and it fails because of a race condition while restarting postgres. Need to test again with new iso to see if it reproduces.

Changed in fuel:
status: New → Incomplete
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

Confirmed on https://bugs.launchpad.net/fuel/+bug/1430653

We should move the puppet check to the end.

Changed in fuel:
status: Incomplete → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/163327

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/163327
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=00df3e31b017a263245c598741e7f19662efcea7
Submitter: Jenkins
Branch: master

commit 00df3e31b017a263245c598741e7f19662efcea7
Author: Matthew Mosesohn <email address hidden>
Date: Wed Mar 11 12:01:46 2015 +0300

    Run puppet container check after service check

    Checking puppet run is complete at the beginning
    fails because puppet does not start immediately
    on every container in its start.sh script.

    Change-Id: Iacf935984d7bd0f64068e514a413de24e10f7c5c
    Closes-Bug: #1430348

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

seems it is partially fixed, according to keystone migration failed with http://paste.openstack.org/show/191801/ on staging tests

VERSION:
  feature_groups:
    - experimental
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "153"
  build_id: "2015-03-12_07-29-02"
  nailgun_sha: "a720a2da99690eb2d2c19ddc5d739384312a8ac2"
  python-fuelclient_sha: "0f4ca9c2798da34797dd082130d22cac04c998a9"
  astute_sha: "5cdd4ae4037aa29f4c876d441af15cad82f5a6cb"
  fuellib_sha: "e660cbdfc506197e7f79a4bf7caab2990953abe3"
  ostf_sha: "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088"
  fuelmain_sha: "0791400dd8224647ff9a5cb8051ce82b2c8863b1"

details into attached snapshot
failed job http://jenkins-product.srt.mirantis.net:8080/view/6.1/job/6.1.staging.ubuntu.bvt_2/133/

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Changed in fuel:
status: Fix Committed → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/163791

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Leontii Istomin (listomin) wrote :

was reproduced with 6.1-187

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/163791
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=021adfbd50511af8b9e4330cfaf78b69c68a13f4
Submitter: Jenkins
Branch: master

commit 021adfbd50511af8b9e4330cfaf78b69c68a13f4
Author: Matthew Mosesohn <email address hidden>
Date: Thu Mar 12 15:07:32 2015 +0300

    Add check_ready between each container start

    Prevents race condition where keystone starts
    before postgres is ready.

    Change-Id: I38a3d43d6a3f023fbc45e64e85a75619c1bd91ab
    Closes-Bug: #1430348

Changed in fuel:
status: In Progress → Fix Committed
tags: added: scale
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "216"
  build_id: "2015-03-22_22-54-44"
  nailgun_sha: "51974b50c3961be3ed0fdc7859570db2eeb83e9c"
  python-fuelclient_sha: "b223dcaf5fdad2f714cd245958fefe03995d6207"
  astute_sha: "4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9"
  fuellib_sha: "a636c680e3c7d8cc66ed3e03645f38250beb8970"
  ostf_sha: "b4d284e9364e30bf5162975c2ba0be6ca0f14ebd"
  fuelmain_sha: "f52e4442df55a2b62637a2cf4038a24ba6f37b6f"
verified

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.