OCF checker race conditions

Bug #1399907 reported by Vladimir Kuklin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Vladimir Kuklin

Bug Description

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "42"
  build_id: "2014-12-05_22-41-00"
  astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
  fuellib_sha: "3e1f23e042ea8f4f25b6bf956aca17e29e57e708"
  ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
  nailgun_sha: "73a2fa3aa669ca8d95788e80dc052dfa16eca24a"
  fuelmain_sha: "134f70e8332d9a52f17565e23e8b14f092efb735"

For some reason neutron agent script did not create PID file constantly. This lead to the infinite start/stop loop of dhcp agent and failure of deployment.

look into http://jenkins-product.srt.mirantis.net:8080/job/6.0.centos.bvt_1/144/ lrmd.log:

http://paste.openstack.org/show/146030/

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :
Changed in fuel:
status: Confirmed → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/139937

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Vladimir Kuklin (vkuklin)
status: Triaged → In Progress
description: updated
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

AMQP connections checks in OCF monitors (introduced at the early 6.0 dev cycle) appeared to have been introduced a race condition between service start and AMQP connection check. At the heavy loaded environments, the service did not managed to establish AMQP connection while starting and was hit by OCF monitor action reported there are no AMQP connections yet for the service. That is a typical race, so all AMQP checks was reverted from OCF scripts

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/139937
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6d2b4a82790bba5c1da1b4f2b7578f75d9178cd7
Submitter: Jenkins
Branch: master

commit 6d2b4a82790bba5c1da1b4f2b7578f75d9178cd7
Author: Vladimir Kuklin <email address hidden>
Date: Mon Dec 8 11:18:07 2014 +0300

    Remove all AMQP checks from OCF scripts

    Remove AMQP checks from OCF scripts
    as they are introducing race conditions
    and triggering false-positive restart
    actions breaking deployments

    Change-Id: Idcefa8346fd3948897550248d3f6bfa2694ac7d0
    Closes-bug: #1399907

Changed in fuel:
status: In Progress → Fix Committed
tags: added: on-verification
tags: added: ha
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Deployments with HA mode for CentOS and Ubuntu with Neutron GRE and VLAN were succcessful on ISO #56

"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"

Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: ha on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.