Ceph ha deploy failed on OSTF verification run for pacemacker

Bug #1532177 reported by Tatyanka
32
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Vasyl Saienko
8.0.x
Fix Released
High
MOS Ironic
Mitaka
Fix Released
High
Vasyl Saienko

Bug Description

Steps:
1. Deploy ceph ha (3 controllers) with neutron
2. Run ostf

Actual result:
Failed 1 OSTF tests; should fail 0 tests. Names of failed tests:
  - Check pacemaker status (failure) On the controller node-2.test.domain.local, resource p_nova_compute_ironic is active but failed to start (managed).. Please refer to OpenStack logs for more details.
2016-01-08 02:20:34 FAILURE Check pacemaker status (fuel_health.tests.ha.test_pacemaker_status.TestPacemakerStatus.test_check_pacemaker_resources) On the controller node-2.test.domain.local, resource p_nova_compute_ironic is active but failed to start (managed).. Please refer to OpenStack logs for more details. File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
    yield
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 601, in run
    testMethod()
  File "/usr/lib/python2.7/site-packages/fuel_health/tests/ha/test_pacemaker_status.py", line 102, in test_check_pacemaker_resources
    "un" if not resource['managed'] else ""))
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 164, in verify_response_true
    self.fail(message.format(failed_step_msg, msg))
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
Step 4 failed: On the controller node-2.test.domain.local, resource p_nova_compute_ironic is active but failed to start (managed).. Please refer to OpenStack logs for more details.

Pacemaker resources status on the controller node-2.test.domain.local: {'vip__vrouter': {'managed': True, 'started': 1, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local']}, 'sysinfo_node-1.test.domain.local': {'managed': True, 'started': 1, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local']}, 'clone_p_neutron-dhcp-agent': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}, 'clone_p_heat-engine': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}, 'clone_p_neutron-l3-agent': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}, 'clone_p_mysql': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}, 'clone_ping_vip__public': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}, 'clone_p_vrouter': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}, 'clone_p_neutron-metadata-agent': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}, 'vip__public': {'managed': True, 'started': 1, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-2.test.domain.local']}, 'vip__vrouter_pub': {'managed': True, 'started': 1, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local']}, 'master_p_conntrackd': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': ['node-1.test.domain.local'], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}, 'clone_p_haproxy': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}, 'sysinfo_node-2.test.domain.local': {'managed': True, 'started': 1, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-2.test.domain.local']}, 'clone_p_neutron-plugin-openvswitch-agent': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}, 'clone_p_ntp': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}, 'p_nova_compute_ironic': {'managed': True, 'started': 1, 'failed': True, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-2.test.domain.local']}, 'vip__management': {'managed': True, 'started': 1, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local']}, 'master_p_rabbitmq-server': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': ['node-1.test.domain.local'], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}, 'sysinfo_node-3.test.domain.local': {'managed': True, 'started': 1, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-3.test.domain.local']}, 'clone_p_dns': {'managed': True, 'started': 3, 'failed': False, 'stopped': 0, 'active': True, 'master': [], 'nodes': ['node-1.test.domain.local', 'node-2.test.domain.local', 'node-3.test.domain.local']}}.

Expected:
there should not be any ironic processes if there is no ironic node.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "408"
  build_id: "408"
  fuel-nailgun_sha: "9ebbaa0473effafa5adee40270da96acf9c7d58a"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "7ef751bdc0e4601310e85b8bf713a62ed4aee305"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "8bb8c70efc61bcf633e02d6054dbf5ec8dcf6699"
  fuelmenu_sha: "2a0def56276f0fc30fd949605eeefc43e5d7cc6c"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "62573cb2a8aa54845de9303b4a30935a90e1db61"

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
tags: added: area-ironic
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/265308

Changed in fuel:
assignee: MOS Ironic (mos-ironic) → Vasyl Saienko (vsaienko)
status: Confirmed → In Progress
Revision history for this message
Vasyl Saienko (vsaienko) wrote :

ironic-compute has a dependency 'ironic-api' which should be run only if ironic is enabled. For some reason nested conditions doesn't work.

https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/ironic/tasks.yaml#L44

vsaienko@vsaienko-pc:/tmp/trash/fuel-snapshot-2016-01-08_02-21-11/node-1/var/log$ grep MODULAR puppet.log |grep ironic
2016-01-08 00:02:24 +0000 Scope(Class[main]) (notice): MODULAR: ironic/ironic-compute.pp

@nailgun team please check why nested conditions are not working in nailgun.

Changed in fuel:
assignee: Vasyl Saienko (vsaienko) → Fuel Python Team (fuel-python)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/265362

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Vasyl Saienko (vsaienko)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/265308
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=933c90492ac5162cc0d46078da87c997b1d14451
Submitter: Jenkins
Branch: master

commit 933c90492ac5162cc0d46078da87c997b1d14451
Author: vsaienko <email address hidden>
Date: Fri Jan 8 17:17:23 2016 +0200

    Add condition for ironic-compute task

    Launch ironic-compute task only if Ironic is enabled.

    Change-Id: I282f853a124d51f07866c92837cd5bf9aa8a488e
    Closes-Bug: 1532177

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Sergey Shevorakov (sshevorakov) wrote :

This is 8.0 SWARM blocker (3 TCs failed).

tags: added: swarm-blocker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/8.0)

Reviewed: https://review.openstack.org/265362
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6caec879b7591ef9c237ddee1b13dac8f2ca40e6
Submitter: Jenkins
Branch: stable/8.0

commit 6caec879b7591ef9c237ddee1b13dac8f2ca40e6
Author: vsaienko <email address hidden>
Date: Fri Jan 8 17:17:23 2016 +0200

    Add condition for ironic-compute task

    Launch ironic-compute task only if Ironic is enabled.

    Change-Id: I282f853a124d51f07866c92837cd5bf9aa8a488e
    Closes-Bug: 1532177

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

verified 493 iso

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.