Command failed: Execution of '/usr/sbin/pcs resource clear p_vrouter node-1' returned 1: Error: error moving/banning/clearing resource

[Bartłomiej Piotrowski]

api: '1.0'
astute_sha: b93c276bd8cf5018589cf762e1320dca1f14b9ad
auth_required: true
build_id: 2015-05-05_03-45-03
build_number: '377'
feature_groups:
- mirantis
fuel-library_sha: b3fa159e997b2965942a0ad784d07fec3c42619c
fuel-ostf_sha: 8a29f80cffe985dea6a3de78b733d07ec4db93fc
fuelmain_sha: 42297c8034836a33506e80d13bd6a898b6d7d71c
nailgun_sha: 30f53858053ef48920009de3ed7e5b148b9d9504
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 603e659ad4b0f35ba1127155037c1866034dc395
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: b93c276bd8cf5018589cf762e1320dca1f14b9ad
      build_id: 2015-05-05_03-45-03
      build_number: '377'
      feature_groups:
      - mirantis
      fuel-library_sha: b3fa159e997b2965942a0ad784d07fec3c42619c
      fuel-ostf_sha: 8a29f80cffe985dea6a3de78b733d07ec4db93fc
      fuelmain_sha: 42297c8034836a33506e80d13bd6a898b6d7d71c
      nailgun_sha: 30f53858053ef48920009de3ed7e5b148b9d9504
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 603e659ad4b0f35ba1127155037c1866034dc395
      release: '6.1'

2015-05-05 11:31:28 DEBUG Unknown node: node-1
2015-05-05 11:31:28 DEBUG Command failed: Execution of '/usr/sbin/pcs resource clear p_vrouter node-1' returned 1: Error: error moving/banning/clearing resource
2015-05-05 11:31:27 DEBUG Executing '/usr/sbin/pcs resource clear p_vrouter node-1'
2015-05-05 11:31:27 DEBUG Executing '/usr/sbin/cibadmin --patch --sync-call --xml-text (…cut by me…)
2015-05-05 11:31:27 WARNING Could not disable basic service for Pacemaker primitive 'p_vrouter' using 'upstart' provider: Could not find init script or upstart conf file for 'vrouter'

Classically provisioned Ubuntu environment, fuel-qa's deploy_neutron_gre_ha test suite. Ping me via IRC/Slack/e-mail for access to the environment.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/180669

[Bogdan Dobrelya (bogdando)]

Manual testing (ISO #393) of the patch with the steps from the test group:
<<<<<##############################[ ha_flat_scalability ]##############################>>>>>
Check HA mode on scalability

        Scenario:
            1. Create cluster
            2. Add 1 controller node
            3. Deploy the cluster
            4. Add 2 controller nodes
            5. Deploy changes
            6. Run network verification
            7. Add 2 controller nodes
            8. Deploy changes
            9. Run network verification
            10. Run OSTF

failed at the step 8 due to unrelated ntpd error:
2015-05-11T14:36:59 err: [696] Error running RPC method granular_deploy: Failed to execute hook 'shell'.
Looks like we have yet another issue blocking the controller scaling.

[Bogdan Dobrelya (bogdando)]

Still failing to pass the test case ha_flat_scalability with ISO #404 , logs are here https://bugs.launchpad.net/fuel/+bug/1453851/+attachment/4396140/+files/fail_error_ha_flat_scalability-2015_05_12__09_40_56.tar.xz

[Bartłomiej Piotrowski (bpiotrowski)]

Bogdan, #404 iso also failed centos smoke tests.

[Bogdan Dobrelya (bogdando)]

rev. 16 of patch have passed manual scale 1+2+2-2 testing. Nodelist, CIB, corosync.conf and pcs status shows consistent date about member nodes.
But there is another issue was found: 2 removed nodes suddenly reappeared in CIB somethere in between Thu May 14 12:32:43 2015 and before May 14 12:54:54 2015. This can be reproduced if pacemaker service restarted manually

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/180669
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=3b01fecf2e480f996f8c59d118986d98bb634921
Submitter: Jenkins
Branch: master

commit 3b01fecf2e480f996f8c59d118986d98bb634921
Author: Dmitry Ilyin <email address hidden>
Date: Fri May 8 22:17:19 2015 +0300

    Fixes for pcmk_nodes and crm_node

    * Use crm_node -n to get node's hostname
    * Check if node is in cluster in service provider
    * implement pacemaker nodes add in pcmk_nodes
    * compatibility between fqdn and hostname in pcmk_nodes
    * fix new node id and number generation in pcmk_nodes

    Change-Id: I2e40e25df975621506141dfe78efa884255f1642
    Related-Bug: 1451795

[Vladimir Kuklin (vkuklin)]

Fix above works and complete fixes the issue.