Deploy fails after removing nodes.

[Olesia Tsvigun]

Deploy has was failed after removing nodes.
Fuel ISO#525
OS CentOS

Steps to reproduce:
1. Install Zabbix plugin.
2. In Wizard fill all required options:
- add name 'cvDeploy'of env and select default release version with OS 'Juno'
- as hypervisor type: default QEMU radiobutton
- network setup : Netron with Vlan segmetation( by default )
- storage backends: default
- additional services: all by default
create cluster

3. Add following nodes:
3 controllers
1 cinder
1 compute

4.Configure interfaces on the nodes:
eth0 - admin(PXE)
eth1 - public
eth2 - management
eth3 - private
eth4 - storage

5. Networks tab:
disable 'use vlan tag' for Public, Storage and Mangment

6.Settings tab
enable plugin 'Zabbix'
change default password

7. Deploy cluster.
8. Remove 2 controllers, cinder and compute nodes from cluster.
9. Deploy cluster

Actual result
Deploy has failed with error:
[649] Error running RPC method granular_deploy: Failed to execute hook 'zabbix_monitoring-1.0.0' Puppet run failed. Check puppet logs for details

from astute logs on master:
http://paste.openstack.org/show/321292/

from puppet logs on controller:
http://paste.openstack.org/show/321293/

Expected result:
Cluster should be deployed.

Fuel ISO version:
api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'

[Alexander Zatserklyany (zatserklyany)]

My attempt of reproducing the bug according your steps lead to the same result, but due to error with virtual_ips.
I didn't find any error records connected to zabbix plugin
http://paste.openstack.org/show/321360/
http://paste.openstack.org/show/321364/

[Irina Povolotskaya (ipovolotskaya)]

Alexander, is the bug related to the Plugin Framework?
If yes, could you assign it to Evgeniy Li to make sure that's not at Zabbix side?

[Tatyana Dubyk (tdubyk)]

I've reproduced this bug again too with error:
"Deployment has failed. Method granular_deploy. Failed to execute hook 'zabbix_monitoring-1.0.0' Puppet run failed. Check puppet logs for details"

configuration in Wizard:
CentOS 6.5
KVM
Neutron vlan
other settings by default

Configure interfaces on the nodes:
eth0 - admin(PXE)
eth1 - public
eth2 - management
eth3 - private
eth4 - storage

Network settings:
all by default

Env with nodes:
1 controller
1 compute
1 cinder

Zabbix settings: by default

steps to reproduce:
1. install plugin
2. create env with options above
3. check network connectivity
4. perform deploy

actual result: deploy is failed with error "Deployment has failed. Method granular_deploy. Failed to execute hook 'zabbix_monitoring-1.0.0' Puppet run failed. Check puppet logs for details"

expected result: deploy must be successfully finished

[Tatyana Dubyk (tdubyk)]

[root@nailgun ~]# fuel --fuel-version

api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'
==============================================
[root@nailgun ~]# fuel nodes list

id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|----------|------------------|---------|-------------|-------------------|------------|---------------|--------|---------
9 | error | Untitled (94:e6) | 4 | 10.109.5.11 | 64:9c:44:02:94:e6 | cinder | | True | 4
8 | error | Untitled (bc:4d) | 4 | 10.109.5.10 | 64:02:95:a0:bc:4d | compute | | True | 4
11 | error | Untitled (ba:77) | 4 | 10.109.5.13 | 64:ae:b4:a8:ba:77 | controller | | True | 4

================================================
заббикс сервер установился на контроллер

[root@nailgun ~]# ssh node-11
Warning: Permanently added 'node-11' (RSA) to the list of known hosts.
Last login: Fri Jun 26 16:15:34 2015 from 10.109.5.2
[root@node-11 ~]# crm status
Last updated: Fri Jun 26 16:16:43 2015
Last change: Fri Jun 26 14:10:29 2015
Stack: corosync
Current DC: node-11.test.domain.local (11) - partition with quorum
Version: 1.1.11-97629de
1 Nodes configured
17 Resources configured

Online: [ node-11.test.domain.local ]

 Clone Set: clone_p_vrouter [p_vrouter]
     Started: [ node-11.test.domain.local ]
 vip__management (ocf::fuel:ns_IPaddr2): Started node-11.test.domain.local
 vip__public_vrouter (ocf::fuel:ns_IPaddr2): Started node-11.test.domain.local
 vip__management_vrouter (ocf::fuel:ns_IPaddr2): Started node-11.test.domain.local
 vip__public (ocf::fuel:ns_IPaddr2): Started node-11.test.domain.local
 Clone Set: clone_p_haproxy [p_haproxy]
     Started: [ node-11.test.domain.local ]
 Clone Set: clone_p_dns [p_dns]
     Started: [ node-11.test.domain.local ]
 Clone Set: clone_p_mysql [p_mysql]
     Started: [ node-11.test.domain.local ]
 Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     Masters: [ node-11.test.domain.local ]
 Clone Set: clone_p_opensta...

[Alexander Zatserklyany (zatserklyany)]

I've reproduced the bug without zabbix plugin.

Steps to reproduce:
1. Deploy environment --group=deploy_ha_vlan (I used variable OPENSTACK_RELEASE=centos)
2. Remove 2 controllers.
3. Deploy cluster

Actual result
Deploy has failed with error message:
"Method granular_deploy. Deployment failed on nodes 3.
Inspect Astute logs for the details"

http://paste.openstack.org/show/328621/
---------------
[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "525"
  build_id: "2015-06-19_13-02-31"
  nailgun_sha: "dbd54158812033dd8cfd7e60c3f6650f18013a37"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "1ea8017fe8889413706d543a5b9f557f5414beae"
  fuel-library_sha: "2e7a08ad9792c700ebf08ce87f4867df36aa9fab"
  fuel-ostf_sha: "8fefcf7c4649370f00847cc309c24f0b62de718d"
  fuelmain_sha: "a3998372183468f56019c8ce21aa8bb81fee0c2f"
---------------
[root@nailgun ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|----------|---------------------|---------|------------|-------------------|------------|---------------|--------|---------
5 | ready | slave-05_compute | 1 | 10.109.0.7 | 64:c3:0c:3c:f4:38 | compute | | True | 1
4 | ready | slave-04_compute | 1 | 10.109.0.6 | 64:b3:ee:b9:54:8c | compute | | True | 1
3 | error | slave-03_controller | 1 | 10.109.0.5 | 64:ef:ab:39:16:f8 | controller | | True | 1
6 | discover | Untitled (47:2a) | None | 10.109.0.4 | 64:e2:0e:a2:47:2a | | | True | None
7 | discover | Untitled (ff:1b) | None | 10.109.0.3 | 64:d5:fb:e1:ff:1b | | | True | None
---------------
[root@node-3 ~]# crm status
Last updated: Tue Jun 30 14:09:08 2015
Last change: Tue Jun 30 13:20:48 2015
Stack: corosync
Current DC: node-3.test.domain.local (3) - partition WITHOUT quorum
Version: 1.1.11-97629de
3 Nodes configured
28 Resources configured

Online: [ node-3.test.domain.local ]
OFFLINE: [ node-1.test.domain.local node-2.test.domain.local ]
---------------

[Irina Povolotskaya (ipovolotskaya)]

Guys, if we don't get any clarification on this issue on Zabbix in a few days, we'd like to get this documented in both Zabbix and Fuel documentation. At least - in Release Notes for 6.1.

Thanks!

[Olesia Tsvigun (otsvigun)]

Verified with issue with new patch of zabbix and issue was not reproduced.

[root@nailgun tmp]# fuel plugins
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
id | name | version | package_version
---|-------------------|---------|----------------
1 | zabbix_monitoring | 1.0.0 | 2.0.0
[root@nailgun tmp]# fuel --fuel-version
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'

[Tatyanka (tatyana-leontovich)]

@Alexander could you please upload snapshot that contains log after deletion nodes and re-deployment

[Olesia Tsvigun (otsvigun)]

Snapshot in comment#14 for my previous comment#12

[Bogdan Dobrelya (bogdando)]

There is no details for reproduces w/o zabbix plugin. @Alexandr, please provide the logs @Tatyanka requested. Marking as incomplete for the Fuel project

[Tatyanka (tatyana-leontovich)]

I can not reproduce the issue described by Alexander, for vlan faced with https://bugs.launchpad.net/fuel/+bug/1470947, but it is easy workaround-ed by replacing data in resolv.conf . After this 2 node was successfully removed, cluster was re-deployed and operable

[Sam Stoelinga (sammiestoel)]

i just had encountered exactly the same or similar error using MOS 6.1 with Zabbix plugin built from source.

http://paste.openstack.org/show/351047/ and related puppet error: http://paste.openstack.org/show/351048/

[Sam Stoelinga (sammiestoel)]

I'm deploying an HA environment and have not removed any nodes. build_number: "525" build_id: "2015-06-19_13-02-31"

[Sam Stoelinga (sammiestoel)]

Any hotfix / workaround I can apply manually to an existing installation without having to re-install the node?

[Bartosz Kupidura (zynzel)]

@sammiestoel please build plugin from latest source. You use previous version.

[Sam Stoelinga (sammiestoel)]

Confirmed that commit: bc7d37abb27f56150d9226bab1977f28d127aa5c fixes the issue by doing the following:

git clone https://github.com/stackforge/fuel-plugin-external-zabbix
cd fuel-plugin-external-zabbix
mv /var/www/nailgun/plugins/zabbix_monitoring-1.0/deployment_scripts{,.bak}
cp -r deployment_scripts /var/www/nailgun/plugins/zabbix_monitoring-1.0/deployment_scripts

Then redeploying the environment which failed previously.

[Roman Prykhodchenko (romcheg)]

Folks, please update the status for 6.1 or it will be marked as Invalid.

[Olesia Tsvigun (otsvigun)]

Move to invalid(for 6.1) because it wasn't reproduced for Fuel ISO 525

DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'