Deploy fails after removing nodes.

Bug #1469053 reported by Olesia Tsvigun
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fuel Plugins
Fix Released
Critical
Fuel Plugin Zabbix
Fuel for OpenStack
Fix Released
Critical
Bartosz Kupidura
6.1.x
Invalid
Critical
Fuel QA Team
7.0.x
Fix Released
Critical
Bartosz Kupidura

Bug Description

Deploy has was failed after removing nodes.
Fuel ISO#525
OS CentOS

Steps to reproduce:
1. Install Zabbix plugin.
2. In Wizard fill all required options:
- add name 'cvDeploy'of env and select default release version with OS 'Juno'
- as hypervisor type: default QEMU radiobutton
- network setup : Netron with Vlan segmetation( by default )
- storage backends: default
- additional services: all by default
create cluster

3. Add following nodes:
3 controllers
1 cinder
1 compute

4.Configure interfaces on the nodes:
eth0 - admin(PXE)
eth1 - public
eth2 - management
eth3 - private
eth4 - storage

5. Networks tab:
disable 'use vlan tag' for Public, Storage and Mangment

6.Settings tab
enable plugin 'Zabbix'
change default password

7. Deploy cluster.
8. Remove 2 controllers, cinder and compute nodes from cluster.
9. Deploy cluster

Actual result
Deploy has failed with error:
[649] Error running RPC method granular_deploy: Failed to execute hook 'zabbix_monitoring-1.0.0' Puppet run failed. Check puppet logs for details

from astute logs on master:
http://paste.openstack.org/show/321292/

from puppet logs on controller:
http://paste.openstack.org/show/321293/

Expected result:
Cluster should be deployed.

Fuel ISO version:
api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'

description: updated
Changed in fuel:
assignee: nobody → Fuel Plugin Zabbix (fuel-plugin-zabbix)
importance: Undecided → Critical
Changed in fuel-plugins:
assignee: nobody → Fuel Plugin Zabbix (fuel-plugin-zabbix)
importance: Undecided → Critical
Revision history for this message
Olesia Tsvigun (otsvigun) wrote :
Changed in fuel:
milestone: none → 6.1
description: updated
description: updated
Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :

My attempt of reproducing the bug according your steps lead to the same result, but due to error with virtual_ips.
I didn't find any error records connected to zabbix plugin
http://paste.openstack.org/show/321360/
http://paste.openstack.org/show/321364/

Revision history for this message
Irina Povolotskaya (ipovolotskaya) wrote :

Alexander, is the bug related to the Plugin Framework?
If yes, could you assign it to Evgeniy Li to make sure that's not at Zabbix side?

tags: added: feature-plugins
Revision history for this message
Tatyana Dubyk (tdubyk) wrote :

I've reproduced this bug again too with error:
"Deployment has failed. Method granular_deploy. Failed to execute hook 'zabbix_monitoring-1.0.0' Puppet run failed. Check puppet logs for details"

configuration in Wizard:
CentOS 6.5
KVM
Neutron vlan
other settings by default

Configure interfaces on the nodes:
eth0 - admin(PXE)
eth1 - public
eth2 - management
eth3 - private
eth4 - storage

Network settings:
all by default

Env with nodes:
1 controller
1 compute
1 cinder

Zabbix settings: by default

steps to reproduce:
1. install plugin
2. create env with options above
3. check network connectivity
4. perform deploy

actual result: deploy is failed with error "Deployment has failed. Method granular_deploy. Failed to execute hook 'zabbix_monitoring-1.0.0' Puppet run failed. Check puppet logs for details"

expected result: deploy must be successfully finished

Revision history for this message
Tatyana Dubyk (tdubyk) wrote :
Revision history for this message
Tatyana Dubyk (tdubyk) wrote :
Download full text (3.9 KiB)

[root@nailgun ~]# fuel --fuel-version

api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'
==============================================
[root@nailgun ~]# fuel nodes list

id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|----------|------------------|---------|-------------|-------------------|------------|---------------|--------|---------
9 | error | Untitled (94:e6) | 4 | 10.109.5.11 | 64:9c:44:02:94:e6 | cinder | | True | 4
8 | error | Untitled (bc:4d) | 4 | 10.109.5.10 | 64:02:95:a0:bc:4d | compute | | True | 4
11 | error | Untitled (ba:77) | 4 | 10.109.5.13 | 64:ae:b4:a8:ba:77 | controller | | True | 4

================================================
заббикс сервер установился на контроллер

[root@nailgun ~]# ssh node-11
Warning: Permanently added 'node-11' (RSA) to the list of known hosts.
Last login: Fri Jun 26 16:15:34 2015 from 10.109.5.2
[root@node-11 ~]# crm status
Last updated: Fri Jun 26 16:16:43 2015
Last change: Fri Jun 26 14:10:29 2015
Stack: corosync
Current DC: node-11.test.domain.local (11) - partition with quorum
Version: 1.1.11-97629de
1 Nodes configured
17 Resources configured

Online: [ node-11.test.domain.local ]

 Clone Set: clone_p_vrouter [p_vrouter]
     Started: [ node-11.test.domain.local ]
 vip__management (ocf::fuel:ns_IPaddr2): Started node-11.test.domain.local
 vip__public_vrouter (ocf::fuel:ns_IPaddr2): Started node-11.test.domain.local
 vip__management_vrouter (ocf::fuel:ns_IPaddr2): Started node-11.test.domain.local
 vip__public (ocf::fuel:ns_IPaddr2): Started node-11.test.domain.local
 Clone Set: clone_p_haproxy [p_haproxy]
     Started: [ node-11.test.domain.local ]
 Clone Set: clone_p_dns [p_dns]
     Started: [ node-11.test.domain.local ]
 Clone Set: clone_p_mysql [p_mysql]
     Started: [ node-11.test.domain.local ]
 Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
     Masters: [ node-11.test.domain.local ]
 Clone Set: clone_p_opensta...

Read more...

Revision history for this message
Tatyana Dubyk (tdubyk) wrote :
Changed in fuel-plugins:
status: New → Confirmed
Changed in fuel:
status: New → Confirmed
Changed in fuel:
assignee: Fuel Plugin Zabbix (fuel-plugin-zabbix) → Bartosz Kupidura (zynzel)
status: Confirmed → In Progress
Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :

I've reproduced the bug without zabbix plugin.

Steps to reproduce:
1. Deploy environment --group=deploy_ha_vlan (I used variable OPENSTACK_RELEASE=centos)
2. Remove 2 controllers.
3. Deploy cluster

Actual result
Deploy has failed with error message:
"Method granular_deploy. Deployment failed on nodes 3.
Inspect Astute logs for the details"

http://paste.openstack.org/show/328621/
---------------
[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "525"
  build_id: "2015-06-19_13-02-31"
  nailgun_sha: "dbd54158812033dd8cfd7e60c3f6650f18013a37"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "1ea8017fe8889413706d543a5b9f557f5414beae"
  fuel-library_sha: "2e7a08ad9792c700ebf08ce87f4867df36aa9fab"
  fuel-ostf_sha: "8fefcf7c4649370f00847cc309c24f0b62de718d"
  fuelmain_sha: "a3998372183468f56019c8ce21aa8bb81fee0c2f"
---------------
[root@nailgun ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|----------|---------------------|---------|------------|-------------------|------------|---------------|--------|---------
5 | ready | slave-05_compute | 1 | 10.109.0.7 | 64:c3:0c:3c:f4:38 | compute | | True | 1
4 | ready | slave-04_compute | 1 | 10.109.0.6 | 64:b3:ee:b9:54:8c | compute | | True | 1
3 | error | slave-03_controller | 1 | 10.109.0.5 | 64:ef:ab:39:16:f8 | controller | | True | 1
6 | discover | Untitled (47:2a) | None | 10.109.0.4 | 64:e2:0e:a2:47:2a | | | True | None
7 | discover | Untitled (ff:1b) | None | 10.109.0.3 | 64:d5:fb:e1:ff:1b | | | True | None
---------------
[root@node-3 ~]# crm status
Last updated: Tue Jun 30 14:09:08 2015
Last change: Tue Jun 30 13:20:48 2015
Stack: corosync
Current DC: node-3.test.domain.local (3) - partition WITHOUT quorum
Version: 1.1.11-97629de
3 Nodes configured
28 Resources configured

Online: [ node-3.test.domain.local ]
OFFLINE: [ node-1.test.domain.local node-2.test.domain.local ]
---------------

Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :
Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :
Revision history for this message
Irina Povolotskaya (ipovolotskaya) wrote :

Guys, if we don't get any clarification on this issue on Zabbix in a few days, we'd like to get this documented in both Zabbix and Fuel documentation. At least - in Release Notes for 6.1.

Thanks!

summary: - [Zabbix]Deploy was failed after removing nodes.
+ Deploy fails after removing nodes.
Revision history for this message
Olesia Tsvigun (otsvigun) wrote :

Verified with issue with new patch of zabbix and issue was not reproduced.

[root@nailgun tmp]# fuel plugins
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
id | name | version | package_version
---|-------------------|---------|----------------
1 | zabbix_monitoring | 1.0.0 | 2.0.0
[root@nailgun tmp]# fuel --fuel-version
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

@Alexander could you please upload snapshot that contains log after deletion nodes and re-deployment

Revision history for this message
Olesia Tsvigun (otsvigun) wrote :
Revision history for this message
Olesia Tsvigun (otsvigun) wrote :

Snapshot in comment#14 for my previous comment#12

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

There is no details for reproduces w/o zabbix plugin. @Alexandr, please provide the logs @Tatyanka requested. Marking as incomplete for the Fuel project

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

I can not reproduce the issue described by Alexander, for vlan faced with https://bugs.launchpad.net/fuel/+bug/1470947, but it is easy workaround-ed by replacing data in resolv.conf . After this 2 node was successfully removed, cluster was re-deployed and operable

Revision history for this message
Sam Stoelinga (sammiestoel) wrote :

i just had encountered exactly the same or similar error using MOS 6.1 with Zabbix plugin built from source.

http://paste.openstack.org/show/351047/ and related puppet error: http://paste.openstack.org/show/351048/

Revision history for this message
Sam Stoelinga (sammiestoel) wrote :

I'm deploying an HA environment and have not removed any nodes. build_number: "525" build_id: "2015-06-19_13-02-31"

Revision history for this message
Sam Stoelinga (sammiestoel) wrote :

Any hotfix / workaround I can apply manually to an existing installation without having to re-install the node?

Revision history for this message
Bartosz Kupidura (zynzel) wrote :

@sammiestoel please build plugin from latest source. You use previous version.

Revision history for this message
Sam Stoelinga (sammiestoel) wrote :

Confirmed that commit: bc7d37abb27f56150d9226bab1977f28d127aa5c fixes the issue by doing the following:

git clone https://github.com/stackforge/fuel-plugin-external-zabbix
cd fuel-plugin-external-zabbix
mv /var/www/nailgun/plugins/zabbix_monitoring-1.0/deployment_scripts{,.bak}
cp -r deployment_scripts /var/www/nailgun/plugins/zabbix_monitoring-1.0/deployment_scripts

Then redeploying the environment which failed previously.

Changed in fuel-plugins:
status: Confirmed → Fix Released
Revision history for this message
Roman Prykhodchenko (romcheg) wrote :

Folks, please update the status for 6.1 or it will be marked as Invalid.

Revision history for this message
Olesia Tsvigun (otsvigun) wrote :

Move to invalid(for 6.1) because it wasn't reproduced for Fuel ISO 525

DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.