[detach-keystone] Deployment with detached-db failed on redeployment step after db node deletion

Bug #1542307 reported by Tatyanka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Volodymyr Shypyguzov
8.0.x
Fix Committed
High
Fuel Library (Deprecated)

Bug Description

Add and delete database node

Scenario:
1. Revert snapshot separate_db_service
2. Add one databse node and re-deploy cluster
3. Run network verification
4. Run OSTF
5. Check hiera hosts are the same for
different group of roles
6. Delete one database node
7. Run network verification
8. Run ostf
9. Check hiera hosts are the same for
different group of roles

Actual:
Re-deployment failed on controller after adding db node with:

2016-02-04 23:28:39 +0000 Puppet (err): pick(): must receive at least one non empty value at /etc/puppet/modules/osnailyfacter/modular/openstack-cinder/db.pp:19 on node node-1.test.domain.local
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
/etc/puppet/modules/stdlib/lib/puppet/parser/functions/pick.rb:24:in `block in <module:Functions>'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:164:in `block (2 levels) in newfunction'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/functions.rb:157:in `block in newfunction'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/function.rb:42:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/vardef.rb:14:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:11:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `each'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast/block_expression.rb:10:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/ast.rb:61:in `safeevaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/type.rb:129:in `evaluate_code'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/resource.rb:77:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:426:in `evaluate_main'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:135:in `block (2 levels) in compile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:135:in `block in compile'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:121:in `compile'
/usr/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:34:in `compile'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:95:in `block (2 levels) in compile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:93:in `block in compile'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:92:in `compile'
/usr/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:52:in `find'
:

[root@nailgun ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|-------------|------------------------------|---------|-------------|-------------------|---------------------|---------------|--------|---------
8 | provisioned | slave-09_standalone-database | 1 | 10.109.5.11 | 64:28:5f:29:90:33 | standalone-database | | True | 1
9 | ready | slave-07_compute | 1 | 10.109.5.9 | 64:f5:58:c4:fc:df | compute | | True | 1
2 | ready | slave-02_controller | 1 | 10.109.5.4 | 64:8d:b0:d0:64:b5 | controller | | True | 1
6 | ready | slave-06_standalone-database | 1 | 10.109.5.8 | 64:b1:a7:b7:44:64 | standalone-database | | True | 1
5 | ready | slave-05_standalone-database | 1 | 10.109.5.7 | 64:0d:1f:c6:f2:59 | standalone-database | | True | 1
4 | ready | slave-04_standalone-database | 1 | 10.109.5.6 | 64:28:58:35:28:d0 | standalone-database | | True | 1
7 | ready | slave-08_cinder | 1 | 10.109.5.10 | 64:9f:6f:32:77:c2 | cinder | | True | 1
3 | ready | slave-01_controller | 1 | 10.109.5.3 | 64:ed:d7:51:88:f9 | controller | | True | 1
1 | error | slave-03_controller | 1 | 10.109.5.5 | 64:6e:6c:b6:c0:4d | controller | | True | 1

[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "523"
  build_id: "523"
  fuel-nailgun_sha: "9cfaee884d5a2a7304905f29bde57adebffa7577"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "71d3a7f43effd457391a1f02558fba4f0ca90eec"
  fuel-ostf_sha: "7bcddf18020f2d94a553a441ff57dff9632865df"
  fuel-mirror_sha: "c25d8931e30322ecf43246c8a957e376259b685c"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"
[root@nailgun ~]#

tags: added: non-release team-bugfix
Changed in fuel:
status: New → Confirmed
assignee: Fuel Library Team (fuel-library) → Matthew Mosesohn (raytrac3r)
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

2016-02-04 23:28:39 +0000 Puppet (err): pick(): must receive at least one non empty value at /etc/puppet/modules/osnailyfacter/modular/openstack-cinder/db.pp:19 on node node-1.test.domain.local

The issue is hiera_hash('mysql_hash',{}) doesn't do a proper deep lookup. We need to get rid of all _hash references sine we have deep hash lookups. I think it's too late to fix this for 8.0, though.

Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

I'll propose a minimal fix that can go to 8.0 and then a complete fix for this issue so that this issue won't appear elsewhere.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/276797

Changed in fuel:
status: Confirmed → In Progress
tags: added: release-notes
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-ostf (master)

Fix proposed to branch: master
Review: https://review.openstack.org/280596

Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

8.0 is fix committed from https://review.openstack.org/#/c/277458/ which landed Feb 8.

tags: added: 8.0 release-notes-done
removed: release-notes
tags: removed: 8.0 release-notes-done
tags: added: life-cycle-management
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-ostf (master)

Reviewed: https://review.openstack.org/280596
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=356a99cd9ed999a8527df36c4ec27c939b990ece
Submitter: Jenkins
Branch: master

commit 356a99cd9ed999a8527df36c4ec27c939b990ece
Author: Matthew Mosesohn <email address hidden>
Date: Tue Feb 16 13:35:58 2016 +0300

    Fix rabbitmq hash lookups

    rabbitmq hash lookups should use 'rabbitmq' key
    instead of 'rabbitmq_hash' to do proper deep hash
    merges.

    Since this patch would create a CI failure if it
    landed before its corresponding fuel-library fix,
    a fallback to rabbit_hash still exists.

    Change-Id: I6bf14fb4e869b8f2d34c92f5b6f2ad1d9afe26b2
    Partial-Bug: #1542307

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/276797
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=052c9824173b7e76f4149b1059e851053961ed49
Submitter: Jenkins
Branch: master

commit 052c9824173b7e76f4149b1059e851053961ed49
Author: Matthew Mosesohn <email address hidden>
Date: Fri Feb 5 19:42:20 2016 +0400

    Clean up globals hash keys and all hash lookups

    hiera deep hash lookup fails if one yaml contains 'key_hash'
    and the other is just 'key'. This is fixed by writing keys
    without the suffix '_hash'.

    Added _hash to some variables where it was inconsistent
    among top level modules.

    Change-Id: I48ec5711a507115d8b7cc167ae881cac4637622f
    Closes-Bug: #1542307

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

This was also fixed for 7.0-updates in commit https://review.openstack.org/300038, but was accidentally marked related to the bug https://bugs.launchpad.net/fuel/+bug/1526917

Changed in fuel:
assignee: Matthew Mosesohn (raytrac3r) → Volodymyr Shypyguzov (vshypyguzov)
tags: added: on-verification
Revision history for this message
Volodymyr Shypyguzov (vshypyguzov) wrote :
tags: removed: on-verification
Revision history for this message
Volodymyr Shypyguzov (vshypyguzov) wrote :

Verified by SWARM on 9.0 iso #477

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.