Deployment failed on simultaneous add/delete of ceph+cinder node

Bug #1578584 reported by Andrey Sledzinskiy
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Vladimir Kuklin
Mitaka
Fix Released
High
Alexey Shtokolov

Bug Description

fuel version - 9.0-275

Steps:
1. Deploy next cluster - Neutron Vlan, cinder for volumes, ceph for images and ephemeral, ceph replication factor - 2, 3 controllers, 1 compute, 2 ceph+cinder
3. After deployment add 1 cinder+ceph node and re-deploy
4. After re-deployment add 1 cinder+ceph node and delete one deployed cinder+ceph node
5. Start re-deployment

Expected result - re-deployment passed
Actual result - re-deployment failed, task status - 'u'values''

from nailgun/app.log:

2016-05-03 05:04:19.905 DEBUG [7f66c1226880] (task_serializer) evaluate yaql expression: changedAny($.storage, $.debug, $.network_scheme, $.get('use_syslog'), $.network_metadata.nodes.values.where(
  ('primary-controller' in $.node_roles) or
  ('controller' in $.node_roles)),
$.get('ceph_tuning_settings'), $.get('syslog_log_level_ceph'), $.get('syslog_log_facility_ceph'))

2016-05-03 05:04:19.954 ERROR [7f66c1226880] (__init__) Cannot evaluate expression - 'changedAny($.storage, $.debug, $.network_scheme, $.get('use_syslog'), $.network_metadata.nodes.values.where(
  ('primary-controller' in $.node_roles) or
  ('controller' in $.node_roles)),
$.get('ceph_tuning_settings'), $.get('syslog_log_level_ceph'), $.get('syslog_log_facility_ceph'))
' (None), : u'values'
2016-05-03 05:04:19.954 ERROR [7f66c1226880] (transaction_serializer) Failed to serialize task top-role-ceph-osd
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/lcm/transaction_serializer.py", line 140, in process_task
    task = task_serializer.serialize(node_id)
  File "/usr/lib/python2.7/site-packages/nailgun/lcm/task_serializer.py", line 190, in serialize
    'yaql_exp': self.context.get_yaql_interpreter(node_id)
  File "/usr/lib/python2.7/site-packages/nailgun/utils/__init__.py", line 161, in traverse
    value, formatter, formatter_context, keywords
  File "/usr/lib/python2.7/site-packages/nailgun/utils/__init__.py", line 144, in traverse
    return keywords[k](data[k])
  File "/usr/lib/python2.7/site-packages/nailgun/lcm/task_serializer.py", line 80, in evaluate
    return parsed_exp.evaluate(data=context['$%new'], context=context)
  File "/usr/lib/python2.7/site-packages/yaql/language/expressions.py", line 165, in evaluate
    return self(utils.NO_VALUE, context, self.engine)
  File "/usr/lib/python2.7/site-packages/yaql/language/expressions.py", line 156, in __call__
    return super(Statement, self).__call__(receiver, context, engine)
  File "/usr/lib/python2.7/site-packages/yaql/language/expressions.py", line 37, in __call__
    return context(self.name, engine, receiver, context)(*self.args)
  File "/usr/lib/python2.7/site-packages/yaql/language/contexts.py", line 65, in <lambda>
    data_context, use_convention, function_filter)
  File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 49, in call
    name, all_overloads, engine, receiver, data_context, args, kwargs)
  File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 117, in choose_overload
    args = tuple(arg_evaluator(i, arg) for i, arg in enumerate(args))
  File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 117, in <genexpr>
    args = tuple(arg_evaluator(i, arg) for i, arg in enumerate(args))
  File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 113, in <lambda>
    and not isinstance(arg, expressions.Constant))
  File "/usr/lib/python2.7/site-packages/yaql/language/expressions.py", line 37, in __call__
    return context(self.name, engine, receiver, context)(*self.args)
  File "/usr/lib/python2.7/site-packages/yaql/language/contexts.py", line 65, in <lambda>
    data_context, use_convention, function_filter)
  File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 51, in call
    result = delegate()
  File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 142, in <lambda>
    return lambda: delegate()
  File "/usr/lib/python2.7/site-packages/yaql/language/specs.py", line 341, in func
    six.iteritems(keyword_args)))
  File "/usr/lib/python2.7/site-packages/nailgun/yaql_ext/datadiff.py", line 62, in changed_any
    return any(changed(finalizer, exp, context) for exp in expressions)
  File "/usr/lib/python2.7/site-packages/nailgun/yaql_ext/datadiff.py", line 62, in <genexpr>
    return any(changed(finalizer, exp, context) for exp in expressions)
  File "/usr/lib/python2.7/site-packages/nailgun/yaql_ext/datadiff.py", line 48, in changed
    new_data = finalizer(get_new(expression, context))
  File "/usr/lib/python2.7/site-packages/nailgun/yaql_ext/datadiff.py", line 28, in get_new
    return expression(context['$%new'])
  File "/usr/lib/python2.7/site-packages/yaql/language/yaqltypes.py", line 269, in func
    engine, args, kwargs)
  File "/usr/lib/python2.7/site-packages/yaql/language/yaqltypes.py", line 236, in _call
    result = value(receiver, context, engine)
  File "/usr/lib/python2.7/site-packages/yaql/language/expressions.py", line 37, in __call__
    return context(self.name, engine, receiver, context)(*self.args)
  File "/usr/lib/python2.7/site-packages/yaql/language/contexts.py", line 65, in <lambda>
    data_context, use_convention, function_filter)
  File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 49, in call
    name, all_overloads, engine, receiver, data_context, args, kwargs)
  File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 117, in choose_overload
    args = tuple(arg_evaluator(i, arg) for i, arg in enumerate(args))
  File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 117, in <genexpr>
    args = tuple(arg_evaluator(i, arg) for i, arg in enumerate(args))
  File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 113, in <lambda>
    and not isinstance(arg, expressions.Constant))
  File "/usr/lib/python2.7/site-packages/yaql/language/expressions.py", line 37, in __call__
    return context(self.name, engine, receiver, context)(*self.args)
  File "/usr/lib/python2.7/site-packages/yaql/language/contexts.py", line 65, in <lambda>
    data_context, use_convention, function_filter)
  File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 51, in call
    result = delegate()
  File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 142, in <lambda>
    return lambda: delegate()
  File "/usr/lib/python2.7/site-packages/yaql/language/specs.py", line 341, in func
    six.iteritems(keyword_args)))
  File "/usr/lib/python2.7/site-packages/yaql/standard_library/collections.py", line 102, in dict_keyword_access
    return d[key]
KeyError: u'values'

Tags: area-python
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
summary: - Deployment failed on simultaneous delete of ceph+cinder node
+ Deployment failed on simultaneous add/delete of ceph+cinder node
Changed in fuel:
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
Changed in fuel:
status: New → Confirmed
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Fuel Toolbox (fuel-toolbox)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/313597

Changed in fuel:
assignee: Fuel Toolbox (fuel-toolbox) → Vladimir Kuklin (vkuklin)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/313597
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=707ca5ea1c7eb911a885b77ad13b020091551a7a
Submitter: Jenkins
Branch: master

commit 707ca5ea1c7eb911a885b77ad13b020091551a7a
Author: Vladimir Kuklin <email address hidden>
Date: Fri May 6 18:21:10 2016 +0300

    Fix typo in ceph-osd YAQL expression

    This is a simple one-line fix to fix a typo in YAQL expression
    for ceph-osd: values should be a function call instead of attribute
    fetching.

    Change-Id: I20d7e3b00b371e86bef9b17b6b4c343db271fbe3
    Closes-bug: #1578584

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/315510

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/315510
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b102375c30113937893ae2b060b5e315945600c7
Submitter: Jenkins
Branch: stable/mitaka

commit b102375c30113937893ae2b060b5e315945600c7
Author: Vladimir Kuklin <email address hidden>
Date: Fri May 6 18:21:10 2016 +0300

    Fix typo in ceph-osd YAQL expression

    This is a simple one-line fix to fix a typo in YAQL expression
    for ceph-osd: values should be a function call instead of attribute
    fetching.

    Change-Id: I20d7e3b00b371e86bef9b17b6b4c343db271fbe3
    Closes-bug: #1578584
    (cherry picked from commit 707ca5ea1c7eb911a885b77ad13b020091551a7a)

Maksym Strukov (unbelll)
tags: added: on-verification
Revision history for this message
Maksym Strukov (unbelll) wrote :

Verified as fixed in 9.0-404

Maksym Strukov (unbelll)
tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.