Openstack configuration fail with KeyError: u'memcache_server_port'

[Ksenia Svechnikova]

ISO http://paste.openstack.org/show/495299/

Detailed bug description:

Huge number tests for service-reconfiguration fails. For example, let's see test for Tests for reconfiguration nova quota.

Steps to reproduce:

Scenario:
            1. Deploy cluster with 3 controllers and 2 computes
            2. Upload a new openstack configuration
            3. Get uptime of process "nova-api" on each controller
            4. Get uptime of process "nova-compute" on each compute
            5. Apply a new quota driver and quota_instances to all nodes
            6. Wait for configuration applying
            7. Verify uptime of process "nova-api" on each controller
            8. Verify uptime of process "nova-compute" on each compute

Expected results: success
Actual result: Task 'deployment' has incorrect status. error != ready, 'u'memcache_server_port''

Yaml file for configuration:

configuration:
  nova_config:
    DEFAULT/quota_driver:
      value: nova.quota.DbQuotaDriver
    DEFAULT/quota_instances:
      value: '1'

Trace for nailgun: http://paste.openstack.org/show/495304/

[Ksenia Svechnikova (kdemina)]

yaml with configuration:

configuration:
  nova_config:
    DEFAULT/quota_driver:
      value: nova.quota.DbQuotaDriver
    DEFAULT/quota_instances:
      value: '1'

[Krzysztof Szukiełojć (kszukielojc)]

2016-04-25 16:48:34.524 DEBUG [7fb7dbb00880] (task_serializer) serialize task top-role-compute for node 3
2016-04-25 16:48:34.524 DEBUG [7fb7dbb00880] (task_serializer) evaluate yaql expression: changedAny($.network_scheme, $.configuration, $.get('compute', ''), $.network_metadata.vips, $.debug, $.storage, $.nova, $.public_ssl, $.get('use_ssl', {}), $.get('nova_rate_limits', {}), $.get('nova_report_interval', 60), $.get('nova_service_down_time', 180), $.get('novanetwork_parameters', {}), $.get('nova_custom', {}), $.rabbit, $.cinder, $.ceilometer, $.get('use_syslog', true), $.get('use_stderr', false), $.get('syslog_log_facility_nova', ''), $.network_metadata.nodes.get(concat('node-', $.uid)), $.libvirt_type, $.get('kombu_compression', ''), $.dpdk, $.get('glance_api_servers', ''), $.get('block_device_allocate_retries', 300), $.get('block_device_allocate_retries_interval', 3), $.get('queue_provider', 'rabbit'), $.quantum_settings, $.get('memcached_addresses', ''), $.get('memcache_roles', ''), len($.network_metadata.nodes.values().where(
  $.network_roles.get('mgmt/memcache') != null)),
$.memcache_server_port, $.get('amqp_hosts', ''), len($.network_metadata.nodes.values().where(
  $.network_roles.get('mgmt/messaging') != null)),
$.resume_guests_state_on_host_boot, $.use_cow_images)

2016-04-25 16:48:34.606 ERROR [7fb7dbb00880] (__init__) Cannot evaluate expression - 'changedAny($.network_scheme, $.configuration, $.get('compute', ''), $.network_metadata.vips, $.debug, $.storage, $.nova, $.public_ssl, $.get('use_ssl', {}), $.get('nova_rate_limits', {}), $.get('nova_report_interval', 60), $.get('nova_service_down_time', 180), $.get('novanetwork_parameters', {}), $.get('nova_custom', {}), $.rabbit, $.cinder, $.ceilometer, $.get('use_syslog', true), $.get('use_stderr', false), $.get('syslog_log_facility_nova', ''), $.network_metadata.nodes.get(concat('node-', $.uid)), $.libvirt_type, $.get('kombu_compression', ''), $.dpdk, $.get('glance_api_servers', ''), $.get('block_device_allocate_retries', 300), $.get('block_device_allocate_retries_interval', 3), $.get('queue_provider', 'rabbit'), $.quantum_settings, $.get('memcached_addresses', ''), $.get('memcache_roles', ''), len($.network_metadata.nodes.values().where(
  $.network_roles.get('mgmt/memcache') != null)),
$.memcache_server_port, $.get('amqp_hosts', ''), len($.network_metadata.nodes.values().where(
  $.network_roles.get('mgmt/messaging') != null)),
$.resume_guests_state_on_host_boot, $.use_cow_images)
' (None), : u'memcache_server_port'

[Krzysztof Szukiełojć (kszukielojc)]

deployment/puppet/openstack_tasks/examples/roles/tasks.yaml
Should be taken care by fuel-library

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/310123

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/310143

[Krzysztof Szukiełojć (kszukielojc)]

ETA: next week. As most of core reviewers are on summit.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310123
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f34798ff49f0eb77eff9091fd6a081f31da0fb40
Submitter: Jenkins
Branch: master

commit f34798ff49f0eb77eff9091fd6a081f31da0fb40
Author: Krzysztof Szukiełojć <email address hidden>
Date: Tue Apr 26 09:54:07 2016 +0200

    Fix for roles/tasks.yaml in memcache_server_port

    Closes-bug: 1574597
    Change-Id: Iee10eed23d79511b0a91d74070b7b32b2846a06c

[Ksenia Svechnikova (kdemina)]

Verify on ISO 10.0 157, version http://paste.openstack.org/show/495641/

No errors with execution task

[OpenStack Infra (hudson-openstack)]

Change abandoned by Krzysztof Szukiełojć (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/310143
Reason: https://review.openstack.org/#/c/310226/1/deployment/puppet/openstack_tasks/examples/roles/tasks.yaml@38 this has fix and already landed on stable/mitaka

[Krzysztof Szukiełojć (kszukielojc)]

https://review.openstack.org/#/c/310226/1/deployment/puppet/openstack_tasks/examples/roles/tasks.yaml@38 this has landed on stable/mitaka and has already fix for it

[Krzysztof Szukiełojć (kszukielojc)]

RCA: In progress to moving from legacy conditions to YAQL conditions somebody made mistake in writing them. Probably due to not full knowledge how it works.

[Ksenia Svechnikova (kdemina)]

Verify issue on ISO fuel-9.0-302 using system tests from group services_reconfiguration_thread_1