Replacement of controller leads to "KeyError: u'memcache_server_port'" in nailgun: (task_serializer) serialize task top-role-compute - cannot evaluate expression

[Vladimir Khlyunev]

Swarm tests failed:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.controller_replacement/88/testReport/

ISO 9.0-242

Steps to reproduce:
1. Create cluster
2. Add 3 node with controller role
3. Add 1 node with compute
4. Verify network
5. Deploy cluster
6. Remove one controller add new controller
7. Deploy changes << Fail here

Expected: No errors, cluster is deployed

Actual result:
Deployment failed on task serialization step:
http://paste.openstack.org/show/495261/

Snapshot:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.controller_replacement/88/artifact/logs/fail_error_deploy_ha_neutron_tun_ctrl_replacement-fuel-snapshot-2016-04-24_22-58-36.tar.xz

[Ksenia Svechnikova (kdemina)]

Could be a duplicate of https://bugs.launchpad.net/fuel/+bug/1574597

[Ilya Kutukov (ikutukov)]

Problem occur at compute node YAQL computation

changedAny(
     $.network_scheme,
     $.configuration,
     $.get('compute', ''),
     $.network_metadata.vips,
     $.debug,
     $.storage,
     $.nova,
     $.public_ssl,
     $.get('use_ssl', {}),
     $.get('nova_rate_limits', {}),
     $.get('nova_report_interval', 60),
     $.get('nova_service_down_time', 180),
     $.get('novanetwork_parameters', {}),
     $.get('nova_custom', {}), $.rabbit,
     $.cinder,
     $.ceilometer,
     $.get('use_syslog', true),
     $.get('use_stderr', false),
     $.get('syslog_log_facility_nova', ''),
     $.network_metadata.nodes.get(concat('node-', $.uid)),
     $.libvirt_type,
     $.get('kombu_compression', ''),
     $.dpdk,
     $.get('glance_api_servers', ''),
     $.get('block_device_allocate_retries', 300),
     $.get('block_device_allocate_retries_interval', 3),
     $.get('queue_provider', 'rabbit'),
     $.quantum_settings,
     $.get('memcached_addresses', ''),
     $.get('memcache_roles', ''),
     len($.network_metadata.nodes.values().where(
         $.network_roles.get('mgmt/memcache') != null)
     ),
     $.memcache_server_port, $.get('amqp_hosts', ''),
     len($.network_metadata.nodes.values().where(
             $.network_roles.get('mgmt/messaging') != null)
     ),
     $.resume_guests_state_on_host_boot, $.use_cow_images
)

Old and new context snapshot are here:
http://paste.openstack.org/show/495449/

[Ilya Kutukov (ikutukov)]

closed in
I9c50339474e84faa644cfe652a4d014b1ada2101
https://github.com/openstack/fuel-library/commit/f1c1200d7bdeca7e8c892ae368bc84d068e4f434