[regression] Error "TypeError: unhashable type" and "500 Internal Server Error" while deploying fuel

[EduardFazliev]

Detailed bug description:

[7f3c0bbc7880] (base) Unexpected exception occured
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/base.py", line 285, in content_json
    resp = func(cls, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/orchestrator.py", line 86, in GET
    return self._serialize(cluster, nodes)
  File "/usr/lib/python2.7/site-packages/nailgun/api/v1/handlers/orchestrator.py", line 158, in _serialize
    graph, cluster, nodes, ignore_customized=True)
  File "/usr/lib/python2.7/site-packages/nailgun/orchestrator/deployment_serializers.py", line 884, in serialize
    cluster, nodes, ignore_customized
  File "/usr/lib/python2.7/site-packages/nailgun/orchestrator/deployment_serializers.py", line 875, in _invoke_serializer
    cluster, nodes, ignore_customized=ignore_customized
  File "/usr/lib/python2.7/site-packages/nailgun/orchestrator/deployment_serializers.py", line 119, in serialize
    self.set_tasks(serialized_nodes)
  File "/usr/lib/python2.7/site-packages/nailgun/orchestrator/deployment_serializers.py", line 336, in set_tasks
    node['tasks'] = self.task_graph.deploy_task_serialize(node)
  File "/usr/lib/python2.7/site-packages/nailgun/orchestrator/orchestrator_graph.py", line 475, in deploy_task_serialize
    if not serializer.should_execute():
  File "/usr/lib/python2.7/site-packages/nailgun/orchestrator/tasks_serializer.py", line 68, in should_execute
    strict=False).evaluate()
  File "/usr/lib/python2.7/site-packages/nailgun/expression/__init__.py", line 24, in __init__
    self.compiled_expression = parse(self)
  File "/usr/lib/python2.7/site-packages/nailgun/expression/expression_parser.py", line 151, in parse
    return parser.parse(expression.expression_text, lexer=lexer)
  File "/usr/lib/python2.7/site-packages/ply/yacc.py", line 265, in parse
    return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
  File "/usr/lib/python2.7/site-packages/ply/yacc.py", line 889, in parseopt_notrack
    lexer.input(input)
  File "/usr/lib/python2.7/site-packages/ply/lex.py", line 253, in input
    c = s[:1]
TypeError: unhashable type
[pid: 23138|app: 0|req: 80/216] 10.109.2.1 () {34 vars in 529 bytes} [Tue Apr 12 21:29:25 2016] GET /api/v1/clusters/1/orchestrator/deployment/defaults/ => generated 15 bytes in 892 msecs (HTTP/1.1 500) 4 headers in 197 bytes (2 switches on core 0)

Steps to reproduce:
 Deploy Fuel ISO #193 with fuel-qa with configuration - 3 controllers, 1 compute node, Cinder LVM, Neutron VLAN

Expected results:
 Successfully deployed Fuel

Actual result:
 Error on Step 010. Deploy environment:
    2016-04-12 21:29:05,119 - ERROR http.py:110 -- Internal Server Error code 500 [unhashable type]
2016-04-12 21:29:05,125 - ERROR __init__.py:66 -- generate_ip_address_before_deploy_start raised: HTTPError()
Traceback: Traceback (most recent call last):
  File "/home/jenkins/workspace/9.0_create_param_environment/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0_create_param_environment/fuel-qa/fuelweb_test/models/nailgun_client.py", line 75, in generate_ip_address_before_deploy_start
    cluster_id))
  File "/home/jenkins/workspace/9.0_create_param_environment/fuel-qa/fuelweb_test/helpers/http.py", line 73, in get
    return self._open(req)
  File "/home/jenkins/workspace/9.0_create_param_environment/fuel-qa/fuelweb_test/helpers/http.py", line 97, in _open
    return self._get_response(req)
  File "/home/jenkins/workspace/9.0_create_param_environment/fuel-qa/fuelweb_test/helpers/http.py", line 121, in _get_response
    return self.opener.open(req)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error

2016-04-12 21:29:05,125 - INFO regenerate_repo.py:301 -- Check puppet logs for packages with unmet dependencies.
2016-04-12 21:29:05,223 - INFO regenerate_repo.py:312 -- Check puppet logs completed.
2016-04-12 21:29:05,270 - INFO fuel_web_client.py:1194 -- Wait for task 600 seconds:
 status running
 name dump
 cluster None
 result {}
 progress 0
 message None
 id 5
 uuid f11f9b1d-44f6-40e2-94d9-0a13c911b347
2016-04-12 21:29:35,421 - INFO fuel_web_client.py:1211 -- Task finished. Took 30.1437630653 seconds.
 status ready
 name dump
 cluster None
 result {}
 progress 100
 message /api/dump/fuel-snapshot-2016-04-12_21-29-05.tar.xz
 id 5
 uuid f11f9b1d-44f6-40e2-94d9-0a13c911b347

Reproducibility:

Workaround:
None at the moment

Impact:
deployment cannot be completed

Description of the environment:
 Operation system: Ubuntu 14.04
 Versions of components: {"release": "9.0", "auth_required": true, "api": "1", "openstack_version": "mitaka-9.0", "feature_groups": []}
 Reference architecture: HA
 Network model: Neutron with VLAN
 Related projects installed: Nothing

[EduardFazliev (efazliev)]

This bug is still in ISO #195, I've attached diagnostic snapshot from master node.

[Timur Nurlygayanov (tnurlygayanov)]

This bug blocks our Tempest CI jobs and all functional tests for MOS components, so, it is blocker issue for QA team.

Please describe at least a workaround for the issue or prepare the fix (will be better, of course).

[Timur Nurlygayanov (tnurlygayanov)]

Status changed to Critical because it blocks deployment of OpenStack clusters

[EduardFazliev (efazliev)]

FYI: ISO #196 is deployed without this error

Link: http://srv41-bud.infra.mirantis.net/fuelweb-iso/fuel-9.0-196-2016-04-13_08-00-00.iso