test_always_same_order in TestOrdered in deployment_graph.py fails randomly

Bug #1438203 reported by Maciej Kwiek
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Dima Shulyak

Bug Description

./run_tests.sh -t nailgun/nailgun/test/unit/test_graph_serializer.py::TestOrdered::test_always_same_order sometimes fails, sometimes don't.

Error is:
    def test_always_same_order(self):
        self.assertEqual(
            [n['id'] for n in self.graph.topology],
> ['a', 'c', 'b', 'd', 'e'])
E AssertionError: Lists differ: ['a', 'd', 'c', 'b', 'e'] != ['a', 'c', 'b', 'd', 'e']
E
E First differing element 1:
E d
E c
E
E - ['a', 'd', 'c', 'b', 'e']
E + ['a', 'c', 'b', 'd', 'e']

It seems that inner structure of graph is not completely deterministic and in some cases may be sorted in a wrong way (`nx.topological_sort` call in `topology` property of `DeploymentGraph` class.

My proposal for a fix: make TASKS graph definition in the test linear (but I don't know why the test was there to begin with, so this may not be a proper approach).

It is possible to reproduce this on master on commit a3c259a4875787274fa01f0eba6514cc01b34308

Dmitry Pyzhov (dpyzhov)
tags: added: tech-tebt
tags: added: feature-tasks tech-debt
removed: tech-tebt
Changed in fuel:
milestone: none → 6.1
tags: removed: module-nailgun tech-debt
Łukasz Oleś (loles)
Changed in fuel:
status: New → Incomplete
status: Incomplete → Confirmed
importance: Undecided → Medium
Dmitry Pyzhov (dpyzhov)
tags: added: tech-debt
Revision history for this message
Dima Shulyak (dshulyak) wrote :

Topology should be deterministic, that is why i added

 https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/orchestrator/deployment_graph.py#L56

The problem is that dictionaries are used for internal data structures

Changed in fuel:
importance: Medium → High
status: Confirmed → Triaged
Dmitry Pyzhov (dpyzhov)
tags: added: module-tasks
removed: feature-tasks
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/169896

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Dima Shulyak (dshulyak)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/169896
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=a1310a1f18b5bc3feaa6e4aca12948e661bf93f1
Submitter: Jenkins
Branch: master

commit a1310a1f18b5bc3feaa6e4aca12948e661bf93f1
Author: Dmitry Shulyak <email address hidden>
Date: Wed Apr 1 12:26:36 2015 -0700

    Fix issue with non-determenistic ordering in networkx sorting

    In current master of networkx it is possible to specify fabric
    that will be used for internal graph data structures.

    To avoid non determinitic issues with ordering we need to use
    OrderedDict that will return same order of keys even with different
    hash seeds. OrderedDict should be used for storing order of
    nodes inserted in graph, and for dependencies between nodes
    in graph.

    Changed tests to verify that order will be same with different
    hash seeds.

    Networkx 1.9.1 (one which we are using)

      http://bit.ly/1CcgdBs

    Networkx change with fixed code

      http://bit.ly/1al6Tpq

    Change-Id: I16a880fe996a03645c0f79e3156ae403b9f006f5
    Closes-Bug: 1438203

Changed in fuel:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.