Kolla tests failing for tripleo-common master

[Adriano Petrich]

5 kolla tests are failing for master tripleo-common (tripleo_common/tests/image/test_kolla_builder.py)

they all follow the same pattern

First differing element 0:
{u'imagename': u'tripleoupstream/centos-binary-aodh-api:latest'}
{u'imagename': u"<class 'jinja2.utils.Namespace'>/centos-binary-aodh-api:latest"}

somehow {{namespace}} on the templates are being rendered as <class 'jinja2.utils.Namespace'> instead of the "tripleoupstream"

log of the errors:

FAIL: tripleo_common.tests.image.test_kolla_builder.TestKollaImageBuilderTemplate.test_container_images_from_template
tags: worker-6
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tripleo_common/tests/image/test_kolla_builder.py", line 157, in test_container_images_from_template
    self.assertEqual(container_images, result)
  File "/home/frac/Code/tripleo-common/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/frac/Code/tripleo-common/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: !=:
reference = [{u'imagename': u'tripleoupstream/heat-docker-agents-centos:latest',
  u'push_destination': u'localhost:8787'},
 {u'imagename': u'tripleoupstream/centos-binary-nova-compute:liberty',
  u'pull_source': u'docker.io',
  u'push_destination': u'localhost:8787',
  u'uploader': u'docker'},
 {u'imagename': u'tripleoupstream/centos-binary-nova-libvirt:liberty',
  u'pull_source': u'docker.io',
  u'uploader': u'docker'},
 {u'imagename': u'tripleoupstream/image-with-missing-tag',
  u'push_destination': u'localhost:8787'}]
actual = [{u'imagename': u"<class 'jinja2.utils.Namespace'>/heat-docker-agents-centos:latest",
  u'push_destination': u'localhost:8787'},
 {u'imagename': u"<class 'jinja2.utils.Namespace'>/centos-binary-nova-compute:liberty",
  u'pull_source': u'docker.io',
  u'push_destination': u'localhost:8787',
  u'uploader': u'docker'},
 {u'imagename': u"<class 'jinja2.utils.Namespace'>/centos-binary-nova-libvirt:liberty",
  u'pull_source': u'docker.io',
  u'uploader': u'docker'},
 {u'imagename': u"<class 'jinja2.utils.Namespace'>/image-with-missing-tag",
  u'push_destination': u'localhost:8787'}]
======================================================================
FAIL: tripleo_common.tests.image.test_kolla_builder.TestKollaImageBuilderTemplate.test_container_images_from_template_filter
tags: worker-4
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tripleo_common/tests/image/test_kolla_builder.py", line 192, in test_container_images_from_template_filter
    self.assertEqual(container_images, result)
  File "/home/frac/Code/tripleo-common/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/frac/Code/tripleo-common/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: !=:
reference = [{'imagename': 'tripleoupstream/centos-binary-nova-compute:liberty',
  'pull_source': 'docker.io',
  'push_destination': 'localhost:8787',
  'uploader': 'docker'},
 {'imagename': 'tripleoupstream/centos-binary-nova-libvirt:liberty',
  'pull_source': 'docker.io',
  'push_destination': 'localhost:8787',
  'uploader': 'docker'},
 {'imagename': 'tripleoupstream/image-with-missing-tag',
  'pull_source': 'docker.io',
  'push_destination': 'localhost:8787'}]
actual = [{u'imagename': u"<class 'jinja2.utils.Namespace'>/centos-binary-nova-compute:liberty",
  u'pull_source': 'docker.io',
  u'push_destination': 'localhost:8787',
  u'uploader': u'docker'},
 {u'imagename': u"<class 'jinja2.utils.Namespace'>/centos-binary-nova-libvirt:liberty",
  u'pull_source': 'docker.io',
  'push_destination': 'localhost:8787',
  u'uploader': u'docker'},
 {u'imagename': u"<class 'jinja2.utils.Namespace'>/image-with-missing-tag",
  'pull_source': 'docker.io',
  u'push_destination': 'localhost:8787'}]
======================================================================
FAIL: tripleo_common.tests.image.test_kolla_builder.TestKollaImageBuilderTemplate.test_container_images_yaml_in_sync
tags: worker-3
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tripleo_common/tests/image/test_kolla_builder.py", line 246, in test_container_images_yaml_in_sync
    remove_images=remove_images)
  File "tripleo_common/tests/image/test_kolla_builder.py", line 227, in _test_container_images_yaml_in_sync_helper
    self.assertSequenceEqual(container_images, result)
  File "/home/frac/Code/tripleo-common/.tox/py27/lib/python2.7/site-packages/unittest2/case.py", line 1014, in assertSequenceEqual
    self.fail(msg)
  File "/home/frac/Code/tripleo-common/.tox/py27/lib/python2.7/site-packages/unittest2/case.py", line 690, in fail
    raise self.failureException(msg)
AssertionError: Sequences differ: [{u'imagename': u'tripleoupstream/centos-binary-aodh-api:l[5938 chars]-7'}] != [{u'imagename': u"<class 'jinja2.utils.Namespace'>/centos-[7366 chars]-7'}]

First differing element 0:
{u'imagename': u'tripleoupstream/centos-binary-aodh-api:latest'}
{u'imagename': u"<class 'jinja2.utils.Namespace'>/centos-binary-aodh-api:latest"}

Diff is 25699 characters long. Set self.maxDiff to None to see it.
======================================================================
FAIL: tripleo_common.tests.image.test_kolla_builder.TestKollaImageBuilderTemplate.test_container_images_yaml_in_sync_for_ovn
tags: worker-1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tripleo_common/tests/image/test_kolla_builder.py", line 275, in test_container_images_yaml_in_sync_for_ovn
    neutron_driver='ovn', remove_images=remove_images)
  File "tripleo_common/tests/image/test_kolla_builder.py", line 227, in _test_container_images_yaml_in_sync_helper
    self.assertSequenceEqual(container_images, result)
  File "/home/frac/Code/tripleo-common/.tox/py27/lib/python2.7/site-packages/unittest2/case.py", line 1014, in assertSequenceEqual
    self.fail(msg)
  File "/home/frac/Code/tripleo-common/.tox/py27/lib/python2.7/site-packages/unittest2/case.py", line 690, in fail
    raise self.failureException(msg)
AssertionError: Sequences differ: [{u'imagename': u'tripleoupstream/centos-binary-aodh-api:l[6296 chars]-7'}] != [{u'imagename': u"<class 'jinja2.utils.Namespace'>/centos-[7809 chars]-7'}]

First differing element 0:
{u'imagename': u'tripleoupstream/centos-binary-aodh-api:latest'}
{u'imagename': u"<class 'jinja2.utils.Namespace'>/centos-binary-aodh-api:latest"}

Diff is 27351 characters long. Set self.maxDiff to None to see it.
======================================================================
FAIL: tripleo_common.tests.image.test_kolla_builder.TestKollaImageBuilderTemplate.test_container_images_yaml_in_sync_for_odl
tags: worker-2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tripleo_common/tests/image/test_kolla_builder.py", line 264, in test_container_images_yaml_in_sync_for_odl
    neutron_driver='odl', remove_images=remove_images)
  File "tripleo_common/tests/image/test_kolla_builder.py", line 227, in _test_container_images_yaml_in_sync_helper
    self.assertSequenceEqual(container_images, result)
  File "/home/frac/Code/tripleo-common/.tox/py27/lib/python2.7/site-packages/unittest2/case.py", line 1014, in assertSequenceEqual
    self.fail(msg)
  File "/home/frac/Code/tripleo-common/.tox/py27/lib/python2.7/site-packages/unittest2/case.py", line 690, in fail
    raise self.failureException(msg)
AssertionError: Sequences differ: [{u'imagename': u'tripleoupstream/centos-binary-aodh-api:l[6021 chars]-7'}] != [{u'imagename': u"<class 'jinja2.utils.Namespace'>/centos-[7466 chars]-7'}]

First differing element 0:
{u'imagename': u'tripleoupstream/centos-binary-aodh-api:latest'}
{u'imagename': u"<class 'jinja2.utils.Namespace'>/centos-binary-aodh-api:latest"}

Diff is 26075 characters long. Set self.maxDiff to None to see it.
Ran 372 tests in 4.316s (+2.855s)
FAILED (id=145, failures=5)

[Julie Pichon (jpichon)]

Triaging as Critical since no patch can merge in tripleo-common until this is resolved.

[Adriano Petrich (apetrich)]

It seems to be using https://github.com/pallets/jinja/blob/master/jinja2/utils.py#L615 instead of just a variable. if I change the variable name from namespace to namespace1 I can get the tests to pass

[Julie Pichon (jpichon)]

Dependency differences between a recent passing job, and a failing job for the same patch on CI:

-certifi==2017.7.27.1
+certifi==2017.11.5
-cmd2==0.7.7
+cmd2==0.7.8
-coverage==4.4.1
+coverage==4.4.2
-cryptography==2.1.2
+cryptography==2.1.3
-docker==2.5.1
+docker==2.6.1
-dulwich==0.18.4
+dulwich==0.18.6
-Jinja2==2.9.6
+Jinja2==2.10
-jsonpointer==1.13
+jsonpointer==1.14
-monotonic==1.3
+monotonic==1.4
-oslo.log==3.32.0
+oslo.log==3.33.0
-oslo.utils==3.30.0
+oslo.utils==3.31.0
-oslotest==2.18.1
+oslotest==3.0.0
-PyNaCl==1.1.2
+PyNaCl==1.2.0
-pyperclip==1.5.27
+pyperclip==1.6.0
-python-mistralclient==3.1.3
+python-mistralclient==3.1.4
-pytz==2017.2
+pytz==2017.3
-simplejson==3.11.1
+simplejson==3.12.0

[Julie Pichon (jpichon)]

Locally, the tests pass again after downgrading Jinja2 back to 2.9.6.

[Julie Pichon (jpichon)]

It looks like a "namespace" function was added to Jinja 2.10 [1] so that corroborates your findings in comment #2, Adriano! I guess choosing another name should resolve the issue.

[1] https://github.com/pallets/jinja/releases/tag/2.10

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/519619

[Julie Pichon (jpichon)]

Note: Jinja2 looks to be capped at 2.9.6 in Pike so it shouldn't be an issue on the stable branch.

https://github.com/openstack/requirements/blob/stable/pike/upper-constraints.txt#L190

[Dougal Matthews (d0ugal)]

This will be fixed by https://bugs.launchpad.net/tripleo/+bug/1732140

[OpenStack Infra (hudson-openstack)]

Change abandoned by Adriano Petrich (<email address hidden>) on branch: master
Review: https://review.openstack.org/519619

[Adriano Petrich (apetrich)]

Duplicate will be fixed by

https://review.openstack.org/#/c/516136/

[Julie Pichon (jpichon)]

I'm going to reopen this so we can track the issue with the tests separately from the refactoring in https://review.openstack.org/#/c/516136/ (it only fixes this coincidentally). That way people can find the bug in a launchpad search when they hit it, until that review merges :) We can update as "Fix released" then.

[Alex Schultz (alex-schultz)]

adding alert ci for visibility since this is causing failing ci jobs. the pending fix is struggling to merge

[Adriano Petrich (apetrich)]

https://review.openstack.org/#/c/516136/ has been merged tests are passing. Moving this to fix released