snat tests are failing as 'ip netns list' is not created.

[vimal]

snat /lbaas tests are failing as 'ip netns list' is not created.

image
--
queens-5.0-153

commands
--------------

Run a command in a running container
[root@nodem5 ~]# docker exec -it vrouter_vrouter-agent_1 bash
(vrouter-agent)[root@nodem5 /]$
(vrouter-agent)[root@nodem5 /]$
(vrouter-agent)[root@nodem5 /]$ ip netns list
(vrouter-agent)[root@nodem5 /]$
(vrouter-agent)[root@nodem5 /]$

<?xml-stylesheet type="text/xsl" href="/universal_parse.xsl"?><ServiceInstanceListResp type="sandesh"><si_names type="list" identifier="1">
<list type="struct" size="1"><ServiceInstance><name type="string" identifier="1">default-domain:ctest-TestRouterSNAT-57448712:snat_ec69bdc6-f98d-4770-a3eb-f7fea304a81c_6
e197c8d-3e3c-4b31-a59e-8f06c77f0213</name><si_type type="string" identifier="2">network-namespace</si_type><si_state type="string" identifier="3">active</si_state><vm_l
ist type="list" identifier="4"><list type="struct" size="2"><ServiceInstanceVM><name type="string" identifier="1">default-domain__ctest-TestRouterSNAT-57448712__snat_e
c69bdc6-f98d-4770-a3eb-f7fea304a81c_6e197c8d-3e3c-4b31-a59e-8f06c77f0213__1: 4db94890-9cca-48c8-9d85-77e2a0bb7283</name><vr_name type="string" identifier="2">nodem5</vr_name><ha type="string" identifier="3">active: 200</ha></ServiceInstanceVM><ServiceInstanceVM><name type="string" identifier="1">default-domain__ctest-TestRouterSNAT-57448712__snat_ec69bdc6-f98d-4770-a3eb-f7fea304a81c_6e197c8d-3e3c-4b31-a59e-8f06c77f0213__2: 272d2453-7c29-4acf-a33d-fcd81dd86e68</name><vr_name type="string" identifier="2">nodem20</vr_name><ha type="string" identifier="3">standby: 100</ha></ServiceInstanceVM></list></vm_list><left_vn type="list" identifier="5"><list type="string" size="2"><element>snat-si-left_snat_ec69bdc6-f98d-4770-a3eb-f7fea304a81c_6e197c8d-3e3c-4b31-a59e-8f06c77f0213</element><element>1f7195f1-369a-4c78-99a4-84a7cee4dfed</element></list></left_vn><right_vn type="list" identifier="6"><list type="string" size="2"><element>10.204.220.200/29</element><element>53186d89-9a86-4bc7-8ab1-fa6e11c47eb1</element></list></right_vn></ServiceInstance></list></si_names></ServiceInstanceListResp>

[Sudheendra Rao (sudheendra-k)]

assigning the bug to Abhay to route it to the right engineer.

[Yuvaraja Mariappan (ymariappan)]

enum-compat package is missing. so it is failing.

Traceback (most recent call last):
  File "/usr/bin/opencontrail-vrouter-netns", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 635, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 829, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'enum-compat' distribution was not found and is required by eventlet

Thanks,
Yuvaraja

[Andrey Pavlov (apavlov-e)]

can you please point me who is load enum-compat?

I didn't find any reference to enum-compat in the project
so I don't understand which module should have the dependency

and also from description of enum-compat
```enum-compat
===========

This is a virtual package, its whole purpose is to install enum34 on
Python older than 3.4. On Python 3.4+ it's a no-op.```
also this package doesn't have any installed files

[Yuvaraja Mariappan (ymariappan)]

eventlet needs it.
[package is needed in the contrail-vrouter-agent container]

(vrouter-agent)[root@nodem19 /usr/lib/python2.7/site-packages/opencontrail_vrouter_netns]$ grep -ri 'eventlet' .
./linux/utils.py: from eventlet.green import subprocess
./linux/utils.py: from eventlet import greenthread
Binary file ./linux/utils.pyc matches
Binary file ./linux/utils.pyo matches

Thanks,
Yuvaraja

[Sudheendra Rao (sudheendra-k)]

i don't know why it's moved to incomplete state, please provide reason while moving the bug to different state.

[Andrey Pavlov (apavlov-e)]

@Sudhee,

When I moved this bug to incomplete state - I didn't find which package should have additional dependency.
I awaited that Yuvaraja will answer to me and move the bug into new state.

Then bug was moved to Lukasz cause he started review before me. He was informed about this bug in the review.

https://review.opencontrail.org/#/c/45001/

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/45001
Submitter: Andrey Pavlov (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/45080
Submitter: Andrey Pavlov (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/45121
Submitter: Andrey Pavlov (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/45122
Submitter: Andrey Pavlov (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/45122
Committed: http://github.com/Juniper/contrail-packages/commit/d03714471a907faa18b7eb4a5fabc0804e7e6a9e
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit d03714471a907faa18b7eb4a5fabc0804e7e6a9e
Author: Andrey Pavlov <email address hidden>
Date: Tue Jul 31 14:13:16 2018 +0300

add python-enum34 for vrouter-netns

eventlet needs it but doesn't have in its dependencies

Change-Id: Iab97b7f595edcc798478383575d0972edbb7bb64
Closes-Bug: #1783519

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/45121
Committed: http://github.com/Juniper/contrail-packages/commit/b6313b2c73224182d77d3998ef242ec21cf56000
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit b6313b2c73224182d77d3998ef242ec21cf56000
Author: Andrey Pavlov <email address hidden>
Date: Tue Jul 31 14:13:16 2018 +0300

add python-enum34 for vrouter-netns

eventlet needs it but doesn't have in its dependencies

Change-Id: Iab97b7f595edcc798478383575d0972edbb7bb64
Closes-Bug: #1783519

[vimal (vappachan)]

This issue is seen on queens-5.0-172

Traceback (most recent call last):
  File "/usr/bin/opencontrail-vrouter-netns", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 635, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 829, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'enum-compat' distribution was not found and is required by eventlet

[Andrey Pavlov (apavlov-e)]

Outout for ocata has python-enum34:

2018-08-02 08:52:21.394910 | builder | ---> Package python-opencontrail-vrouter-netns.x86_64 0:5.0-172.el7 will be installed
2018-08-02 08:52:21.396193 | builder | --> Processing Dependency: python-websocket-client >= 0.32.0 for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64
2018-08-02 08:52:21.398369 | builder | --> Processing Dependency: python-docker-py >= 1.4.0 for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64
2018-08-02 08:52:21.401955 | builder | --> Processing Dependency: python-unittest2 for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64
2018-08-02 08:52:21.403211 | builder | --> Processing Dependency: python-pyOpenSSL for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64
2018-08-02 08:52:21.404462 | builder | --> Processing Dependency: python-keystoneclient for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64
2018-08-02 08:52:21.405893 | builder | --> Processing Dependency: python-eventlet for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64
2018-08-02 08:52:21.407108 | builder | --> Processing Dependency: python-enum34 for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64
2018-08-02 08:52:21.409196 | builder | --> Processing Dependency: python-barbicanclient for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64

Outout for queens doesn't have python-enum34 in dependencies:
2018-08-02 08:52:08.302161 | builder | ---> Package python-opencontrail-vrouter-netns.x86_64 0:5.0-172.el7 will be installed
2018-08-02 08:52:08.303907 | builder | --> Processing Dependency: python-websocket-client >= 0.32.0 for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64
2018-08-02 08:52:08.306047 | builder | --> Processing Dependency: python-docker-py >= 1.4.0 for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64
2018-08-02 08:52:08.310982 | builder | --> Processing Dependency: python-unittest2 for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64
2018-08-02 08:52:08.312257 | builder | --> Processing Dependency: python-keystoneclient for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64
2018-08-02 08:52:08.313462 | builder | --> Processing Dependency: python-eventlet for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64
2018-08-02 08:52:08.314577 | builder | --> Processing Dependency: python-barbicanclient for package: python-opencontrail-vrouter-netns-5.0-172.el7.x86_64

[Wojciech Urbański (woid)]

This package is already installed in a base container for queens, see logs here: http://logs.opencontrail.org/periodic-nightly/review.opencontrail.org/R5.0/176/contrail-vnc-build-containers-centos7-queens/ara/result/34cfd5f1-a058-4e22-bb94-6c9c04901213/

--> Processing Dependency: python2-cffi >= 1.7 for package: python2-cryptography-2.1.4-1.el7.x86_64
--> Processing Dependency: python2-asn1crypto >= 0.21 for package: python2-cryptography-2.1.4-1.el7.x86_64
--> Processing Dependency: python-enum34 for package: python2-cryptography-2.1.4-1.el7.x86_64

[Andrey Pavlov (apavlov-e)]

It works for me on Ubuntu 18.04 with containers that I built from packages from master-218-ocata build.

If it's specific to rhel packaging then please provide env.

[vimal (vappachan)]

This issue is seen in queens-5.0-176

/usr/bin/opencontrail-vrouter-netns create source-nat b2e77959-1c62-4e1d-9952-37895746cd89 d9ebaa1e-a7a0-4f92-9f3f-750460c56534 5bce3098-5d2d-4899-879a-1fbad12a0b8a --vmi-left-ip 100.64.0.4/29 --vmi-right-ip 10.204.220.203/29 --vmi-left-mac 02:dd:61:60:b8:54 --vmi-right-mac 02:f4:02:73:ec:e9 --gw-ip 100.64.0.6

Traceback (most recent call last):
  File "/usr/bin/opencontrail-vrouter-netns", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 635, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 829, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'enum-compat' distribution was not found and is required by eventlet

[Andrey Pavlov (apavlov-e)]

@vimal, is it a RHOSP setup?

[Andrey Pavlov (apavlov-e)]

eventlet of version 0.20.x and 0.21.x requires enum-compat in setup.py.
enum-compat is not available in RedHat repos and in openstack-ocata repos (CentOS)

there are two solutions:

1) patch eventlet of used version 0.20.1 and store it into our tpc repo (remove enum-compat from setup.py or change it to enum34 that is present as a package and present in teh system)

2) use eventlet of version 0.18.2 that works well in Ocata

[Andrey Pavlov (apavlov-e)]

Ocata uses python2-eventlet.noarch 0:0.18.4-2.el7 from openstack-ocata repo in CentOS

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/45357
Submitter: Andrey Pavlov (<email address hidden>)

[Andrey Pavlov (apavlov-e)]

similar issue https://bugzilla.redhat.com/show_bug.cgi?id=1432165
so another solution is to ask RedHat to rebuild eventlet without enum-compat in its setup.py

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/45408
Submitter: Andrey Pavlov (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/45357
Committed: http://github.com/Juniper/contrail-packages/commit/94d2546d3064d1ddecab94b0d181d0b7d30f6228
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 94d2546d3064d1ddecab94b0d181d0b7d30f6228
Author: Andrey Pavlov <email address hidden>
Date: Tue Aug 7 20:18:59 2018 +0300

use more specific version of eventlet for vrouter-netns

Change-Id: I7700d446e72c942b73852c3e1657929a03f8f827
Closes-Bug: #1783519

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/45408
Committed: http://github.com/Juniper/contrail-packages/commit/32750eb6c2be596d286f60efa7fc27387c1fac28
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 32750eb6c2be596d286f60efa7fc27387c1fac28
Author: Andrey Pavlov <email address hidden>
Date: Tue Aug 7 20:18:59 2018 +0300

use more specific version of eventlet for vrouter-netns

Change-Id: I7700d446e72c942b73852c3e1657929a03f8f827
Closes-Bug: #1783519