[ Build 4.0-10 ]:K8S : Ingress netns creation is failing due to permission issue

[chhandak]

Build 4.0-10
base os ubuntu 16.06 container ubuntu16.04
K8S mode Baremetal

Description:
While creating ingress corresponding netns instance is not getting created in agent container. The following traceback is observed. required permission is missing.

Traceback
------------
From: http://10.87.121.36:8085/Snh_ServiceInstanceReq?uuid=

Traceback (most recent call last):
  File "/usr/bin/opencontrail-vrouter-netns", line 9, in <module>
    load_entry_point('opencontrail-vrouter-netns==0.1', 'console_scripts', 'opencontrail-vrouter-netns')()
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/vrouter_netns.py", line 500, in main
    vrouter_netns.args.func()
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/vrouter_netns.py", line 453, in create
    if (netns_mgr.set_lbaas() == False):
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/vrouter_netns.py", line 160, in set_lbaas
    self.create()
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/vrouter_netns.py", line 103, in create
    ip.ensure_namespace(self.namespace)
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/linux/ip_lib.py", line 137, in ensure_namespace
    ip = self.netns.add(name)
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/linux/ip_lib.py", line 489, in add
    self._as_root('add', name, use_root_namespace=True)
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/linux/ip_lib.py", line 220, in _as_root
    kwargs.get('use_root_namespace', False))
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/linux/ip_lib.py", line 69, in _as_root
    namespace)
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/linux/ip_lib.py", line 80, in _execute
    root_helper=root_helper)
  File "/usr/lib/python2.7/dist-packages/opencontrail_vrouter_netns/linux/utils.py", line 86, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'ip', 'netns', 'add', 'vrouter-253b4c2e-c8ef-4f14-9ed7-d774c68b599c:cbe173ca-3f8b-11e7-9cd7-0cc47aa89e64']
Exit code: 1
Stdout: ''
Stderr: 'mkdir /var/run/netns failed: Read-only file system\n'

[Hari Prasad Killi (haripk)]

Please check if the following works:
In /lib/systemd/system/contrail-vrouter-agent.service, add the following line and restart the service / container.
ReadWriteDirectories=-/var/run/netns

[chhandak (chhandak)]

1. Updated the file with below value.
2. Restarted the agent container. (docket agent restart )

Still seeing the same issue

Conf file
-----------
root@5b7s20(agent):/# cat /lib/systemd/system/contrail-vrouter-agent.service
[Unit]
Description=Contrail vrouter agent service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/contrail-vrouter-agent
PIDFile=/var/run/contrail/contrail-vrouter-agent.pid
TimeoutStopSec=0
Restart=always
ExecStop=/bin/kill -s TERM $MAINPID
PrivateTmp=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/log/contrail
ReadWriteDirectories=-/var/lib/contrail
ReadWriteDirectories=-/dev
ReadWriteDirectories=-/var/run/netns >> Suggested config

[Install]
WantedBy=multi-user.target

[Yuvaraja Mariappan (ymariappan)]

It is working fine in u14.04.

[Yuvaraja Mariappan (ymariappan)]

workaround for ubuntu16.04
---------------------------
1. add "ReadWriteDirectories=-/run" in /lib/systemd/system/contrail-vrouter-agent.service
2. run "systemctl daemon-reload"
3. run "systemctl restart contrail-vrouter-agent.service"

Thanks,
Yuvaraja

[chhandak (chhandak)]

Tried the steps Yuvaraja mentioned (Giving global permission to run folder). It is working fine with that.

2017-05-23 11:33:01,258 - INFO - Responses seen from all pods, lb seems fine.Hits : {'ctest-nginx-pod-09243501': 1, 'ctest-nginx-pod-64242933': 1}
2017-05-23 11:33:01,259 - INFO - Deleting pod default:ctest-busybox-pod-34546908
2017-05-23 11:33:01,337 - WARNING - Pod uuid 3a645b26-3fe6-11e7-9cd7-0cc47aa89e64 is still seen in agent 10.87.121.36 VM list
2017-05-23 11:33:06,352 - INFO - Verified that pod ctest-busybox-pod-34546908 is removed in agent
2017-05-23 11:33:06,352 - INFO - Deleting Ingress : ctest-nginx-ingress-14444607
2017-05-23 11:33:06,370 - INFO - Deleting pod default:ctest-nginx-pod-09243501
2017-05-23 11:33:06,577 - WARNING - Pod uuid 388cc486-3fe6-11e7-9cd7-0cc47aa89e64 is still seen in agent 10.87.121.36 VM list
2017-05-23 11:33:11,592 - INFO - Verified that pod ctest-nginx-pod-09243501 is removed in agent
2017-05-23 11:33:11,593 - INFO - Deleting pod default:ctest-nginx-pod-64242933
2017-05-23 11:33:11,647 - WARNING - Pod uuid 36bd782b-3fe6-11e7-9cd7-0cc47aa89e64 is still seen in agent 10.87.121.36 VM list
2017-05-23 11:33:16,663 - INFO - Verified that pod ctest-nginx-pod-64242933 is removed in agent
2017-05-23 11:33:16,663 - INFO - Deleting service : ctest-nginx-svc-57943968
2017-05-23 11:33:16,960 - INFO - END TEST : test_ingress_1 : PASSED[0:00:32]
2017-05-23 11:33:16,961 - INFO - --------------------------------------------------------------------------------

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/32020
Submitter: Yuvaraja Mariappan

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/32021
Submitter: Yuvaraja Mariappan

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/32021
Committed: http://github.com/Juniper/contrail-controller/commit/2cfcff64b6587fd47b1884cc15d84a660bc917dc
Submitter: Zuul (<email address hidden>)
Branch: master

commit 2cfcff64b6587fd47b1884cc15d84a660bc917dc
Author: Yuvaraja Mariappan <email address hidden>
Date: Tue May 23 11:59:27 2017 -0700

Fixed netns creation issue in ubuntu16.04

Netns script need proper permssion to
create directories in /var/run-->/run
Has given rw permission to /run

Change-Id: I9fa42a8a9f5fd048f1584689a51de09a6cbc771b
Closes-bug: #1692832

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/32020
Submitter: Yuvaraja Mariappan

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/32020
Committed: http://github.com/Juniper/contrail-controller/commit/4d4cfe4ae31d47a710ede3b0405c82b16c9e8114
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 4d4cfe4ae31d47a710ede3b0405c82b16c9e8114
Author: Yuvaraja Mariappan <email address hidden>
Date: Tue May 23 21:19:27 2017 -0700

Fixed netns creation issue in ubuntu16.04

Netns script need proper permssion to
create directories in /var/run-->/run
Has given rw permission to /run

Change-Id: I9fa42a8a9f5fd048f1584689a51de09a6cbc771b
Closes-bug: #1692832