SSH / SCP to VM failed using NAMESPACE

[Ricardo Perez]

Brief Description
-----------------
In a Duplex configuration, while using the NAMESPACE to perform SSH or SCP from the compute to the VM, this is no possible. Port 22 Connection Timed out / lost connection error message is seen.

Severity
--------
Provide the severity of the defect.
<Critical: System/Feature is not usable due to the defect>

Steps to Reproduce
------------------

1.- Follow the steps described here to set up a Duplex Configuration:
https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnAIODX

2.- Add the following property to the flavor that you are going to use to create VMs:
openstack flavor list
openstack flavor show <Specifc_Flavor_for_VM_Creation>
openstack flavor set <Flavor_ID> --property hw:mem_page_size=large

3.- Create an image
openstack mage create --container-format bare --disk-format qcow2 --file cirros-0.4.0-x86_64-disk.img cirros

4.- Create a VM
openstack server create --image cirros --flavor my_tiny --network public-net0 richo1

5.- Perform the following commands to perform the NAMESPACE commands:

controller-0:~# IP=`openstack server list --name richo1 -f value -c Networks | awk '{ split($1, v, "="); print v[2]}'`

controller-0:~# NAMESPACE=$(ip netns | grep $(neutron net-list --name public-net0 -f value -c id))

controller-0:~# sudo ip netns exec $NAMESPACE scp <file_name> cirros@$IP:~/

Expected Behavior
------------------
You should be able to perform SSH / SCP using NAMESPACE

Actual Behavior
----------------
No route to host / lost connection message is seen in the console when trying to perform SSH / SCP using NAMESPACE

Reproducibility
---------------
<Reproducible/Intermittent/Seen once>
The issue is 100% reproducible

System Configuration
--------------------
<Two node system (Duplex)>

Branch/Pull Time/Commit
-----------------------

controller-0:~# cat /etc/build.info
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190715T233000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="182"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-07-15 23:30:00 +0000"

Last Pass
---------
October Release (2018) / Possibly in the first 2019 releases.

Timestamp/Logs
--------------
IP=`openstack server list --name $VM_NAME -f value -c Networks | awk '{ split($1, v, "="); print v[2]}'``

NAMESPACE=$(ip netns | grep $(neutron net-list --name $NET_NAME -f value -c id))

sudo ip netns exec $NAMESPACE scp <file> $USER@$IP:~/

controller-0:~# neutron net-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+---------------+----------------------------------+-------------------------------------------------------+
| id | name | tenant_id | subnets |
+--------------------------------------+---------------+----------------------------------+-------------------------------------------------------+
| 4275c17e-3e7e-4832-88d7-054af8559251 | private-net0 | 744e785dd01141c58d7830e374c99cc9 | 06563e72-52eb-4909-a6e5-5d4642f1cba4 192.168.201.0/24 |
| 89ede0af-f03f-4fd4-ade4-65236e1b3b0c | external-net0 | 744e785dd01141c58d7830e374c99cc9 | 20b21126-d6c7-4b8d-b1f4-acc8e9ff33d3 192.168.51.0/24 |
| c0fa315a-d187-44f5-b75a-43dc7fae8e41 | internal-net0 | 744e785dd01141c58d7830e374c99cc9 | ff7d347f-978f-4e1c-b596-4dce578d7120 10.1.1.0/24 |
| f3620ff1-6306-449a-abb9-fc66aa9de716 | public-net0 | 744e785dd01141c58d7830e374c99cc9 | ca69d82b-dca5-4dcc-881e-e0d1261d86d5 192.168.101.0/24 |
+--------------------------------------+---------------+----------------------------------+-------------------------------------------------------+
controller-0:~# openstack server list
+--------------------------------------+------------+--------+-----------------------------+--------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------------+--------+-----------------------------+--------+---------+
| 242991a4-6d43-4553-b5ef-8d6817528dbd | richo_key1 | ACTIVE | public-net0=192.168.101.189 | | my_tiny |
| 629487f6-fca9-430e-95ac-77caf23f1c78 | richo2 | ACTIVE | public-net0=192.168.101.237 | cirros | my_tiny |
| ca0f9090-bcb0-4fda-84e5-335eee435ae8 | richo1 | ACTIVE | public-net0=192.168.101.109 | cirros | my_tiny |
+--------------------------------------+------------+--------+-----------------------------+--------+---------+
controller-0:~# IP=`openstack server list --name richo1 -f value -c Networks | awk '{ split($1, v, "="); print v[2]}'`
controller-0:~# echo $IP
192.168.101.109
controller-0:~# NAMESPACE=$(ip netns | grep $(neutron net-list --name public-net0 -f value -c id))
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
controller-0:~# echo $NAMESPACE
qdhcp-f3620ff1-6306-449a-abb9-fc66aa9de716 (id: 26)
controller-0:~# NAMESPACE=qdhcp-f3620ff1-6306-449a-abb9-fc66aa9de716
controller-0:~# sudo ip netns exec $NAMESPACE scp /home/sysadmin/^CUSER@$IP:~/
controller-0:~# sudo ip netns exec $NAMESPACE scp /home/sysadmin/cirros-0.4.0-x86_64-disk.img cirros@$IP:~/
ssh: connect to host 192.168.101.109 port 22: Connection timed out (after ~1 minute you will see the message)
lost connection
controller-0:~#

Test Activity
-------------
[Regression Testing]

[Ghada Khalil (gkhalil)]

Previously, in stx.1.0, this was not expected to work. The following bugs are marked as Invalid:
https://bugs.launchpad.net/starlingx/+bug/1797217
https://bugs.launchpad.net/starlingx/+bug/1799591

I am unsure if namespaces are expected to work now that we are using upstream neutron. Assigning to the networking team for input.

[Ghada Khalil (gkhalil)]

This may be a neutron upstream issue given stx is aligned with stein with no additional stx-specific neutron patches. Follow-up is likely required with the neutron team. Marking as stx.3.0 gating as this is not a high issue.

[YaoLe (yaole)]

Hi, Ricardo Perez

You use this command:
  openstack flavor set <Flavor_ID> --property hw:mem_page_size=large
is that means you use dpdk?

And I cannot see your operations to create a security group, is this issue caused by that?

[YaoLe (yaole)]

Hi, Ricardo Perez

Try to create a new security instead of using the dafault security which doesn't have needed rules:

   openstack security group create security1
   openstack security group rule create --ingress --protocol icmp --remote-ip 0.0.0.0/0 security1
   openstack security group rule create --ingress --protocol tcp --remote-ip 0.0.0.0/0 security1
   openstack security group rule create --ingress --protocol udp --remote-ip 0.0.0.0/0 security1
   openstack server create --image cirros --flavor m1.tiny --network public-net0 --security-group security1

And then test ssh/scp using namespace

[Ricardo Perez (richomx)]

Hi Le Yao,

Thanks for your comments, following the security group creation as you mention. I'm able to do SCP / SSH to an instance. Please see the below output log section:

openstack security group create security1
openstack security group rule create --ingress --protocol icmp --remote-ip 0.0.0.0/0 security1
openstack security group rule create --ingress --protocol tcp --remote-ip 0.0.0.0/0 security1
openstack security group rule create --ingress --protocol udp --remote-ip 0.0.0.0/0 security1

controller-0:~# virsh list
 Id Name State
-----------------------------------
 6 instance-0000000f running

controller-0:~# virsh console 6
Connected to domain instance-0000000f
Escape character is ^]

login as 'cirros' user. default password: 'cubswin:)'. use 'sudo' for root.
richo login: cirros
Password:
$

controller-0:~# sudo ip netns exec $NAMESPACE scp /home/sysadmin/hola.txt cirros@$IP:~/
The authenticity of host '192.168.101.81 (192.168.101.81)' can't be established.
RSA key fingerprint is SHA256:126BpM0NpXIlDPKMHmUVwiWV5f3FK0yX8mxjjkiPQaA.
RSA key fingerprint is MD5:39:d6:8a:3c:d4:5a:5c:65:3e:00:b4:19:e2:9b:ae:01.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.101.81' (RSA) to the list of known hosts.
cirros@192.168.101.81's password:
hola.txt 100% 0 0.0KB/s 00:00
controller-0:~# sudo ip netns exec $NAMESPACE scp /home/sysadmin/cirros-0.3.4-x86_64-disk.img cirros@$IP:~/
cirros@192.168.101.81's password:
cirros-0.3.4-x86_64-disk.img 0% 0 0.0KB/s --:-- ETAscp: /home/cirros//cirros-0.3.4-x86_64-disk.img
cirros-0.3.4-x86_64-disk.img 100% 13MB 14.0MB/s 00:00
controller-0_20190805.113102.tar
controller-0:~# sudo ip netns exec $NAMESPACE ssh cirros@$IP
cirros@192.168.101.81's password:
$

All of this was done in the following StarlingX ISO version:

controller-0:~# cat /etc/build.info
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190802T013000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="200"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-08-02 01:30:00 +0000"
controller-0:~#

Thanks !

[YaoLe (yaole)]

Confirmed by Ricardo Perez, this bug is an issue with lack of steps. So this bug can be marked as invalid.