Nova Unable to ssh VM via NatBox client

[Yvonne Ding]

Brief Description
-----------------
Ssh to VM mgmt net via NatBox client is refused.

Severity
--------
Major

Steps to Reproduce
------------------
1. Create a flavor with 1 vcpus, 0G ephemera disk, 512M swap disk
2. Add extra specs: {'hw:cpu_policy': 'dedicated'}
3. Boot a vm from image
4. Auto mount ephemeral, swap, and attached volume

TC-name:
test_migrate_vms.py::test_cold_migrate_vm[local_image-0-512-dedicated-1-image-confirm]

Expected Behavior
-----------------
Ssh to compute-1 via NatBox client success

Actual Behavior
----------------
Ssh to compute-1 via NatBox client failed

Reproducibility
---------------
Intermittent

System Configuration
--------------------
Regular standard 2+2

Lab-name:
ip_1_4

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20210226T024233Z"

Timestamp/Logs
--------------
[2021-03-07 05:36:33,449] 1300 INFO MainThread ssh.connect :: Attempt to connect to 192.168.101.23 from 128.224.186.181...
[2021-03-07 05:36:33,449] 314 DEBUG MainThread ssh.send :: Send 'ssh -v -o RSAAuthentication=no -o PubkeyAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@192.168.101.23'
[2021-03-07 05:36:33,572] 436 DEBUG MainThread ssh.expect :: Output:
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 192.168.101.23 [192.168.101.23] port 22.
debug1: connect to address 192.168.101.23 port 22: Connection refused
ssh: connect to host 192.168.101.23 port 22: Connection refused
svc-cgcsauto@tis-lab-nat-box:~$
[2021-03-07 05:38:24,000] 1341 INFO MainThread ssh.connect :: Unable to ssh to 192.168.101.23
[2021-03-07 05:38:24,000] 1357 INFO MainThread ssh.connect :: Retry in 10 seconds
[2021-03-07 05:38:37,015] 351 DEBUG MainThread ssh.flush :: Buffer is flushed by reading out the rest of the output
[2021-03-07 05:38:37,016] 2674 WARNING MainThread vm_helper.ssh_to_vm_from_natbox:: Failed to ssh to VM a40a1f77-1123-43b6-8a74-70aa8aff3e58! Collecting vm console log

logs of .tar and TIS_AUTOMATION.log as below,
https://files.starlingx.kube.cengn.ca/launchpad/
Note: above url is currently not accessible

Test Activity
-------------
STX Nova regression

[Frank Miller (sensfan22)]

Screening note: This may be an issue with the NAT box and not with nova so marking gating for the next release.

[Austin Sun (sunausti)]

Hi Douglas and Frank:
    do we have conclusion for this issue ? is it nat box issue?

[Ghada Khalil (gkhalil)]

I'm closing this LP due to lack of activity. Note: The reporter is no longer at WR.