Install/Restore: AIO-SX IPV6 bootstrap fails - RTNETLINK answers: No route to host

[Senthil Mukundakumar]

Brief Description
-----------------

AIO-SX restore operation failed due to the following error.

TASK [bootstrap/bringup-essential-services : Add loopback interface] ***********************************
changed: [localhost] => (item=source /etc/platform/openrc; system host-if-add controller-0 lo virtual none lo -c platform -m 1500)
changed: [localhost] => (item=source /etc/platform/openrc; system interface-network-assign controller-0 lo mgmt)
changed: [localhost] => (item=source /etc/platform/openrc; system interface-network-assign controller-0 lo cluster-host)
changed: [localhost] => (item=ip addr add abcd:205::2/64 brd abcd:205::ffff:ffff:ffff:ffff dev lo scope host label lo:5)
changed: [localhost] => (item=ip addr add abcd:204::2/64 brd abcd:204::ffff:ffff:ffff:ffff dev lo scope host label lo:1)
changed: [localhost] => (item=ip addr add 192.168.202.1/24 dev lo scope host)
changed: [localhost] => (item=ip addr add abcd:205::1/64 dev lo scope host)
changed: [localhost] => (item=ip addr add abcd:204::4/64 dev lo scope host)
changed: [localhost] => (item=ip addr add abcd:204::5/64 dev lo scope host)

TASK [bootstrap/bringup-essential-services : Fail if adding interface addresses failed for reason other than it has been done before] ***

TASK [bootstrap/bringup-essential-services : Remove previous management floating address if management network config has changed] ***

TASK [bootstrap/bringup-essential-services : Remove existing /etc/hosts] *******************************
changed: [localhost]

TASK [bootstrap/bringup-essential-services : Populate /etc/hosts] **************************************

TASK [bootstrap/bringup-essential-services : Update /etc/hosts for subcloud] ***************************

TASK [bootstrap/bringup-essential-services : Save hosts file to permanent location] ********************

TASK [bootstrap/bringup-essential-services : Strip the leading slash from config_permdir and assign it to a new variable] ***
ok: [localhost]

TASK [bootstrap/bringup-essential-services : restore /etc/hosts file] **********************************
changed: [localhost]

TASK [bootstrap/bringup-essential-services : restore hosts in config permdir] **************************
changed: [localhost]

TASK [bootstrap/bringup-essential-services : Update name service caching server] ***********************
changed: [localhost]

TASK [bootstrap/bringup-essential-services : Set up default route to the oam gateway] ******************
included: /usr/share/ansible/stx-ansible/playbooks/roles/bootstrap/bringup-essential-services/tasks/setup_default_route.yml for localhost

TASK [bootstrap/bringup-essential-services : Set the ip command based on IP address version] ***********
ok: [localhost]

TASK [bootstrap/bringup-essential-services : Check if the default route exists] ************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "ip -6 route show | grep 'default via 2620:10a:a001:a103::6:0'", "delta": "0:00:00.003375", "end": "2020-02-19 16:41:35.526617", "msg": "non-zero return code", "rc": 1, "start": "2020-02-19 16:41:35.523242", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
...ignoring

TASK [bootstrap/bringup-essential-services : Configure the default route] ******************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "ip -6 route replace default via 2620:10a:a001:a103::6:0", "delta": "0:00:00.002915", "end": "2020-02-19 16:41:35.664827", "msg": "non-zero return code", "rc": 2, "start": "2020-02-19 16:41:35.661912", "stderr": "RTNETLINK answers: No route to host", "stderr_lines": ["RTNETLINK answers: No route to host"], "stdout": "", "stdout_lines": []}

PLAY RECAP *********************************************************************************************
localhost : ok=229 changed=100 unreachable=0 failed=1

Severity
--------
Critical - Restore operation fails

Steps to Reproduce
------------------
1. Backup the file using ansible backup cli.
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/backup.yml -e "ansible_become_pass=Li69nux* admin_password=Li69nux*"
2. FTP backup file
3. Reinstall the AIO-SX
4. Restore the backup file using below cli.
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/restore_platform.yml -e "backup_filename=localhost_platform_backup_2020_01_21_18_30_21.tgz admin_passwori69nux* ansible_become_pass=Li69nux* initial_backup_dir=/home/sysadmin"

Expected Behavior
------------------
Restore completed

Actual Behavior
----------------
Restore was not successful.

Reproducibility
---------------
100% reproducible in AIO-SX system

System Configuration
--------------------
AIO-SX wildcat-112

Branch/Pull Time/Commit
-----------------------

BUILD_ID="2020-02-18_10-46-00"

Timestamp/Logs
--------------
2020-02-19T16:39:02.437

Test Activity
-------------
Regression

[Ghada Khalil (gkhalil)]

stx.4.0 / medium priority - backup & restore issue.

We believe this was last tested in stx.3.0

[Frank Miller (sensfan22)]

This issue is no longer seen in recent B&R tests and I believe unrelated code changes were done to address this issue. Request the originator re-test and confirm.

[Senthil Mukundakumar (smukunda)]

The issue is reproducible using wcp_112 using load 2020-03-25_21-02-05. The lab was installed with platform+stx-monitor. Restored with wipe_osd_disk default operation false.

[Senthil Mukundakumar (smukunda)]

Verified using wcp_112 using 2020-04-25_13-17-56