Sanity : Infra scripts not Exception handling correctly inside sshpass

[Sarath]

As inside "sshpass" when batching multiple cmds, whatever individual remote cmds executed can't be verified return value. So, this means if there is real Exception raised due to individual remote cmd failure, current infra scripts with "sshpass" batched cmds will break causing unreliable Sanity failures/issues

Below are some of infra scripts which has this kind of usage in multiple places and all of the current Sanity testbeds using this Infra code,
 >> testers/utils
 >> testers/smlite_utils
 >> testers/smgr_utils

Possible Solution: As this reference on multiple of Infra-scripts, either change all of the references to be single remote-cmd sshpass where return value needs to be checked for Exception handling (or) some proc need to be designed/developed which takes the batch of remote-cmds and verify them for Exception handling

Also, below the email which details this as per debugging outputs,

######################################################################################################
From: Sarathbabu Narasimhan
Sent: Tuesday, October 24, 2017 2:01 AM
To: Sandip Dey <email address hidden>; Pavana Prakash <email address hidden>
Cc: Sudheendra Rao <email address hidden>; Jeba Paulaiyan <email address hidden>; Sachchidanand Vaidya <email address hidden>
Subject: RE: Vcenter-only Jenkins integration with SM provisioning

Okay, to keep all sync of this update,
  On debugging this failure, I found my new code exposed the existing automation-infra-bug which relies on Exception-handling by checking the exit-status of earlier command for success.
The existing bug is when this done inside “sshpass” structure it actually not checking earlier command inside “sshpass” per script expectations and instead it checks the earlier command outside to “sshpass”
which is actually Echo/print statement which always returns constant “0/Pass” previously but when I added new logic to count the tags instead of Echo, it got other value and so the sanity failures…

Note: To fix existing bug, it involves efforts either to find ways to change “sshpass” structure/procs (or) modifying multiple of scripts using this to parse single lines at a time for sshpass.
         So, for now I too used the hack by keeping “echo” earlier to sshpass in my new-code to get sanity going without issues for all testbeds.

    echo "Connect to SM_SERVER to reimage the targets" >>>>>>>>>>>>>>>>>>> dummy echo cmd
    sshpass -p "c0ntrail123" ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -l root $SM_SERVER_IP " (
        set -e
        echo "reimage the target setup"
        server-manager-client reimage -F --cluster_id $CLUSTER_NAME ${image_id} >>>>> impacted remote cmd
        if [ $? == 0 ]
        then
            echo "Issue of Reimage command to reimage target nodes is sucessful!!!"
        else
            echo "Issue of Reimage command to reimage target nodes failed, aborting the process"
            exit 1
        fi
        echo "reimage in progress please wait ..."
        sleep 10
        #check if reimage is completed.
        /root/sm_files/check_reimage_cliff.sh $CLUSTER_NAME >>>>> impacted remote cmd
        if [ $? == 0 ]
        then
            echo "reimaged the target $CLUSTER_NAME successfully"
        else
            echo "reimage of target $CLUSTER_NAME failed!!!"
            exit 1
        fi
    ) "