'system host-add' error msg is cryptic when mgmt_mac is not provided

[mhg]

Brief Description
-----------------
Attempt to add a new host/node to a system using CLI "system host-add -n compute-0 -p worker -l 'manual-added-compute-0'" failed.
The CLI produced messgage:
'NoneType' object has no attribute 'rstrip'
The return code is 1

Checking the usage of the CLI, could not find any problematic usage:
    usage: system host-add [-n <hostname>] [-p <personality>] [-s <subfunctions>]
                        [-m <mgmt_mac>] [-i <mgmt_ip>] [-I <bm_ip>]
                        [-T <bm_type>] [-U <bm_username>] [-P <bm_password>]
                        [-b <boot_device>] [-r <rootfs_device>]
                        [-o <install_output>] [-c <console>]
                        [-v <vsc_controllers>] [-l <location>]
                        [-D <true/false>]
Note: Although there is no CLI 'system host-modify' any more, users still can alternatively use 'Edit Host' on Horizon to provision the node.

Severity
--------
Major

Steps to Reproduce
------------------
....
system host-add -n compute-0 -p worker -l 'manual-added-compute-0'

TC-name:

Expected Behavior
------------------
The host/node was successfully added into the system

Actual Behavior
----------------
The CLI failed with messages and return code was 1.
No new host/node with the specified name was added into the system host-list

Reproducibility
---------------
Reproducible

System Configuration
--------------------
found on Multi-node system IPv4 system, but it was likely to happen on all types (except AIO-SX)

Lab-name:
ip-1-4

Branch/Pull Time/Commit
-----------------------
stx master as of 20190506T233000Z

Last Pass
---------

Timestamp/Logs
--------------
20190508 14:45:48

Test Activity
-------------
Feature Testing

[John Kung (john-kung)]

Issue is reproducible. The following AttributeError occurs before sysinv-api performs the validate_and_normalize_mac() which would have raised an exception for the missing MAC address. The client needs to provide the proper mgmt_mac address for the new host.

2019-05-08 20:47:54.112 223748 ERROR wsme.api [-] Server-side error: "'NoneType' object has no attribute 'rstrip'". Detail:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/wsmeext/pecan.py", line 85, in callfunction
    result = f(self, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/sysinv/api/controllers/v1/host.py", line 1717, in post
    return self._do_post(ihost_dict)
  File "/usr/lib64/python2.7/site-packages/sysinv/api/controllers/v1/host.py", line 1346, in _do_post
    ihost_dict['mgmt_mac'])
  File "/usr/lib64/python2.7/site-packages/sysinv/objects/__init__.py", line 101, in wrapper
    result = fn(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/sysinv/db/sqlalchemy/api.py", line 1297, in ihost_get_by_mgmt_mac
    mgmt_mac = mgmt_mac.rstrip()
AttributeError: 'NoneType' object has no attribute 'rstrip'

[Ghada Khalil (gkhalil)]

Marking as release gating; this would prevent the addition of new nodes to the system.
Expect this to be part of typical initial system configuration for multi-node systems. Unclear when this issue started occurring.

[Ghada Khalil (gkhalil)]

Lowering priority based on discussion with David Sullivan. The issue here is the error message is not clear. If the user provides the proper parameters to the add-host cmd, the command passes.

[Bill Zvonar (billzvonar)]

Assigned to Bruce for re-assignment.

[Abraham Arce (xe1gyq)]

I am taking the ownership of this bug to solve, after a initial review I was able to reproduce, here is my first set of questions to get mode guidance into the right approach to solve it.

To avoid having this cryptic msg, are we expecting a code change to notify mgmt_mac is required when executing 'system host-add' as it is found under the following files?

- stx-config/.../sysinv/conductor/manager.py
- stx-metal/.../inventory/conductor/manager.py

  if 'mgmt_mac' not in values:
    raise exception.SysinvException(_(
      "Invalid method call: create_ihost requires mgmt_mac."))

If the above code change is true, is the following file the right place to include this check?

- stx-config/.../sysinv/api/controllers/v1/host.py

This assumes calls from 'system host-add ...' and API are both covered, is this a right assumption?

[Abraham Arce (xe1gyq)]

Email sent to Yong

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/674875

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.2.0
Review: https://review.opendev.org/678081

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/674875
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=46dbbd2cc3963602bb2d73083b403c6c470c6d6c
Submitter: Zuul
Branch: master

commit 46dbbd2cc3963602bb2d73083b403c6c470c6d6c
Author: Abraham Arce <email address hidden>
Date: Tue Aug 6 11:48:52 2019 -0500

    SysInv: Host Add MAC Address Message

    host-add command error message is not clear when user provides no
    MAC Address of the host mgmt interface, it is a [REQUIRED] parameter.

    Tested with a Controller Storage Configuration:

    $ system host-list
    +----+--------------+-------------+----------------+-------------+
    | id | hostname | personality | administrative | operational |
    +----+--------------+-------------+----------------+-------------+
    | 5 | None | None | locked | disabled |
    +----+--------------+-------------+----------------+-------------+

    $ system host-add -n compute-0 -p worker
    Host-add Rejected: Must provide MAC Address of the host mgmt interface

    $ system host-add -n compute-1 -p worker -m 52:54:00:59:02:9
    Host-add Rejected: Must provide a valid format of a MAC Address

    $ system host-add -n compute-1 -p worker -m 52:54:00:59:02:95
    +---------------------+--------------------------------------+
    | Property | Value |
    +---------------------+--------------------------------------+
    | hostname | compute-1 |
    | mgmt_ip | 192.168.204.10 |
    | mgmt_mac | 52:54:00:59:02:95 |
    | personality | worker |
    +---------------------+--------------------------------------+

    Closes-Bug: 1828247

    Change-Id: I086df1d0c716992808e10b0fa627109dff0294ad
    Cc: John Kung <email address hidden>
    Signed-off-by: Abraham Arce <email address hidden>

[Frank Miller (sensfan22)]

Since this issue is only the error message is not clear, this should not be stx.2.0 gating but instead stx.3.0 gating. Changing the tag to be stx.3.0 gating.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Abraham Arce (<email address hidden>) on branch: r/stx.2.0
Review: https://review.opendev.org/678081
Reason: Regated to stx.3.0.

[John Kruszewski (jiggernaut)]

# Testing Status
PASSED
- host-add command is rejected with the expected message if the mgmt MAC is missing or incorrect format

# Configuration
2 + 2 Controller Storage Config

# Load Tested
2019-09-05_00-10-00

# Test host-add mgmt MAC with controller node

[sysadmin@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | None | None | locked | disabled | offline |
+----+--------------+-------------+----------------+-------------+--------------+

[sysadmin@controller-0 ~(keystone_admin)]$ system host-add -p controller
Host-add Rejected: Must provide MAC Address of the host mgmt interface

[sysadmin@controller-0 ~(keystone_admin)]$ system host-add -p controller -n controller-1
Host-add Rejected: Must provide MAC Address of the host mgmt interface

[sysadmin@controller-0 ~(keystone_admin)]$ system host-add -p controller -o graphical -c tty1
Host-add Rejected: Must provide MAC Address of the host mgmt interface

[sysadmin@controller-0 ~(keystone_admin)]$ id=`system host-list | grep -i none | awk '{print $2}'`
[sysadmin@controller-0 ~(keystone_admin)]$ system host-show ${id} | grep mgmt_mac | awk '{print $4}'
00:1e:67:38:bc:91

[sysadmin@controller-0 ~(keystone_admin)]$ system host-add -p controller -m 00:1e:67:38:bc:9 -o graphical -c tty1
Host-add Rejected: Must provide a valid format of a MAC Address

[sysadmin@controller-0 ~(keystone_admin)]$ system host-add -p controller -m ABCDEFGHIJKLMNOPQ -o graphical -c tty1
Host-add Rejected: Must provide a valid format of a MAC Address

[sysadmin@controller-0 ~(keystone_admin)]$ system host-add -p controller -n controller-1
Host-add Rejected: Must provide MAC Address of the host mgmt interface

[sysadmin@controller-0 ~(keystone_admin)]$ system host-add -p controller -n controller-1 -o graphical -c tty1
Host-add Rejected: Must provide MAC Address of the host mgmt interface

[sysadmin@controller-0 ~(keystone_admin)]$ system host-add -p controller -n controller-1 -o text -c tty1
Host-add Rejected: Must provide MAC Address of the host mgmt interface

[sysadmin@controller-0 ~(keystone_admin)]$ system host-add -p controller -n controller-1 -m 00:1e:67:38:bc:9 -o text -c tty1
Host-add Rejected: Must provide a valid format of a MAC Address

[sysadmin@controller-0 ~(keystone_admin)]$ system host-add -p controller -n controller-1 -m 00:1e:67:38:bc:91 -o text -c tty1
+---------------------+-----------------------------------------+
| Property | Value |
+---------------------+-----------------------------------------+
....
| capabilities | {u'Personality': u'Controller-Standby'} |
| hostname | controller-1 |
| install_output | text |
| mgmt_ip | 192.168.204.4 |
| mgmt_mac | 00:1e:67:3...