BMC address is getting replaced by 0.0.0.0 in the ramdisk data received

[ankit]

When doing inspection using ironic-inspector, in the data that is received from ramdisk the bmc address is replaced by 0.0.0.0 and inspection is timing out.
Providing inspector logs

2017-08-28 07:16:23.003 26053 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic_inspector.wsgi_service.periodic_clean_up' _process_scheduled /usr/local/lib/python2.7/dist-packages/futurist/periodics.py:639
2017-08-28 07:16:33.743 26053 DEBUG ironic_inspector.main [-] [node: MAC 9c:b6:54:01:90:2e] Received data from the ramdisk: {u'pci_devices': [{u'vendor_id': u'8086', u'product_id': u'0e80'}, {u'vendor_id': u'8086', u'product_id': u'0e90'}, {u'vendor_id': u'8086', u'product_id': u'0eb0'}, {u'vendor_id': u'8086', u'product_id': u'0eb1'}, {u'vendor_id': u'8086', u'product_id': u'0eb2'}, {u'vendor_id': u'8086', u'product_id': u'0eb3'}, {u'vendor_id': u'8086', u'product_id': u'0eb4'}, {u'vendor_id': u'8086', u'product_id': u'0eb5'}, {u'vendor_id': u'8086', u'product_id': u'0eb6'}, {u'vendor_id': u'8086', u'product_id': u'1d02'}, {u'vendor_id': u'103c', u'product_id': u'3306'}, {u'vendor_id': u'102b', u'product_id': u'0533'}, {u'vendor_id': u'103c', u'product_id': u'3307'}, {u'vendor_id': u'103c', u'product_id': u'3300'}, {u'vendor_id': u'8086', u'product_id': u'1521'}, {u'vendor_id': u'8086', u'product_id': u'1521'}, {u'vendor_id': u'1000', u'product_id': u'0087'}, {u'vendor_id': u'8086', u'product_id': u'0e80'}, {u'vendor_id': u'8086', u'product_id': u'0e90'}, {u'vendor_id': u'8086', u'product_id': u'0eb0'}, {u'vendor_id': u'8086', u'product_id': u'0eb1'}, {u'vendor_id': u'8086', u'product_id': u'0eb2'}, {u'vendor_id': u'8086', u'product_id': u'0eb3'}, {u'vendor_id': u'8086', u'product_id': u'0eb4'}, {u'vendor_id': u'8086', u'product_id': u'0eb5'}, {u'vendor_id': u'8086', u'product_id': u'0eb6'}, {u'vendor_id': u'8086', u'product_id': u'0eb7'}, {u'vendor_id': u'8086', u'product_id': u'0e1d'}, {u'vendor_id': u'8086', u'product_id': u'0e34'}, {u'vendor_id': u'8086', u'product_id': u'0e81'}, {u'vendor_id': u'8086', u'product_id': u'0e2c'}], u'logs': '<hidden>', u'boot_interface': u'01-9c-b6-54-01-90-2e', u'ipmi_address': u'0.0.0.0', u'inventory': {u'bmc_address': u'0.0.0.0', u'interfaces': [{u'lldp': None, u'product': u'0x1521', u'vendor': u'0x8086', u'name': u'eno2', u'has_carrier': False, u'ipv4_address': None, u'biosdevname': None, u'client_id': None, u'mac_address': u'9c:b6:54:01:90:2f'}, {u'lldp': None, u'product': u'0x1521', u'vendor': u'0x8086', u'name': u'eno1', u'has_carrier': True, u'ipv4_address': u'172.17.1.229', u'biosdevname': None, u'client_id': None, u'mac_address': u'9c:b6:54:01:90:2e'}], u'disks': [{u'rotational': True, u'vendor': u'HP', u'name': u'/dev/sda', u'hctl': u'0:0:0:0', u'wwn_vendor_extension': None, u'wwn_with_extension': u'0x5000cca03800c9bc', u'model': u'EF0600FATFF', u'wwn': u'0x5000cca03800c9bc', u'serial': u'5000cca03800c9bc', u'size': 600127266816}], u'boot': {u'current_boot_mode': u'bios', u'pxe_interface': u'01-9c-b6-54-01-90-2e'}, u'system_vendor': {u'serial_number': u'SGH352ADNM', u'product_name': u'ProLiant SL230s Gen8', u'manufacturer': u'HP'}, u'memory': {u'physical_mb': 49152, u'total': 50678013952}, u'cpu': {u'count': 24, u'frequency': u'2600.0000', u'flags': [u'fpu', u'vme', u'de', u'pse', u'tsc', u'msr', u'pae', u'mce', u'cx8', u'apic', u'sep', u'mtrr', u'pge', u'mca', u'cmov', u'pat', u'pse36', u'clflush', u'dts', u'acpi', u'mmx', u'fxsr', u'sse', u'sse2', u'ss', u'ht', u'tm', u'pbe', u'syscall', u'nx', u'pdpe1gb', u'rdtscp', u'lm', u'constant_tsc', u'arch_perfmon', u'pebs', u'bts', u'rep_good', u'nopl', u'xtopology', u'nonstop_tsc', u'aperfmperf', u'eagerfpu', u'pni', u'pclmulqdq', u'dtes64', u'monitor', u'ds_cpl', u'vmx', u'smx', u'est', u'tm2', u'ssse3', u'cx16', u'xtpr', u'pdcm', u'pcid', u'dca', u'sse4_1', u'sse4_2', u'x2apic', u'popcnt', u'tsc_deadline_timer', u'aes', u'x
save', u'avx', u'f16c', u'rdrand', u'lahf_lm', u'epb', u'tpr_shadow', u'vnmi', u'flexpriority', u'ept', u'vpid', u'fsgsbase', u'smep', u'erms', u'xsaveopt', u'dtherm', u'ida', u'arat', u'pln', u'pts'], u'model_name': u'Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz', u'architecture': u'x86_64'}}, u'error': None, u'root_disk': {u'rotational': True, u'vendor': u'HP', u'name': u'/dev/sda', u'hctl': u'0:0:0:0', u'wwn_vendor_extension': None, u'wwn_with_extension': u'0x5000cca03800c9bc', u'model': u'EF0600FATFF', u'wwn': u'0x5000cca03800c9bc', u'serial': u'5000cca03800c9bc', u'size': 600127266816}} api_continue /opt/stack/ironic-inspector/ironic_inspector/main.py:193
2017-08-28 07:16:33.748 26053 DEBUG ironic_inspector.process [-] [node: MAC 9c:b6:54:01:90:2e] Running pre-processing hook ramdisk_error _run_pre_hooks /opt/stack/ironic-inspector/ironic_inspector/process.py:117
2017-08-28 07:16:33.749 26053 DEBUG ironic_inspector.process [-] [node: MAC 9c:b6:54:01:90:2e] Running pre-processing hook root_disk_selection _run_pre_hooks /opt/stack/ironic-inspector/ironic_inspector/process.py:117
2017-08-28 07:16:33.749 26053 DEBUG ironic_inspector.process [-] [node: MAC 9c:b6:54:01:90:2e] Running pre-processing hook scheduler _run_pre_hooks /opt/stack/ironic-inspector/ironic_inspector/process.py:117
2017-08-28 07:16:33.750 26053 DEBUG ironic_inspector.process [-] [node: MAC 9c:b6:54:01:90:2e] Running pre-processing hook validate_interfaces _run_pre_hooks /opt/stack/ironic-inspector/ironic_inspector/process.py:117
2017-08-28 07:16:33.751 26053 DEBUG ironic_inspector.plugins.standard [-] [node: MAC 9c:b6:54:01:90:2e] No BMC address provided in introspection data, assuming virtual environment before_processing /opt/stack/ironic-inspector/ironic_inspector/plugins/standard.py:229
2017-08-28 07:16:33.752 26053 DEBUG ironic_inspector.plugins.standard [-] [node: MAC 9c:b6:54:01:90:2e] Found interface eno2 with MAC "9c:b6:54:01:90:2f", IP address "None" and client_id "None" _get_interfaces /opt/stack/ironic-inspector/ironic_inspector/plugins/standard.py:174
2017-08-28 07:16:33.752 26053 DEBUG ironic_inspector.plugins.standard [-] [node: MAC 9c:b6:54:01:90:2e] Found interface eno1 with MAC "9c:b6:54:01:90:2e", IP address "172.17.1.229" and client_id "None" _get_interfaces /opt/stack/ironic-inspector/ironic_inspector/plugins/standard.py:174
2017-08-28 07:16:33.753 26053 DEBUG ironic_inspector.plugins.standard [-] [node: MAC 9c:b6:54:01:90:2e] Skipping interface eno2 as it was not PXE booting _validate_interfaces /opt/stack/ironic-inspector/ironic_inspector/plugins/standard.py:206
2017-08-28 07:16:33.753 26053 INFO ironic_inspector.plugins.standard [-] [node: MAC 9c:b6:54:01:90:2e] Using network interface(s): eno1 {'ip': u'172.17.1.229', 'mac': u'9c:b6:54:01:90:2e', 'pxe': True, 'client_id': None}
2017-08-28 07:16:33.754 26053 DEBUG ironic_inspector.process [-] [node: MAC 9c:b6:54:01:90:2e] Running pre-processing hook capabilities _run_pre_hooks /opt/stack/ironic-inspector/ironic_inspector/process.py:117
2017-08-28 07:16:33.754 26053 DEBUG ironic_inspector.process [-] [node: MAC 9c:b6:54:01:90:2e] Running pre-processing hook pci_devices _run_pre_hooks /opt/stack/ironic-inspector/ironic_inspector/process.py:117
2017-08-28 07:16:33.755 26053 DEBUG ironic_inspector.node_cache [-] Trying to use bmc_address of value [u'0.0.0.0'] for node look up find_node /opt/stack/ironic-inspector/ironic_inspector/node_cache.py:805
2017-08-28 07:16:33.758 26053 DEBUG ironic_inspector.node_cache [-] Trying to use mac of value [u'9c:b6:54:01:90:2e', u'9c:b6:54:01:90:2f'] for node look up find_node /opt/stack/ironic-inspector/ironic_inspector/node_cache.py:805
2017-08-28 07:16:33.760 26053 INFO ironic_inspector.utils [-] Could not find a node for attributes {'bmc_address': u'0.0.0.0', 'mac': [u'9c:b6:54:01:90:2e', u'9c:b6:54:01:90:2f']}
2017-08-28 07:16:33.769 26053 INFO ironic_inspector.process [-] [node: MAC 9c:b6:54:01:90:2e] Ramdisk logs were stored in file unknown_20170828-071633.760737.tar.gz
2017-08-28 07:16:33.770 26053 ERROR ironic_inspector.utils [-] [node: MAC 9c:b6:54:01:90:2e] The following failures happened during running pre-processing hooks:
Look up error: Could not find a node for attributes {'bmc_address': u'0.0.0.0', 'mac': [u'9c:b6:54:01:90:2e', u'9c:b6:54:01:90:2f']}
2017-08-28 07:16:33.770 26053 DEBUG ironic_inspector.main [-] Returning error to client: The following failures happened during running pre-processing hooks:
Look up error: Could not find a node for attributes {'bmc_address': u'0.0.0.0', 'mac': [u'9c:b6:54:01:90:2e', u'9c:b6:54:01:90:2f']} error_response /opt/stack/ironic-inspector/ironic_inspector/main.py:67

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/500534

[Dmitry Tantsur (divius)]

Please provide the ramdisk logs

[Sam Betts (sambetts)]

Which ramdisk is this? tinyipa, coreos or DIB?? Getting the BMC address requires the IPMI kernel modules and if those aren't present in the image getting the BMC address will fail.

[Dmitry Tantsur (divius)]

Just to clarify: the inspector patch only fixes treating this 0.0.0.0 address. It won't fix the actual problem, it must be somewhere in IPA. Probably a regression caused by https://github.com/openstack/ironic-python-agent/commit/7c28332e9a59da316a9d6e6652f4269b27b3ca66.

I've heard about this problem with a DIB-built IPA (using RHEL 7.4, I think).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/500534
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=a1d19d97b5973898b07ac05bb8c2c3cf5c199605
Submitter: Jenkins
Branch: master

commit a1d19d97b5973898b07ac05bb8c2c3cf5c199605
Author: Dmitry Tantsur <email address hidden>
Date: Mon Sep 4 14:43:13 2017 +0200

    Do not rely on the older ipmi_address field on lookup

    We need to always use bmc_address from inventory, the ipmi_address field is
    there only for compatibility with older processing hooks.

    Change-Id: Ibf00ecd9278af4ee9318ada44c7c670d13ac22aa
    Closes-Bug: #1714944

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/502024

[Dmitry Tantsur (divius)]

We have a wrong assumption in IPA that a valid channel cannot follow an invalid one. Here is a proof:

# for i in $(seq 1 16); do sudo ipmitool lan print $i | grep "IP Address"; done
Invalid channel: 1
IP Address Source : DHCP Address
IP Address : 10.12.8.144
Invalid channel: 3
Invalid channel: 4
Invalid channel: 5
Invalid channel: 6
Invalid channel: 7
Invalid channel: 8
Invalid channel: 9
Invalid channel: 10
Invalid channel: 11
Invalid channel: 12
Invalid channel: 13
Invalid channel: 14
Invalid channel: 15
Invalid channel: 16

[John L. Villalovos (happycamp)]

Interesting that can have a valid channel after an invalid channel. I had asked that in the review and was told that it could not happen.

https://review.openstack.org/#/c/480679/7/ironic_python_agent/hardware.py@937

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/502478

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/502478
Committed: https://git.openstack.org/cgit/openstack/ironic-python-agent/commit/?id=d6ff5116f4a5586238d44a196ec89d844b151f9d
Submitter: Jenkins
Branch: master

commit d6ff5116f4a5586238d44a196ec89d844b151f9d
Author: Dmitry Tantsur <email address hidden>
Date: Mon Sep 11 16:08:33 2017 +0200

    Remove assumption that a valid IPMI channel cannot follow an invalid one

    It seems to be incorrect at least for some iLO machines.
    Also harden the code against invalid output from ipmitool.

    Change-Id: I733785e9c7d86eadca963f0776910504bf91bcfe
    Closes-Bug: #1714944

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/503038

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/503038
Committed: https://git.openstack.org/cgit/openstack/ironic-python-agent/commit/?id=81ac0b4115633437a53da99cd73ac6663e0b98e4
Submitter: Jenkins
Branch: stable/pike

commit 81ac0b4115633437a53da99cd73ac6663e0b98e4
Author: Dmitry Tantsur <email address hidden>
Date: Mon Sep 11 16:08:33 2017 +0200

    Remove assumption that a valid IPMI channel cannot follow an invalid one

    It seems to be incorrect at least for some iLO machines.
    Also harden the code against invalid output from ipmitool.

    Change-Id: I733785e9c7d86eadca963f0776910504bf91bcfe
    Closes-Bug: #1714944
    (cherry picked from commit d6ff5116f4a5586238d44a196ec89d844b151f9d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/502024
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=77e2b1a84569df95b67e113d7b7178cdfb479001
Submitter: Jenkins
Branch: stable/pike

commit 77e2b1a84569df95b67e113d7b7178cdfb479001
Author: Dmitry Tantsur <email address hidden>
Date: Mon Sep 4 14:43:13 2017 +0200

    Do not rely on the older ipmi_address field on lookup

    We need to always use bmc_address from inventory, the ipmi_address field is
    there only for compatibility with older processing hooks.

    Change-Id: Ibf00ecd9278af4ee9318ada44c7c670d13ac22aa
    Closes-Bug: #1714944
    (cherry picked from commit a1d19d97b5973898b07ac05bb8c2c3cf5c199605)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic-python-agent 2.2.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic-inspector 6.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic-python-agent 3.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic-inspector 6.0.1 release.