Introspection fails on nodes with the same IPMI address but different IPMI ports

[Alfredo Moralejo]

This issue has been found in RDO-CI jobs running in master branch with tripleo-quickstart minimal profile.

Following error appears in ironic-introspector.log [1]:

2016-12-21 06:39:16.254 24006 ERROR ironic_inspector.node_cache [-] [node: f06c26d1-8008-47a9-a687-d5ef05083e41 state None] Database integrity error (pymysql.err.IntegrityError) (1062, u"Duplicate entry 'bmc_address-127.0.0.1' for key 'PRIMARY'") [SQL: u'INSERT INTO attributes (name, value, uuid) VALUES (%(name)s, %(value)s, %(uuid)s)'] [parameters: {'name': 'bmc_address', 'value': '127.0.0.1', 'uuid': u'f06c26d1-8008-47a9-a687-d5ef05083e41'}] during adding attributes
2016-12-21 06:39:16.255 24006 ERROR ironic_inspector.utils [-] [node: f06c26d1-8008-47a9-a687-d5ef05083e41 state None] Some or all of bmc_address's ['127.0.0.1'] are already on introspection

According to nova logs, it seems that overcloud deploy uses this node based on static information provided by instackenv.json.

[1] https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-minimal-806/undercloud/var/log/ironic-inspector/ironic-inspector.log.gz

[Jay Faulkner (jason-oldos)]

Is this a bug in Ironic/ironic-inspector? This seems like multiple nodes registered as "127.0.0.1", which is an invalid configuration.

[Dmitry Tantsur (divius)]

I agree with Jay - several nodes with IPMI address 127.0.0.1 does not sound right... I wonder if it's because of switch to virtualbmc.

[Dmitry Tantsur (divius)]

Yeah, I believe what ironic-inspector does is not super compatible with virtualbmc. As a temporary measure, we can discard IPMI address, if ipmi_port is set.

[John Trowbridge (trown)]

Since this bug breaks tripleo-quickstart when using introspection with master, we are going to disable introspection until it is resolved. We can revert as soon as there is a fix available.

https://review.openstack.org/418096

[wes hayutin (weshayutin)]

any updates in the status?

[milan k (vetrisko)]

There's a patch for Inspector: https://review.openstack.org/#/c/419010/
This patch stores the port together w/ the address as opposed to dropping
Currently polling folks about preferences for their preferred solution.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/419526

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419526
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=51015b17c81ba7197df4c07ee853d6e348b203c0
Submitter: Jenkins
Branch: master

commit 51015b17c81ba7197df4c07ee853d6e348b203c0
Author: Dmitry Tantsur <email address hidden>
Date: Thu Jan 12 16:36:19 2017 +0100

    Do not use loopback BMC addresses for lookup

    With virtualbmc it's possible to have loopback as IPMI address for several
    nodes, which breaks introspection. This change make ironic-inspector
    ignore loopback addresses.

    Change-Id: Idf141c027e9aa08aaed014398fe8b3506a287afa
    Partial-Bug: #1651719

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/421909

[OpenStack Infra (hudson-openstack)]

Change abandoned by Milan Kováčik (<email address hidden>) on branch: master
Review: https://review.openstack.org/419010
Reason: not needed (another fix merged)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421909
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=ed69b6f30db61e35a0a40725facb1e1c64b744bc
Submitter: Jenkins
Branch: stable/newton

commit ed69b6f30db61e35a0a40725facb1e1c64b744bc
Author: Dmitry Tantsur <email address hidden>
Date: Thu Jan 12 16:36:19 2017 +0100

    Do not use loopback BMC addresses for lookup

    With virtualbmc it's possible to have loopback as IPMI address for several
    nodes, which breaks introspection. This change make ironic-inspector
    ignore loopback addresses.

    Conflicts:
     ironic_inspector/test/unit/test_introspect.py

    Change-Id: Idf141c027e9aa08aaed014398fe8b3506a287afa
    Partial-Bug: #1651719
    (cherry picked from commit 51015b17c81ba7197df4c07ee853d6e348b203c0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421285
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=0ce5cdb7c8344e981a919aab136c4c2eba79d4a2
Submitter: Jenkins
Branch: master

commit 0ce5cdb7c8344e981a919aab136c4c2eba79d4a2
Author: dparalen <email address hidden>
Date: Mon Jan 16 18:55:08 2017 +0100

    Find a node by multiple attributes

    This patch modifies current attributes matching from a single
    name-value->node hit to a best-match score.

    Also using an UUID as the attributes table primary key to allow exposing
    attributes in API later (bug 1525231).

    Change-Id: I205e31652b21b9e030b9530149e533b29c52a387
    Closes-Bug: 1651719
    Partial-Bug: 1525231

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic-inspector 5.0.0 release.

[Ben Nemec (bnemec)]

It appears this was fixed in Inspector. Closing for TripleO too.