Introspection fails on nodes with the same IPMI address but different IPMI ports

Bug #1651719 reported by Alfredo Moralejo on 2016-12-21
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic Inspector
Fix Released
High
milan k
tripleo
High
Unassigned

Bug Description

This issue has been found in RDO-CI jobs running in master branch with tripleo-quickstart minimal profile.

Following error appears in ironic-introspector.log [1]:

2016-12-21 06:39:16.254 24006 ERROR ironic_inspector.node_cache [-] [node: f06c26d1-8008-47a9-a687-d5ef05083e41 state None] Database integrity error (pymysql.err.IntegrityError) (1062, u"Duplicate entry 'bmc_address-127.0.0.1' for key 'PRIMARY'") [SQL: u'INSERT INTO attributes (name, value, uuid) VALUES (%(name)s, %(value)s, %(uuid)s)'] [parameters: {'name': 'bmc_address', 'value': '127.0.0.1', 'uuid': u'f06c26d1-8008-47a9-a687-d5ef05083e41'}] during adding attributes
2016-12-21 06:39:16.255 24006 ERROR ironic_inspector.utils [-] [node: f06c26d1-8008-47a9-a687-d5ef05083e41 state None] Some or all of bmc_address's ['127.0.0.1'] are already on introspection

According to nova logs, it seems that overcloud deploy uses this node based on static information provided by instackenv.json.

[1] https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-minimal-806/undercloud/var/log/ironic-inspector/ironic-inspector.log.gz

Changed in tripleo:
status: New → Triaged
importance: Undecided → High
milestone: none → ocata-3
Jay Faulkner (jason-oldos) wrote :

Is this a bug in Ironic/ironic-inspector? This seems like multiple nodes registered as "127.0.0.1", which is an invalid configuration.

Changed in ironic:
status: New → Incomplete
Dmitry Tantsur (divius) wrote :

I agree with Jay - several nodes with IPMI address 127.0.0.1 does not sound right... I wonder if it's because of switch to virtualbmc.

summary: - Database integrity error in ironic-introspector when enabling
- introspection
+ Introspection fails on nodes with the same IPMI address but different
+ ports
summary: Introspection fails on nodes with the same IPMI address but different
- ports
+ IPMI ports
affects: ironic → ironic-inspector
Changed in ironic-inspector:
status: Incomplete → Triaged
importance: Undecided → High
Dmitry Tantsur (divius) wrote :

Yeah, I believe what ironic-inspector does is not super compatible with virtualbmc. As a temporary measure, we can discard IPMI address, if ipmi_port is set.

milan k (vetrisko) on 2017-01-09
Changed in ironic-inspector:
assignee: nobody → milan k (vetrisko)
John Trowbridge (trown) wrote :

Since this bug breaks tripleo-quickstart when using introspection with master, we are going to disable introspection until it is resolved. We can revert as soon as there is a fix available.

https://review.openstack.org/418096

Changed in ironic-inspector:
status: Triaged → In Progress
wes hayutin (weshayutin) wrote :

any updates in the status?

milan k (vetrisko) wrote :

There's a patch for Inspector: https://review.openstack.org/#/c/419010/
This patch stores the port together w/ the address as opposed to dropping
Currently polling folks about preferences for their preferred solution.

Fix proposed to branch: master
Review: https://review.openstack.org/419526

Changed in ironic-inspector:
assignee: milan k (vetrisko) → Dmitry Tantsur (divius)
Changed in tripleo:
milestone: ocata-3 → ocata-rc1
Changed in ironic-inspector:
assignee: Dmitry Tantsur (divius) → milan k (vetrisko)

Reviewed: https://review.openstack.org/419526
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=51015b17c81ba7197df4c07ee853d6e348b203c0
Submitter: Jenkins
Branch: master

commit 51015b17c81ba7197df4c07ee853d6e348b203c0
Author: Dmitry Tantsur <email address hidden>
Date: Thu Jan 12 16:36:19 2017 +0100

    Do not use loopback BMC addresses for lookup

    With virtualbmc it's possible to have loopback as IPMI address for several
    nodes, which breaks introspection. This change make ironic-inspector
    ignore loopback addresses.

    Change-Id: Idf141c027e9aa08aaed014398fe8b3506a287afa
    Partial-Bug: #1651719

Change abandoned by Milan Kováčik (<email address hidden>) on branch: master
Review: https://review.openstack.org/419010
Reason: not needed (another fix merged)

Reviewed: https://review.openstack.org/421909
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=ed69b6f30db61e35a0a40725facb1e1c64b744bc
Submitter: Jenkins
Branch: stable/newton

commit ed69b6f30db61e35a0a40725facb1e1c64b744bc
Author: Dmitry Tantsur <email address hidden>
Date: Thu Jan 12 16:36:19 2017 +0100

    Do not use loopback BMC addresses for lookup

    With virtualbmc it's possible to have loopback as IPMI address for several
    nodes, which breaks introspection. This change make ironic-inspector
    ignore loopback addresses.

    Conflicts:
     ironic_inspector/test/unit/test_introspect.py

    Change-Id: Idf141c027e9aa08aaed014398fe8b3506a287afa
    Partial-Bug: #1651719
    (cherry picked from commit 51015b17c81ba7197df4c07ee853d6e348b203c0)

tags: added: in-stable-newton
Changed in ironic-inspector:
assignee: milan k (vetrisko) → Anton Arefiev (aarefiev)
Changed in ironic-inspector:
assignee: Anton Arefiev (aarefiev) → milan k (vetrisko)

Reviewed: https://review.openstack.org/421285
Committed: https://git.openstack.org/cgit/openstack/ironic-inspector/commit/?id=0ce5cdb7c8344e981a919aab136c4c2eba79d4a2
Submitter: Jenkins
Branch: master

commit 0ce5cdb7c8344e981a919aab136c4c2eba79d4a2
Author: dparalen <email address hidden>
Date: Mon Jan 16 18:55:08 2017 +0100

    Find a node by multiple attributes

    This patch modifies current attributes matching from a single
    name-value->node hit to a best-match score.

    Also using an UUID as the attributes table primary key to allow exposing
    attributes in API later (bug 1525231).

    Change-Id: I205e31652b21b9e030b9530149e533b29c52a387
    Closes-Bug: 1651719
    Partial-Bug: 1525231

Changed in ironic-inspector:
status: In Progress → Fix Released

This issue was fixed in the openstack/ironic-inspector 5.0.0 release.

Changed in tripleo:
milestone: ocata-rc1 → ocata-rc2
Changed in tripleo:
milestone: ocata-rc2 → pike-1
Changed in tripleo:
milestone: pike-1 → pike-2
Changed in tripleo:
milestone: pike-2 → pike-3
Changed in tripleo:
milestone: pike-3 → pike-rc1
Ben Nemec (bnemec) wrote :

It appears this was fixed in Inspector. Closing for TripleO too.

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers