Repeated boot of ironic instances on added conductor failed

Bug #1526303 reported by Kyrylo Romanenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Confirmed
Medium
MOS Ironic

Bug Description

Steps:
1. Prepared cluster configuration:
    1 Controller
    1 Ironic
    1 compute
    1 not allocated node
Compute - QEMU
Network - Neutron with VLAN segmentation
Storage Backends - Cinder LVM over iSCSI for volumes

2. Deployed it to the operational state.
3. Then booted Ironic instance VM according to happy path Configuration Instruction.
4. Shutdown ironic instance with "nova delete" command.
5. Add 1 more Ironic Conductor node to Cluster.
6. Deploy changes.
7. Boot Ironic instance VM. Shutdown it and boot more several times.

Actual result:
Only twice VM booted successfully.

Inspection of ironic-conductor logs discovered that after redeployment only 2nd conductor was used.
Here is snippet of 1st Conductor log: http://paste.openstack.org/show/481939/
Note that last updates from 2015-12-14 22:36:38.
UPD: looks that it is expected when only one conductor is used while load is very low.

Conductor Log snippet from 2nd Conductor node:
http://paste.openstack.org/show/481940/

Please note there two errors occurred:
AttributeError: 'ConductorManager' object has no attribute '_keepalive_e
vt'
and
Cannot execute SSH cmd LC_ALL=C /usr/bin/virsh --connect qemu:/
//system destroy kromanenko-275-scaling_ironic-slave-01. Reason: Unexpected error while running command.

VM on virtual manager unexpectedly gone offline.
When I switched ON it back - it booted Ubuntu that was deployed on last successful nova boot.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "275"
  build_id: "275"

description: updated
Changed in mos:
status: New → Confirmed
tags: added: mos-ironic
description: updated
description: updated
Revision history for this message
Yuriy Zveryanskyy (yzveryanskyy) wrote :

"AttributeError: 'ConductorManager' object has no attribute '_keepalive_e" <- this bug fixed in upstream https://bugs.launchpad.net/ironic/+bug/1496017

Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

Yuriy, thank you for the upstream link.

description: updated
Revision history for this message
Pavlo Shchelokovskyy (pshchelo) wrote :

as the root cause is the same upstream bug as with bug 1526693, marking this as duplicate

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.