when hard-reboot a instance with serial-port multiple times, instance will not start for port exhausted

Bug #1489853 reported by lica
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Sahid Orentino
Liberty
Fix Released
Medium
Unassigned
Mitaka
Fix Released
Medium
Sahid Orentino

Bug Description

My running environment is
openstack-nova-compute-2015.1.0-3.el7.noarch
python-nova-2015.1.0-3.el7.noarch
openstack-nova-novncproxy-2015.1.0-3.el7.noarch
openstack-nova-conductor-2015.1.0-3.el7.noarch
openstack-nova-api-2015.1.0-3.el7.noarch
openstack-nova-console-2015.1.0-3.el7.noarch
openstack-nova-scheduler-2015.1.0-3.el7.noarch
openstack-nova-serialproxy-2015.1.0-3.el7.noarch
openstack-nova-common-2015.1.0-3.el7.noarch

In my nova.conf ,port_range=20000:20020

I boot a instance with two serial-port ,it works well.
When i hard reboot this instance muti-times,it can't start and its status is shut-off
The log is below :
2015-08-28 17:06:46.635 7258 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/d
river.py", line 3830, in _create_serial_console_devices
2015-08-28 17:06:46.635 7258 TRACE oslo_messaging.rpc.dispatcher console.listen_host))
2015-08-28 17:06:46.635 7258 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_concurrency/lo
ckutils.py", line 445, in inner
2015-08-28 17:06:46.635 7258 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-08-28 17:06:46.635 7258 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/console/serial
.py", line 82, in acquire_port
2015-08-28 17:06:46.635 7258 TRACE oslo_messaging.rpc.dispatcher raise exception.SocketPortRangeExhaustedException(host=h
ost)

after check code,i thought the hard-reboot code process has some problem。

The function release_port() is only in function cleanup()

When a instance is deleted ,function cleanup() is called

But when a instance is hard-boot,in _hard_reboot function it only call _destroy().

and then it call _get_guest_xml() and in it acquire_port() function is called.

so the instance will always acquire port but not release. at last port will be exhausted in logic

I think in _hard_reboot() of libvrit/driver.py,it should be edited below:

.....
self._destroy(instance)
if CONF.serial_console.enabled:
            serials = self._get_serial_ports_from_instance(instance)
            for hostname, port in serials:
                serial_console.release_port(host=hostname, port=port)
....

tags: added: console libvirt
Changed in nova:
assignee: nobody → Chung Chih, Hung (lyanchih)
Revision history for this message
Alexander Schmidt (alexs-h) wrote :

This problem is fixed when the changes for bug 1455252 are merged. I'll not mark this as duplicate because it describes a different problem, but I'll add a "closes bug" reference when I push the code for the next patchset.

Changed in nova:
assignee: Chung Chih, Hung (lyanchih) → Alexander Schmidt (alexs-h)
Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/275800

Changed in nova:
assignee: Alexander Schmidt (alexs-h) → Markus Zoeller (markus_z) (mzoeller)
Changed in nova:
assignee: Markus Zoeller (markus_z) (mzoeller) → Marco Pavone (pavone)
Changed in nova:
assignee: Marco Pavone (pavone) → Markus Zoeller (markus_z) (mzoeller)
tags: added: liberty-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/301247

Changed in nova:
assignee: Markus Zoeller (markus_z) (mzoeller) → sahid (sahid-ferdjaoui)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Markus Zoeller (markus_z) (<email address hidden>) on branch: master
Review: https://review.openstack.org/275800
Reason: Sahid is solving bug 1489853 in change https://review.openstack.org/301247

Jay Pipes (jaypipes)
summary: - when hard-reboot a instance with serial-port mutil times ,instance will
- not start for port exhausted
+ when hard-reboot a instance with serial-port multiple times, instance
+ will not start for port exhausted
Changed in nova:
assignee: sahid (sahid-ferdjaoui) → Dan Smith (danms)
Matt Riedemann (mriedem)
Changed in nova:
assignee: Dan Smith (danms) → sahid (sahid-ferdjaoui)
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/301247
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=95da4e87881424524c8ddda6a546764d43eb755e
Submitter: Jenkins
Branch: master

commit 95da4e87881424524c8ddda6a546764d43eb755e
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Mon Apr 4 10:24:23 2016 -0400

    libvirt: release serial console ports when destroying guests

    The part of code responsible to release serial ports does not cover
    all the cases. This commit moves that part inside the method _destroy
    which is responsible for destroying the guest from a libvirt
    perspective which is when we want to release ports.

    Change-Id: I24f2a1fa3b7ef3f2a3196f3689e5f2ba7d49bd87
    Closes-Bug: #1489853

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/314207

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/nova 14.0.0.0b1

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/mitaka)

Reviewed: https://review.openstack.org/314207
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=26364b206ddc39770079f519e4b463f06e06a475
Submitter: Jenkins
Branch: stable/mitaka

commit 26364b206ddc39770079f519e4b463f06e06a475
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Mon Apr 4 10:24:23 2016 -0400

    libvirt: release serial console ports when destroying guests

    The part of code responsible to release serial ports does not cover
    all the cases. This commit moves that part inside the method _destroy
    which is responsible for destroying the guest from a libvirt
    perspective which is when we want to release ports.

    (cherry picked from commit 95da4e87881424524c8ddda6a546764d43eb755e)

    Change-Id: I24f2a1fa3b7ef3f2a3196f3689e5f2ba7d49bd87
    Closes-Bug: #1489853

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/nova 13.1.1

This issue was fixed in the openstack/nova 13.1.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.