nova-libvirt.yaml vnc port range is very limited (port exhaustion)

Bug #1678025 reported by Tiago Batista on 2017-03-31
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
High
Ben Nemec

Bug Description

On my setup I have compute hosts with 32 physical cpu cores. Using the default cpu overcommit this allows me to run a lot of single vcpu instances on each node. I found this out while trying to test the cpu overcommit ratio:

Sympthom:
After spawning about 250 vms in 50 batches I tried to connect to the console of one of the later ones and failed - Failed to connect to server (code: 1006)
Trying to connect to one of the VMs from the first batch however succeeded.

What I figured out so far:

On https://github.com/openstack/tripleo-heat-templates/blob/2527e459e271ddcfbe38319644dad4f76d51f32b/puppet/services/nova-libvirt.yaml#L72, you can see that the libvirt port range for vnc connections is of only 100 ports, from 5900-5999.

Using the default allocation of 16:1, this range can easily be exhausted using single vcpu flavours on an octactore CPU, something that is not all that uncommon nowadays.

The result when this happens is that it is impossible to connect the horizon supplied console to any vm that is spawned with a vnc port above 5999

And a potential (short term) solution:

Given that the modern CPU architectures are becoming more and more biased towards multicore, with 20 cores becoming a familiar number, I propose expanding this port range.

My initial proposal would be to something closer to 1000 ports, but I would like other's opinions on this.

Changed in tripleo:
milestone: none → pike-1
importance: Undecided → High
status: New → Triaged
Changed in tripleo:
milestone: pike-1 → pike-2
Changed in tripleo:
milestone: pike-2 → pike-3
Changed in tripleo:
milestone: pike-3 → pike-rc1

Fix proposed to branch: master
Review: https://review.openstack.org/493585

Changed in tripleo:
assignee: nobody → Ben Nemec (bnemec)
status: Triaged → In Progress

Reviewed: https://review.openstack.org/493585
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=fb2c13795dee6edba1987c3b3256cfb50d711b43
Submitter: Jenkins
Branch: master

commit fb2c13795dee6edba1987c3b3256cfb50d711b43
Author: Ben Nemec <email address hidden>
Date: Mon Aug 14 10:23:38 2017 -0500

    Extend VNC port range

    Per the attached bug, if a large number of instances are colocated
    on a single compute node it is possible to exhaust the allowed VNC
    ports. This change extends the range to include 1024 ports, which
    with the default 16x overcommit ratio in Nova means we could handle
    a fully loaded 64 core server. That's _probably_ overkill, but I
    think it makes sense to overshoot a bit on this and ensure nobody
    runs into weird problems because their VNC ports weren't allowed
    through the firewall.

    Change-Id: Ia48602e82b8e0fbb585371ea514eea3c2334dab0
    Closes-Bug: 1678025

Changed in tripleo:
status: In Progress → Fix Released

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers