nova-libvirt.yaml vnc port range is very limited (port exhaustion)

[Tiago Batista]

On my setup I have compute hosts with 32 physical cpu cores. Using the default cpu overcommit this allows me to run a lot of single vcpu instances on each node. I found this out while trying to test the cpu overcommit ratio:

Sympthom:
After spawning about 250 vms in 50 batches I tried to connect to the console of one of the later ones and failed - Failed to connect to server (code: 1006)
Trying to connect to one of the VMs from the first batch however succeeded.

What I figured out so far:

On https://github.com/openstack/tripleo-heat-templates/blob/2527e459e271ddcfbe38319644dad4f76d51f32b/puppet/services/nova-libvirt.yaml#L72, you can see that the libvirt port range for vnc connections is of only 100 ports, from 5900-5999.

Using the default allocation of 16:1, this range can easily be exhausted using single vcpu flavours on an octactore CPU, something that is not all that uncommon nowadays.

The result when this happens is that it is impossible to connect the horizon supplied console to any vm that is spawned with a vnc port above 5999

And a potential (short term) solution:

Given that the modern CPU architectures are becoming more and more biased towards multicore, with 20 cores becoming a familiar number, I propose expanding this port range.

My initial proposal would be to something closer to 1000 ports, but I would like other's opinions on this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/493585

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/493585
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=fb2c13795dee6edba1987c3b3256cfb50d711b43
Submitter: Jenkins
Branch: master

commit fb2c13795dee6edba1987c3b3256cfb50d711b43
Author: Ben Nemec <email address hidden>
Date: Mon Aug 14 10:23:38 2017 -0500

    Extend VNC port range

    Per the attached bug, if a large number of instances are colocated
    on a single compute node it is possible to exhaust the allowed VNC
    ports. This change extends the range to include 1024 ports, which
    with the default 16x overcommit ratio in Nova means we could handle
    a fully loaded 64 core server. That's _probably_ overkill, but I
    think it makes sense to overshoot a bit on this and ensure nobody
    runs into weird problems because their VNC ports weren't allowed
    through the firewall.

    Change-Id: Ia48602e82b8e0fbb585371ea514eea3c2334dab0
    Closes-Bug: 1678025

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0rc1 release candidate.