default pcie hotplug behavior changes when using q35

[sean mooney]

The q35 machine type support native pcie instead of legacy ahci
based pci hotplug. This has several advantages and one majour disadvantage
with the new pcie approch you need to pre allocate the pcie slot so that
they will be available for use with hotplug if needed.

to support this a new num_pcie_ports config option was added to the libvirt section of
the nova.conf

https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.num_pcie_ports

the default value of this config is 0 which mean we use libvirts default.
libvirts default is to allocate 1 free pcie port as a result by default
you cannot attach more then 1 device without hard rebooting the vm.

previously when using the pc machine type with the i440fx chipset it was possible to attach
multiple interfaces or volumes. as a result the end user behavior has changed
as observed by the failrure in tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_create_list_show_delete_interfaces_by_network_port with the default setting and q35 enabled as
reported in this downstream bug https://bugzilla.redhat.com/show_bug.cgi?id=1716356

to fix this the suggestion is to set the max value and default value of the
num_pcie_ports config option to 32

based on some minimal local testing the memory usage of this change is ~0.4MB per port or ~12.5 mb per vm in addtion qemu overhead. this is based on testing done with libvirt directly with memroy preallocation enabeld
for a 2G guest with the pc machinetype and i440fx chipset total memroy of 2036 MB was observed,
q35 4 ports (the default value that will be calulated by libvirt for the default devices) increesed this to 2056 MB and q35 32 ports to 2066 MB

as such this i a minimal overhead increase which can still be controlled by setting the config to a lower value explicitly.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/663614

[sean mooney (sean-k-mooney)]

our downstream virt folks have started to do some more testing of different confguration
and on differnet acrchitecutres and it appears that 32 is not a good default.
on arch64 values over 24 prevent the guest form properly booting and intial indication
are that the memory use while initally low ~0.7m per port grows after the the guest os intiallies the pci device as such it looks like defaultign to 32 or 24 would cause excessive memory usage to be a sane default.

initial testing a sweet spot in terms of memory overhead may be 16 ports or less.
after the guest fully initializes the devices the overhead grows form about 0.7M per device to ~4M per device based on the average useage over 15 spawns of an instance but the deleta betwen 4 pcie ports and 16 is within 1 sandard devation.

this testing was done by dan berrangé using libvirt directly with
the following setup.
 - a Fedora 30 host
 - qemu 3.1.0
 - libvirt 5.1.0
 - A Fedora 28 x86_64 guest OS
 - 2000 MB of RAM allocated
 - 8 vCPUs, freely floating
 - Memory set to preallocated, and locked in physical RAM
 - One disk, as an qcow2 overlay on a master readonly f28 image
 - One network PCI device
 - One video PCI device
 - One memory balloon PCI device
 - VNC graphics, no client connected

graph or data todate
https://berrange.fedorapeople.org/q35-usage.png
note the yellow line uses the second y axis on the right
were as the magenta and blue axis use the y axis on the left.