Containers: VM console does not work in AIO-DX lab when VM on standby controller and novnc pod is on the active controller

[Gerry Kopec]

Brief Description
-----------------
VM consoles (e.g. via address provided by openstack console url show <instance-uuid>) do not work if. Instance is on standby controller and nova-novncproxy pod is on active controller.

Severity
--------
Major

Steps to Reproduce
------------------
- on AIO-DX config
- move nova-nonvcproxy pod to active controller. Delete via kubectl delete pod <X> to force reschedule if necessary.
- launch instance
- move instance to standby controller via cold migration if necessary
- obtain url from this command: openstack console url show <instance-uuid>
- test url in browser with access to cluster oam address

Expected Behavior
------------------
VM consoles should work

Actual Behavior
----------------
VM consoles do not work

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Two node system, Multi-node system, Dedicated storage, https, IPv4, IPv6 etc.>

Branch/Pull Time/Commit
-----------------------
master & f/stein

Timestamp/Logs
--------------
n/a

[Ghada Khalil (gkhalil)]

Marking as release gating; related to container env

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/stein
Review: https://review.openstack.org/637354

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/stein
Review: https://review.openstack.org/637355

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637354
Committed: https://git.openstack.org/cgit/openstack/stx-upstream/commit/?id=cb921ff9348fec9c634ef191dba293e33088daae
Submitter: Zuul
Branch: f/stein

commit cb921ff9348fec9c634ef191dba293e33088daae
Author: Gerry Kopec <email address hidden>
Date: Fri Feb 15 18:11:55 2019 -0500

    Update nova helm chart to fix console addressing

    Upstream nova helm chart attempts to figure out the address for VM
    consoles by running an init container that checks for ip routes and
    addresses on a compute host. It then sets the appropriate nova config
    options in a config file which it passes to nova-compute. However this
    effectively overwrites the same config option that stx has already
    set in nova.conf via per host overrides causing us to communicate over
    the wrong network or not to connect at all.

    This fix introduces an option to enable/disable passing of this
    additional config file to nova-compute. Default upstream behaviour is
    unchanged, but we will disable it in our overrides.

    Partial-Bug: #1815490
    Change-Id: I86eb80578b23fd89b7f9643b943ee759f26a15be
    Signed-off-by: Gerry Kopec <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637355
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=db199ed89be3ba792521f9c02c531401559742ae
Submitter: Zuul
Branch: f/stein

commit db199ed89be3ba792521f9c02c531401559742ae
Author: Gerry Kopec <email address hidden>
Date: Fri Feb 15 20:10:56 2019 -0500

    Update nova overrides to fix console addressing

    Use new nova helm chart config option introduced in dependent commit to
    prevent nova from overriding our per host override for VM console
    address.

    Closes-Bug: #1815490
    Depends-On: I86eb80578b23fd89b7f9643b943ee759f26a15be
    Change-Id: I7617157b3b2848cbbe2d9014b900cd437ac082a6
    Signed-off-by: Gerry Kopec <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/640464
Committed: https://git.openstack.org/cgit/openstack/stx-config/commit/?id=1b22b5313d0618792732066a8fe47460d8ef06de
Submitter: Zuul
Branch: master

commit 654c05df0e45aa47d18ce72e5ba003195872790f
Author: Al Bailey <email address hidden>
Date: Fri Feb 22 16:35:12 2019 -0600

    The --kubernetes flag no longer has an effect.

    kubernetes mode is always enabled, the flag cannot be used to
    enable or disable it.

    The option in the CLI will be removed completely once the wiki
    and any test tools are updated.

    The code that handles the "else" will also be updated in a
    later commit

    Story: 2004751
    Task: 29756
    Change-Id: I75a81ab852252ee108fefeca5682e5b1a9d7374e
    Signed-off-by: Al Bailey <email address hidden>

commit 03b08b9722e83597797de93abef54f787b93bab5
Author: Mingyuan Qi <email address hidden>
Date: Wed Jan 30 09:41:27 2019 +0800

    Allow user specified registries for config_controller

    Currently docker images were pulled from public registries during
    config_controller. For some users, the connection to the public
    docker registry may be slow such that installing the containerized
    services images may timeout or the system simply does not have
    access to the public internet.

    This change allows users to specify alternative public/private
    registries to replace k8s.gcr.io, gcr.io, quay.io and docker.io.
    Insecure registry is supported if all default registries were
    replaced by one unified registry. It lowers the complexity for
    those who build his own registry without internet access.

    Docker doesn't support ipv6 addr as registry name, instead
    hostname or domain name in ipv6 network is allowed.

    Test:
    AIO-SX/AIO-DX/Standard(2+2):
      Alternative public registry (ipv4/domain) with proxy
        - config_controller pass
      Private registry (ipv4/ipv6/domain) without internet
        - config_controller pass
      Default registry with/without proxy
        - config_controller pass

    Story: 2004711
    Task: 28742

    Change-Id: I4fee3f4e0637863b9b5ef4ef556082ac75f62a1d
    Signed-off-by: Mingyuan Qi <email address hidden>
    (cherry picked from commit 611a68a96ab915dc4e97d39dffa5c379bbffef3d)

commit 7471ef852b7c37c742ef273f0df6b8ccce3bd928
Author: Bin Qian <email address hidden>
Date: Thu Feb 21 14:46:34 2019 -0500

    Boost sm process priority in VBox environment

    There is an instance that sm claimed its main thread ran sluggish
    as some critical timer run behind the scheuled timing.
    The issue could prevent the sm from scheduling services.
    As the result, the controller could fail to enable.

    The issue was found only on vbox labs on AIO-SX, the fix is to boost
    sm process priority to nice value -10 from current -2.

    Closes-Bug: 1816764
    Depends-On: https://review.openstack.org/638664
    Change-Id: Iafa17b1c47d65cc7394552ea1c8e7a78398e4869
    Signed-off-by: Bin Qian <email address hidden>
    (cherry picked from commit a6934ac9d27e0357d0025018077441d989679409)

commit 5e61519ac92822b959dffe63b76956cf0...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos76
Review: https://review.openstack.org/640918

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/640918
Committed: https://git.openstack.org/cgit/openstack/stx-upstream/commit/?id=1443710d8f88509b5e31ebdb2ce302767cd229df
Submitter: Zuul
Branch: f/centos76

commit 53cfbc9f06b58dee663a8f857dc269bde610c349
Author: Don Penney <email address hidden>
Date: Fri Mar 1 23:50:56 2019 -0500

    Remove openstack images from pike build

    Change-Id: Idd211511e42fa4c1290fdfa382ac579a1a8de91a
    Story: 2004751
    Task: 29788
    Signed-off-by: Don Penney <email address hidden>

commit bcce8c810f56a32710277341b0d6fe23226e7532
Author: Al Bailey <email address hidden>
Date: Thu Feb 21 09:52:23 2019 -0600

    Setting the worker_rlimit_nofile minimum to 2048 for nginx

    In the docker image for mariadb-ingress if there are many cores
    the calculated value for worker_rlimit_nofile ends up being 1024
    which is too small. This change sets the min to 2048.

    Closes-Bug: 1816479
    Change-Id: I4f198b703eda61d9a9531640ec01a2770f9ec172
    Signed-off-by: Al Bailey <email address hidden>

commit db10c94d9e26e4150b2a57c5d2e5673fd97a3481
Author: Alex Kozyrev <email address hidden>
Date: Wed Feb 20 12:15:55 2019 -0500

    Create Docker image for Barbican in StarlingX

    In order to provide the secure management of secrets service
    as a container in StarlingX we need to create Barbican Docker
    image and include it into StarlingX repository.

    Change-Id: I3b4483f74d233348ec49729deff11ba7776af01b
    Story: 2003108
    Task: 29579
    Signed-off-by: Alex Kozyrev <email address hidden>

commit b6e3badac62686a14a0d837d2677feccfa1dfd70
Author: Al Bailey <email address hidden>
Date: Mon Feb 4 11:59:06 2019 -0600

    Fix the version string in cinder and glance clients

    cinderclient was showing 0.0.0 for cinder --version
    Same problem for glance.

    The pbr version needed to be set when building from
    outside of a git tree.

    All other clients had this set properly.

    This bug was introduced when the new stein clients were
    added.

    The cinderclient also needed some BuildRequires updated.
    These BuildRequires were for building wheels.

    Closes-Bug: 1814573
    Change-Id: I4afe783e25ab2172ae999787e6b0e3ec91f78419
    Signed-off-by: Al Bailey <email address hidden>

commit cb921ff9348fec9c634ef191dba293e33088daae
Author: Gerry Kopec <email address hidden>
Date: Fri Feb 15 18:11:55 2019 -0500

    Update nova helm chart to fix console addressing

    Upstream nova helm chart attempts to figure out the address for VM
    consoles by running an init container that checks for ip routes and
    addresses on a compute host. It then sets the appropriate nova config
    options in a config file which it passes to nova-compute. However this
    effectively overwrites the same config option that stx has already
    set in nova.conf via per host overrides causing us to communicate over
    the wrong network or not to connect at all.

    This fix introduces an option to enable/disable passing of this
    additional config file to nova-compute. Default upstream behaviour is
    unchange...