k8s NodePort allocation of port 30001 lets vim service crash

[Sten Grüner]

Using k8s on top of StarlingX we did allocate a NodePort service to use port 30001.
Everything worked right a couple of days.
Then we did lock and unlock the host for configuration management. Cluster became instable.

After a troubleshooting session we found that vim service is using port 30001.

Also it looks like port 30004 is allocated by vim

nfv-vim 183011 root 29u IPv4 345017 0t0 TCP 127.0.0.1:30001 (LISTEN)
nfv-vim 183011 root 30u IPv4 345018 0t0 TCP 192.168.204.1:30004 (LISTEN)

The problem is that k8s by default can allocate these ports to NodePort service.

Steps to reproduce
------------------
Create a k8s service with speicfic NodePort=300001
Lock and unlock a host.

Fix proposal
------------
Restrict k8s service node port ranges to exclude those (and maybe other?) ports needed by StarlingX
http://www.thinkcode.se/blog/2019/02/20/kubernetes-service-node-port-range

[Sten Grüner (sten-gruener)]

we use duplex setup and 19.12 version, BUILD_ID="r/stx.3.0"

[Ghada Khalil (gkhalil)]

Would be good to fix for the next stx release

[Bart Wensley (bartwensley)]

Thanks for reporting this - this is something missing from our documentation. When using NodePorts, only ports in the range of 31,500 to 32,767 should be used by applications. Other ports are reserved for use by the StarlingX platform.

A new page should be added to the "Kubernetes Operation" section of our documentation found here: https://docs.starlingx.io/operations/index.html

The new page would be titled something like "NodePort Usage Restrictions" and would say:

The following usage restrictions apply when using NodePorts:
- Ports in the NodePort range 30,000 to 31,499 are reserved for the StarlingX Platform, and additional StarlingX applications that are supported on top of the StarlingX Platform (for example, StarlingX OpenStack).
- Ports in the NodePort range 31,500 to 32,767 are reserved for applications that use Kubernetes NodePort service to expose the application externally.

I will assign this LP to the documentation team.

[Sten Grüner (sten-gruener)]

Thanks, would be also great if you could restrict the range
programmatically, s.t. a forbidden port cannot be automatically assigned by
k8s.

On Tue, 7 Jul 2020 at 14:11, Bart Wensley <email address hidden>
wrote:

> Thanks for reporting this - this is something missing from our
> documentation. When using NodePorts, only ports in the range of 31,500
> to 32,767 should be used by applications. Other ports are reserved for
> use by the StarlingX platform.
>
> A new page should be added to the "Kubernetes Operation" section of our
> documentation found here:
> https://docs.starlingx.io/operations/index.html
>
> The new page would be titled something like "NodePort Usage Restrictions"
> and would say:
>
> The following usage restrictions apply when using NodePorts:
> - Ports in the NodePort range 30,000 to 31,499 are reserved for the
> StarlingX Platform, and additional StarlingX applications that are
> supported on top of the StarlingX Platform (for example, StarlingX
> OpenStack).
> - Ports in the NodePort range 31,500 to 32,767 are reserved for
> applications that use Kubernetes NodePort service to expose the application
> externally.
>
> I will assign this LP to the documentation team.
>
>
> ** Tags added: stx.docs
>
> ** Changed in: starlingx
> Assignee: (unassigned) => M Camp (mcamp859)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1874823
>
> Title:
> k8s NodePort allocation of port 30001 lets vim service crash
>
> Status in StarlingX:
> Triaged
>
> Bug description:
> Using k8s on top of StarlingX we did allocate a NodePort service to use
> port 30001.
> Everything worked right a couple of days.
> Then we did lock and unlock the host for configuration management.
> Cluster became instable.
>
> After a troubleshooting session we found that vim service is using
> port 30001.
>
> Also it looks like port 30004 is allocated by vim
>
> nfv-vim 183011 root 29u IPv4 345017 0t0 TCP
> 127.0.0.1:30001 (LISTEN)
> nfv-vim 183011 root 30u IPv4 345018 0t0 TCP
> 192.168.204.1:30004 (LISTEN)
>
> The problem is that k8s by default can allocate these ports to
> NodePort service.
>
> Steps to reproduce
> ------------------
> Create a k8s service with speicfic NodePort=300001
> Lock and unlock a host.
>
> Fix proposal
> ------------
> Restrict k8s service node port ranges to exclude those (and maybe
> other?) ports needed by StarlingX
>
> http://www.thinkcode.se/blog/2019/02/20/kubernetes-service-node-port-range
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/starlingx/+bug/1874823/+subscriptions
>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/739847

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/739847
Committed: https://git.openstack.org/cgit/starlingx/docs/commit/?id=0c793580c30e51c723fd5cbd4fa00d53bdeb2301
Submitter: Zuul
Branch: master

commit 0c793580c30e51c723fd5cbd4fa00d53bdeb2301
Author: MCamp859 <email address hidden>
Date: Tue Jul 7 16:28:13 2020 -0400

    Describe K8S NodePorts restrictions

    Closes-Bug: 1874823

    Change-Id: Ia45aab96bc7aa512c941d555d052edde9a925417
    Signed-off-by: MCamp859 <email address hidden>

[M Camp (mcamp859)]

Hi Sten,
The new doc topic is here: https://docs.starlingx.io/operations/k8s_nodeport_usage.html
Thanks again for submitting this LP,
Mary C.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.4.0
Review: https://review.opendev.org/741508

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741508
Committed: https://git.openstack.org/cgit/starlingx/docs/commit/?id=b81641d400222bf83c4f4b59178e6d58e24684e4
Submitter: Zuul
Branch: r/stx.4.0

commit b81641d400222bf83c4f4b59178e6d58e24684e4
Author: MCamp859 <email address hidden>
Date: Tue Jul 7 16:28:13 2020 -0400

    Describe K8S NodePorts restrictions

    Closes-Bug: 1874823

    Change-Id: Ia45aab96bc7aa512c941d555d052edde9a925417
    Signed-off-by: MCamp859 <email address hidden>
    (cherry picked from commit 0c793580c30e51c723fd5cbd4fa00d53bdeb2301)