Stx-ingress is using the container file system when buffering requests with big body size

[Thales Elero Cervi]

Brief Description
-----------------
When requests with big body size go through the stx ingress before reaching the application service, the container is buffering the request body on its /tmp dir that is not mounted to any Kubernetes volume, temporally increasing the docker fs instead of the kubelet fs usage.

Example: Glance image (huge size, >20g) upload when stx-openstack has HTTPS enabled.

Severity
--------
Minor: Feature is usable with minor issue.

Steps to Reproduce
------------------
* Apply stx-openstack application to the system
* Enable HTTPS for stx-openstack application
* Try to upload an OpenStack image (size >20g): openstack image create --file 50gb_dummy_image.img --disk-format raw --container-format bare --public Image50gb

Expected Behavior
------------------
User should be able to upload the image if system has enough free size for a temporary disk usage increase on kubelet-lv

Actual Behavior
----------------
User is not able to upload the image even when the system has enough free size for a temporary disk usage peak on kubelet-lv

Reproducibility
---------------
Reproducible when docker-lv does not have enough free size for a temporary disk usage peak

System Configuration
--------------------
Stx-openstack with HTTPS

Branch/Pull Time/Commit
-----------------------
master

Last Pass
---------
Never

Timestamp/Logs
--------------
2022-05-05T15:24:50.269383275Z stderr F 2022/05/05 15:24:50 [warn] 55#55: *126572 a client request body is buffered to a temporary file /tmp/client-body/0000000009, client: 128.1.10.17, server: glance.my.server.com, request: "PUT /v2/images/ae1ba364-4657-8f7b-28fdc0283691/file HTTP/1.1", host: "glance.my.server.com"

Test Activity
-------------
Feature Testing

Workaround
----------
User needs to extended the available disk on docker-lv so the upload is able to complete.
That's not desirable, since when stx-opesntack is running with HTTP user needs to extended the available disk on kubelet-lv for this kind of operation. We should keep consistency between operation guides.

The reason why kubelet-lv stores the buffering for HTTP requests is that the openstack ingress is able to resolve the service inside the cluster and the openstack ingress has its /tmp mounted to a Kubernetes Ephemeral Volume (emptyDir).
When HTTPS is enabled there is a service name resolution that will lead the request to reach the stx-ingress before reaching the service. This ingress is not mounting its /tmp to a Kubernetes volume and therefore it is using the container file system when buffering.

The container file system will be place where containerd is mapped to, in this case it will be on docker-lv. It can be considered a bad usage of the container file system.

[sysadmin@controller-0]$ sudo cat /etc/containerd/config.toml
root = "/var/lib/docker"
state = "/var/run/containerd"

Platform Ingress:
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from ic-nginx-ingress-ingress-nginx-token-vb2tq (ro)

OpenStack Ingress:
Mounts:
/tmp from pod-tmp (rw)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/nginx-ingress-controller-armada-app/+/841624

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/nginx-ingress-controller-armada-app/+/841624
Committed: https://opendev.org/starlingx/nginx-ingress-controller-armada-app/commit/cac8f610db3240cce2687e99fd5156a1f1616668
Submitter: "Zuul (22348)"
Branch: master

commit cac8f610db3240cce2687e99fd5156a1f1616668
Author: Thales Elero Cervi <email address hidden>
Date: Thu May 12 14:37:55 2022 -0300

    Adds tmp volume mount to ingress controller

    When requests that are going through the stx ingress have big body size
    the platform docker-lv is temporally increasing its disk usage due to
    a request body buffering on the container /tmp directory. Since the tmp
    directory is not mounted to any kubernetes volume it is using the
    container file system, limited to where containerd is installed [1]

    To avoid docker-lv misuse and any related issue it would be interesting
    to mount the controller /tmp directory on a kubernetes volume. It would
    also be consistent with what is already done on some application
    ingresses (e.g. stx-openstack ingress). This way we also keep consistent
    documentation for operational procedures (e.g. instructing user on how
    much to increase the lv available size for use cases that required huge
    request body buffering).

    TEST PLAN:
    PASS: Build a stx-platform/ingress-nginx chart and apply it to the
    system
    PASS: Check that the nginx-ingress-controller pod is up and has it /tmp
    directory correctly mounted to a kubernetes volume
    PASS: Check that requests with body buffering are using the correct
    platform kubelet-lv instead of docker-lv (stx-openstack scenario)

    [1] https://opendev.org/starlingx/stx-puppet/src/branch/master/puppet-manifests/src/modules/platform/manifests/filesystem.pp#L264

    Closes-Bug: 1973212
    Signed-off-by: Thales Elero Cervi <email address hidden>
    Change-Id: Ibb53255e3f533900afa2a1921f76f164bacae437