| 2022-05-19 17:08:44 |
Thales Elero Cervi |
description |
Brief Description
-----------------
When requests with big body size go through the stx ingress before reaching the application service, the container is buffering the request body on its /tmp dir that is not mounted to any Kubernetes volume, temporally increasing the docker fs instead of the kubelet fs usage.
Example: Glance image (huge size, >20g) upload when stx-openstack has HTTPS enabled.
Severity
--------
Minor: Feature is usable with minor issue.
Steps to Reproduce
------------------
* Apply stx-openstack application to the system
* Enable HTTPS for stx-openstack application
* Try to upload an OpenStack image (size >20g): openstack image create --file 50gb_dummy_image.img --disk-format raw --container-format bare --public Image50gb
Expected Behavior
------------------
User should be able to upload the image if system has enough free size for a temporary disk usage increase on vg-kubelet--lv
Actual Behavior
----------------
User is not able to upload the image even when the system has enough free size for a temporary disk usage increase on vg-kubelet--lv
Reproducibility
---------------
Reproducible when vg-docker--lv does not have enough free size for a temporary disk usage increase
System Configuration
--------------------
Stx-openstack with HTTPS
Branch/Pull Time/Commit
-----------------------
master
Last Pass
---------
Never
Timestamp/Logs
--------------
2022-05-05T15:24:50.269383275Z stderr F 2022/05/05 15:24:50 [warn] 55#55: *126572 a client request body is buffered to a temporary file /tmp/client-body/0000000009, client: 128.1.10.17, server: glance.my.server.com, request: "PUT /v2/images/ae1ba364-4657-8f7b-28fdc0283691/file HTTP/1.1", host: "glance.my.server.com"
Test Activity
-------------
Feature Testing
Workaround
----------
User needs to extended the available disk on vg-docker--lv so the upload is able to complete.
That's not desirable, since when stx-opesntack is running with HTTP user needs to extended the available disk on vg-kubelet--lv for this kind of operation. We should keep consistency between operation guides.
The reason why vg-kubelet--lv stores the buffering for HTTP requests is that the openstack ingress is able to resolve the service inside the cluster and the openstack ingress has its /tmp mounted to a Kubernetes Ephemeral Volume (emptyDir).
When HTTPS is enabled there is a service name resolution that will lead the request to reach the stx-ingress before reaching the service. This ingress is not mounting its /tmp to a Kubernetes volume and therefore it is using the container file system when buffering.
The container file system will be place where containerd is mapped to, in this case it will be on vg-docker--lv. It can be considered a bad usage of the container file system.
[sysadmin@controller-0]$ sudo cat /etc/containerd/config.toml
root = "/var/lib/docker"
state = "/var/run/containerd"
Platform Ingress:
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from ic-nginx-ingress-ingress-nginx-token-vb2tq (ro)
OpenStack Ingress:
Mounts:
/tmp from pod-tmp (rw) |
Brief Description
-----------------
When requests with big body size go through the stx ingress before reaching the application service, the container is buffering the request body on its /tmp dir that is not mounted to any Kubernetes volume, temporally increasing the docker fs instead of the kubelet fs usage.
Example: Glance image (huge size, >20g) upload when stx-openstack has HTTPS enabled.
Severity
--------
Minor: Feature is usable with minor issue.
Steps to Reproduce
------------------
* Apply stx-openstack application to the system
* Enable HTTPS for stx-openstack application
* Try to upload an OpenStack image (size >20g): openstack image create --file 50gb_dummy_image.img --disk-format raw --container-format bare --public Image50gb
Expected Behavior
------------------
User should be able to upload the image if system has enough free size for a temporary disk usage increase on kubelet-lv
Actual Behavior
----------------
User is not able to upload the image even when the system has enough free size for a temporary disk usage peak on kubelet-lv
Reproducibility
---------------
Reproducible when docker-lv does not have enough free size for a temporary disk usage peak
System Configuration
--------------------
Stx-openstack with HTTPS
Branch/Pull Time/Commit
-----------------------
master
Last Pass
---------
Never
Timestamp/Logs
--------------
2022-05-05T15:24:50.269383275Z stderr F 2022/05/05 15:24:50 [warn] 55#55: *126572 a client request body is buffered to a temporary file /tmp/client-body/0000000009, client: 128.1.10.17, server: glance.my.server.com, request: "PUT /v2/images/ae1ba364-4657-8f7b-28fdc0283691/file HTTP/1.1", host: "glance.my.server.com"
Test Activity
-------------
Feature Testing
Workaround
----------
User needs to extended the available disk on docker-lv so the upload is able to complete.
That's not desirable, since when stx-opesntack is running with HTTP user needs to extended the available disk on kubelet-lv for this kind of operation. We should keep consistency between operation guides.
The reason why kubelet-lv stores the buffering for HTTP requests is that the openstack ingress is able to resolve the service inside the cluster and the openstack ingress has its /tmp mounted to a Kubernetes Ephemeral Volume (emptyDir).
When HTTPS is enabled there is a service name resolution that will lead the request to reach the stx-ingress before reaching the service. This ingress is not mounting its /tmp to a Kubernetes volume and therefore it is using the container file system when buffering.
The container file system will be place where containerd is mapped to, in this case it will be on docker-lv. It can be considered a bad usage of the container file system.
[sysadmin@controller-0]$ sudo cat /etc/containerd/config.toml
root = "/var/lib/docker"
state = "/var/run/containerd"
Platform Ingress:
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from ic-nginx-ingress-ingress-nginx-token-vb2tq (ro)
OpenStack Ingress:
Mounts:
/tmp from pod-tmp (rw) |
|
