parallel volume-attachment requests might starve out nova-api for others

[Johannes Kulik]

When doing volume-attachemnts, nova-api does an RPC-call (with a long_rpc_timeout) into nova-compute to reserve_block_device_name(). This takes a lock on the instance. If a volume-attachment was in progress, which also takes the instance-lock, nova-api's RPC-call needs to wait.

Having RPC-calls in nova-api, that can take a long time, will block the process handling the request. If a project does a lot of volume-attachments (e.g. for a k8s workload > 10 attachments per instance), this could starve out other users of nova-api by occupying all available processes.

When running nova-api with eventlet, a small number of processes can handle a lot of requests in parallel and some blocking rpc-calls don't matter too much.

When switching to uWSGI, the number of processes would have to be increased drastically to accommodate for that - unless it's possible to map those requests to threads and use a high number of threads instead.

What's the recommended way to run nova-api on uWSGI to handle this? Low number of processes with high number of threads to mimic eventlet?

[Lee Yarwood (lyarwood)]

So I think I can reproduce this in a basic devstack env with a simple 2 process uwsgi config and enable-threads = true:

$ cat /<email address hidden>

[Unit]
Description = Devstack <email address hidden>

[Service]
RestartForceExitStatus = 100
NotifyAccess = all
Restart = always
KillMode = process
Type = notify
ExecReload = /usr/bin/kill -HUP $MAINPID
ExecStart = /usr/local/bin/uwsgi --procname-prefix nova-api --ini /etc/nova/nova-api-uwsgi.ini
User = stack
SyslogIdentifier = <email address hidden>

[Install]
WantedBy = multi-user.target

$ cat /etc/nova/nova-api-uwsgi.ini

[uwsgi]
chmod-socket = 666
socket = /var/run/uwsgi/nova-api-wsgi.socket
lazy-apps = true
add-header = Connection: close
buffer-size = 65535
hook-master-start = unix_signal:15 gracefully_kill_them_all
thunder-lock = true
plugins = http,python3
enable-threads = true
worker-reload-mercy = 90
exit-on-reload = false
die-on-term = true
master = true
processes = 2
wsgi-file = /usr/local/bin/nova-api-wsgi

$ for i in $(seq 1 9); do openstack volume create --size 1 test-$i ; done
$ for i in $(seq 1 9); do openstack server add volume test test-$i & done
[1] 888087
[2] 888088
[3] 888089
[4] 888090
[5] 888091
[6] 888092
[7] 888093
[8] 888094
[9] 888095
[5] Done openstack server add volume test test-$i
[2] Done openstack server add volume test test-$i
[4] Done openstack server add volume test test-$i
[8]- Done openstack server add volume test test-$i
[9]+ Done openstack server add volume test test-$i
[1] Done openstack server add volume test test-$i
[3] Done openstack server add volume test test-$i
[6]- Done openstack server add volume test test-$i
[7]+ Done openstack server add volume test test-$i
$ openstack server event list test
+------------------------------------------+--------------------------------------+---------------+----------------------------+
| Request ID | Server ID | Action | Start Time |
+------------------------------------------+--------------------------------------+---------------+----------------------------+
| req-df477597-befd-4122-bc3d-d54e9b0daaae | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T09:00:10.000000 |
| req-17e8c4af-a435-4d91-854a-9308a19284d3 | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T09:00:10.000000 |
| req-336f0d30-716a-4294-90d9-33869192e961 | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T09:00:03.000000 |
| req-7b9d3b8a-e94d-446e-890f-216cc8f8d6c3 | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T09:00:03.000000 |
| req-a2e80b18-113f-436f-8fa8-4c2373b9d831 | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T08:59:57.000000 |
| req-8d24416c-b80d-4230-8cf4-9a979690e7ff | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T08:59:56.000000 |
| req-0d91f187-e4c7-4469-bd66-4880add4b2df | 589295a7-14bf-40cb-8f05-1b225448002e | attach_volume | 2021-06-02T08:5...

[Lee Yarwood (lyarwood)]

I can't capture it in an update here but uwsgitop graphically demonstrates this as well.

I've wanted to drop the RPC call to reserve_block_device_name on the compute for a while as the virt drivers can't guarantee the assigned device name within the guestOS and it's just quicker to create our BDMs in the API workers before returning:

https://docs.openstack.org/nova/latest/reference/attach-volume.html

This would require a microversion as the response of os-volume_attachments would change, making device optional:

https://docs.openstack.org/api-ref/compute/?expanded=attach-a-volume-to-an-instance-detail#attach-a-volume-to-an-instance

I can try to write up a spec for this in the coming days.

[sean mooney (sean-k-mooney)]

just an FYI the nova-api when run under uwsgi is still using eventlet monkey patching.
in the past it did not but it has for about 18month or more since we added the multi
cell sactter gatter code around Stine/train.
currently we do not recommend using multiple threads when usign nova-api with uwsgi

[melanie witt (melwitt)]

+1 what Sean said. We have had a known issue documented since Stein indicating that threads=1 has to be used otherwise it will be possible for oslo.messaging to attempt a reply on a different eventlet thread than the request came from and cause timeouts.

Unfortunately I'm not aware of anything we can do here other than increase the number of processes.

This might be yet another reason we will want to enable nova to run with regular threads -- then we wouldn't have to run oslo.messaging rpc with the eventlet executor and theoretically could run wsgi with threads > 1.

[Johannes Kulik (jkulik)]

Is there any update on this topic?

[Johannes Kulik (jkulik)]

As a note: There was a (now abandoned) spec for the removal of the device from the os-volume_attachments call, that could be re-proposed https://review.opendev.org/c/openstack/nova-specs/+/765097

[Johannes Kulik (jkulik)]

There's also been some work on getting eventlet out of nova-api in this review-request: https://review.opendev.org/c/openstack/nova/+/650172