API_WORKERS default too high on machines with many CPUs

Bug #1838688 reported by Adam Spiers
This bug affects 1 person
Affects Status Importance Assigned to Milestone

Bug Description

I've installed devstack on some machines with as many as 256 CPUs. stackrc has:

    API_WORKERS=${API_WORKERS:=$(( ($(nproc)/4)<2 ? 2 : ($(nproc)/4) ))}

so a lot of the services end up getting configured with 64 workers. This causes massive numbers of connections to mysqld and ends up hammering mysqld with a large number of heartbeats (from nova services AFAICS).

(This in turn seems to cause mysqld to reach sustained writes to disk of 2--5 MB/s. In theory heartbeats shouldn't cause it to write to disk at all, and I'm not entirely sure yet but it looks like this is happening due to another bug where after every SQL SELECT heartbeat something is also triggering a COMMIT then a ROLLBACK. I'll file a separate bug for that though.)

I suggest capping API_WORKERS at 8 or 16 which should be enough for devstack scenarios.

Revision history for this message
Adam Spiers (adam.spiers) wrote :

Notice that due to bug #1838689, lowering API_WORKERS does not currently lower neutron's number of RPC workers. However that will be fixed by https://review.opendev.org/#/c/674125/

Changed in devstack:
status: New → Triaged
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.