Setting glance workers to higher value in uwsgi.ini results in 500 for any command

[Brian Rosmaita]

Reported by Abhishek.

Environment:
Devstack all in one setup
No of cores: 4

Steps to reproduce:
1. Ensure image-list command is working
   $ glance image-list
   Output:
   List available glance images
2. Edit /etc/glance/glance-uwsgi.ini and set processes = 40
3. Restart glance-api service
   $ sudo systemctl restart <email address hidden>
4. Ensure 21 processes/workers are running for glance-api
   $ ps auxf | awk '{print $6/1024 " MB\t\t" $11 " " $12}' | sort -n | grep -o glance-api | wc -l
   Output:
   $ 21
5. Try running list command again
   $ glance image-list
   Output:
   InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-90629274-b863-4384-a5dd-1a79344faf09)
6. Try another command
   $ nova list
   Output:
   ERROR (InternalServerError): An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-c29869ac-930a-4cf3-aad5-45c4a3584a91)
7. Reset processes value in /etc/glance/glance-uwsgi.ini to 2
8. Restart glance-api service
   $ sudo systemctl restart <email address hidden>
9. Ensure 3 processes/workers are running for glance-api
   $ ps auxf | awk '{print $6/1024 " MB\t\t" $11 " " $12}' | sort -n | grep -o glance-api | wc -l
   Output:
   $ 3
10. Try running list command again
   $ glance image-list
   Output:
   InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-90629274-b863-4384-a5dd-1a79344faf09)

[Matthew Treinish (treinish)]

I attempted to do this locally in a devstack environment and I was not able to reproduce this at all. Setting the number of processes to launch to 10x the number of cores did not cause anything to fail. (I eventually exhausted all of my VM's memory but then the OOMkiller just killed things)

Looking at these steps my assumption is that something else is breaking and it just happens to coincide with you changing the uwsgi config and restarting glance. (or it maybe a side effect of misconfiguring your glance)This is kinda confirmed by the fact that nova list is also returning 500 because that call has nothing to do with glance at all. It's isolated to nova's DB and api service.

Additionally there aren't any service logs as to what's causing the 500. A 500 error normally means there is an unhandled exception in the api service, so there should be stacktraces in the logs that will show you what's wrong. My guess is though that you exhausted all of the ram on your machine and the oomkiller killed mysql.