Too many connections to nova-api (and not cleaning up)?
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Dashboard (Horizon) |
Fix Released
|
High
|
Matthias Runge | ||
Havana |
Fix Released
|
High
|
Matthias Runge | ||
python-novaclient |
Fix Released
|
High
|
Tihomir Trifonov |
Bug Description
We hit this bug while doing a tripleo/tuskar provision against 7 baremetal machines. Basically after everything was up and running for a while, whilst using the Horizon UI to view the active instances (the 7 baremetal machines that were provisioned as nova compute nodes) Horizon threw an error complaining about too may open files.
[stack@
2073
Restarting openstack-nova-api closed them all (put them all into FIN_WAIT2 / CLOSE_WAIT.)
I was able to recreate this on a more 'standard' setup with devstack. To recreate:
1. Run devstack
2. Monitor connections to nova-api in a terminal: while true; sudo lsof -i :8774; date; sleep 2; done
At this point for me the output here was steady at 10.
3. Log into Horizon and Launch an instance, or two.
4. In Horizon, alternate between "Project-
5. Watch the output from lsof. In a short time I got this up to 150+. Leaving it idle (doing nothing more anywhere), the connections hang around (i.e. all in ESTABLISHED state). In fact they hang around even after I log out of Horizon.
Is this expected behaviour?
thanks, marios
Changed in horizon: | |
status: | New → Confirmed |
importance: | Undecided → Medium |
Changed in horizon: | |
assignee: | nobody → Tihomir Trifonov (ttrifonov) |
Changed in python-novaclient: | |
status: | New → In Progress |
Changed in python-novaclient: | |
assignee: | nobody → Tihomir Trifonov (ttrifonov) |
tags: | added: havana-backport-potential |
Changed in horizon: | |
milestone: | icehouse-3 → icehouse-rc1 |
Changed in horizon: | |
milestone: | icehouse-rc1 → icehouse-3 |
Changed in python-novaclient: | |
importance: | Undecided → High |
tags: | removed: havana-backport-potential |
Changed in horizon: | |
milestone: | icehouse-3 → 2014.1 |
tags: | removed: in-stable-havana |
I can reproduce this on a Havana and also on an Icehouse install;
During the tests, I got ~ 1700 ESTABLISHED connections
and if it errors out, it's on the page /project/overview
Currently, I'm a bit concerned, if a cloud provider already hit that too.