CPU/MEM usage not accurate. Hits quotas while resources are available

[William Kitsos]

I tried to set my quotas to exactly the hardware specs of my compute node for accurate reporting. My compute node runs on bare metal. After I did this I got quotas exceeded, unable to launch instance. I checked the hypervisor from the web interface and cli. 5 of 12 VCPUs used and 6GB of 16GB used. When you try and change the quotas to match the hardware it errors. From the CLI it reports that quota cant be set to 12 VCPUs because 17 are in use. Same with mem says 17GB are in use. But they arent in use clearly. So the bandaid is to set the quotas really high and then it ignores them and allows you to create instances against the actual nova usage stats. This also causes some failure to launch instance errors, no host is available meaning there arent enough resources even though there are. Im running this as a production environment for my domain so I have spent hundreds of hours chasing my tail. Hope this is helpful for debugging the issue.

[Davanum Srinivas (DIMS) (dims-v)]

William, what version of OpenStack/Nova? Any specific distribution?

[William Kitsos (greek35t)]

Davanum,

I am running OpenStack Juno. Its running on three pieces of bare metal.
Nova --version returns 2.19.0 on host compute1. This is running over Ubuntu
server 14.04 LTS. The systems are up to date. I run an 'apt-get update &&
apt-get dist-upgrade' weekly. Please let me know if you need any more
information. Thanks!

King Regards,

William

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of
Davanum Srinivas (DIMS)
Sent: Friday, February 20, 2015 11:31 AM
To: <email address hidden>
Subject: [Bug 1423644] Re: CPU/MEM usage not accurate. Hits quotas while
resources are available

William, what version of OpenStack/Nova? Any specific distribution?

** Changed in: nova
       Status: New => Incomplete

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1423644

Title:
  CPU/MEM usage not accurate. Hits quotas while resources are available

Status in OpenStack Compute (Nova):
  Incomplete
Status in Ubuntu:
  New

Bug description:
  I tried to set my quotas to exactly the hardware specs of my compute
  node for accurate reporting. My compute node runs on bare metal. After
  I did this I got quotas exceeded, unable to launch instance. I checked
  the hypervisor from the web interface and cli. 5 of 12 VCPUs used and
  6GB of 16GB used. When you try and change the quotas to match the
  hardware it errors. From the CLI it reports that quota cant be set to
  12 VCPUs because 17 are in use. Same with mem says 17GB are in use.
  But they arent in use clearly. So the bandaid is to set the quotas
  really high and then it ignores them and allows you to create
  instances against the actual nova usage stats. This also causes some
  failure to launch instance errors, no host is available meaning there
  arent enough resources even though there are. Im running this as a
  production environment for my domain so I have spent hundreds of hours
  chasing my tail. Hope this is helpful for debugging the issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1423644/+subscriptions

[William Kitsos (greek35t)]

Davanum,

I am running OpenStack Juno. Its running on three pieces of bare metal.
Nova --version returns 2.19.0 on host compute1. This is running over Ubuntu
server 14.04 LTS. The systems are up to date. I run an 'apt-get update &&
apt-get dist-upgrade' weekly. Please let me know if you need any more
information. Thanks!

King Regards,

William

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of
Davanum Srinivas (DIMS)
Sent: Friday, February 20, 2015 11:31 AM
To: <email address hidden>
Subject: [Bug 1423644] Re: CPU/MEM usage not accurate. Hits quotas while
resources are available

William, what version of OpenStack/Nova? Any specific distribution?

** Changed in: nova
       Status: New => Incomplete

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1423644

Title:
  CPU/MEM usage not accurate. Hits quotas while resources are available

Status in OpenStack Compute (Nova):
  Incomplete
Status in Ubuntu:
  New

Bug description:
  I tried to set my quotas to exactly the hardware specs of my compute
  node for accurate reporting. My compute node runs on bare metal. After
  I did this I got quotas exceeded, unable to launch instance. I checked
  the hypervisor from the web interface and cli. 5 of 12 VCPUs used and
  6GB of 16GB used. When you try and change the quotas to match the
  hardware it errors. From the CLI it reports that quota cant be set to
  12 VCPUs because 17 are in use. Same with mem says 17GB are in use.
  But they arent in use clearly. So the bandaid is to set the quotas
  really high and then it ignores them and allows you to create
  instances against the actual nova usage stats. This also causes some
  failure to launch instance errors, no host is available meaning there
  arent enough resources even though there are. Im running this as a
  production environment for my domain so I have spent hundreds of hours
  chasing my tail. Hope this is helpful for debugging the issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1423644/+subscriptions

[William Kitsos (greek35t)]

Davanum,

I am running OpenStack Juno. Its running on three pieces of bare metal.
Nova --version returns 2.19.0 on host compute1. This is running over Ubuntu
server 14.04 LTS. The systems are up to date. I run an 'apt-get update &&
apt-get dist-upgrade' weekly. Please let me know if you need any more
information. Thanks!

Affects Openstack/Nova

King Regards,

William

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of
Davanum Srinivas (DIMS)
Sent: Friday, February 20, 2015 11:31 AM
To: <email address hidden>
Subject: [Bug 1423644] Re: CPU/MEM usage not accurate. Hits quotas while
resources are available

William, what version of OpenStack/Nova? Any specific distribution?

** Changed in: nova
       Status: New => Incomplete

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1423644

Title:
  CPU/MEM usage not accurate. Hits quotas while resources are available

Status in OpenStack Compute (Nova):
  Incomplete
Status in Ubuntu:
  New

Bug description:
  I tried to set my quotas to exactly the hardware specs of my compute
  node for accurate reporting. My compute node runs on bare metal. After
  I did this I got quotas exceeded, unable to launch instance. I checked
  the hypervisor from the web interface and cli. 5 of 12 VCPUs used and
  6GB of 16GB used. When you try and change the quotas to match the
  hardware it errors. From the CLI it reports that quota cant be set to
  12 VCPUs because 17 are in use. Same with mem says 17GB are in use.
  But they arent in use clearly. So the bandaid is to set the quotas
  really high and then it ignores them and allows you to create
  instances against the actual nova usage stats. This also causes some
  failure to launch instance errors, no host is available meaning there
  arent enough resources even though there are. Im running this as a
  production environment for my domain so I have spent hundreds of hours
  chasing my tail. Hope this is helpful for debugging the issue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1423644/+subscriptions

[Joe Gordon (jogo)]

Do your flavors match up with your barematal node size? If not you will have issues. Also without API calls and logs to figure out whats going on, its very hard to triage this bug.

[Markus Zoeller (markus_z) (mzoeller)]

@William Kitsos (greek35t):
Reminder: Could you please provide the information Joe Gordon (jogo) asked for in comment #5? You can provide your logs with [1] for example. When you have provided the information, please set the bug status to "new".

[1] http://paste.openstack.org/

[Markus Zoeller (markus_z) (mzoeller)]

Cleanup
=======

This bug report has the status "Incomplete" since more than 30 days
and it looks like that there are no open reviews for it. To keep
the bug list sane, I close this bug with "won't fix". This does not
mean that it is not a valid bug report, it's more to acknowledge that
no progress can be expected here anymore. You are still free to push a
new patch for this bug. If you could reproduce it on the current master
code or on a maintained stable branch, please switch it to "Confirmed".

Juno has reached its EOL, so it's "invalid" for that release.