nova compute log for not enough resources is not informative enough

Bug #1292090 reported by Roey Dekel
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Opinion
Wishlist
Unassigned

Bug Description

Tried to boot 128 mini VM's (cirros 0.3.1 with 64 MB RAM and no disk), and got the next error:

2014-03-10 11:09:11.089 3177 ERROR nova.compute.manager [req-2d77626c-6fe6-4e33-9a25-c7c43df07c5a 36feb44e6d71467aa01d46894654252c 8338e986a31b4f3cb30476126f147192] [instance: 0c06ed9e-52d2-4140-ac05-6fac7dc3b1f8] Error: Insufficient compute resources.

This error message doesn't explains which limit was reached (maybe some quota-limit was reached).

Reproduce:
----------------
for i in $(seq 1 128); do
nova boot stress-${i} --flavor 0 --image cirros-0.3.1 --nic net-id=`neutron net-list | grep netInt | cut -d" " -f2` & done

Results:
-----------
( take notice for stress-i, for i in {107, 111, 119, 123, 124, 126, 13, 15, 17, 18, 19, 28, 35, 37, 4, 41, 42, 44, 48, 49, 5, 51, 54, 60, 62, 64, 65, 69, 83, 9, 91, 98} )
[root@cougar16 ~(keystone_admin)]# nova list
+--------------------------------------+------------+--------+------------+-------------+-------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------+--------+------------+-------------+-------------------------------------------+
| 7b171137-55f3-4051-a4b2-a98ac394fe39 | instance1 | ERROR | None | NOSTATE | |
| 2735176e-9109-44e2-8534-8b9e40b91fbc | stress-1 | ACTIVE | None | Running | netInt216=192.168.216.74 |
| 04fc9bed-fa6a-45b6-aea1-d2dd0c0a461e | stress-10 | ACTIVE | None | Running | netInt216=192.168.216.80 |
| 196eb4e8-4fa1-4237-abd2-dd99203de3ad | stress-100 | ACTIVE | None | Running | netInt216=192.168.216.78 |
| 4b10d616-cb5e-433c-bd70-09cc5fb6c650 | stress-101 | ACTIVE | None | Running | netInt216=192.168.216.27 |
| 4b3b3b9b-061c-432b-9c39-d765842f4760 | stress-102 | ACTIVE | None | Running | netInt216=192.168.216.45 |
| 20e66ca3-9177-4ae3-a755-0dcca5dc24ff | stress-103 | ACTIVE | None | Running | netInt216=192.168.216.6 |
| e111b279-27bf-4f96-87df-cd3407eb90ed | stress-104 | ACTIVE | None | Running | netInt216=192.168.216.38 |
| f7fb8e0b-d141-47d5-8d0f-b4ed3b853c23 | stress-105 | ACTIVE | None | Running | netInt216=192.168.216.61 |
| 3550af11-3591-4ee2-9ea8-83dcd9e77e9a | stress-106 | ACTIVE | None | Running | netInt216=192.168.216.30 |
| 0bdca75d-fe9c-49a0-b789-58a6ca9fd0fe | stress-107 | ERROR | None | NOSTATE | |
| a79317ec-e3ff-4d96-b64f-8389a3ab79b1 | stress-108 | ACTIVE | None | Running | netInt216=192.168.216.76 |
| cfca6ffd-97c9-48c9-a0ee-a887482b94be | stress-109 | ACTIVE | None | Running | netInt216=192.168.216.31 |
| b7422a03-8f3c-491b-bf41-8ac8cdd2f1d1 | stress-11 | ACTIVE | None | Running | netInt216=192.168.216.55 |
| 93ee7204-a53f-4bdc-a34a-ff7a954ad373 | stress-110 | ACTIVE | None | Running | netInt216=192.168.216.9 |
| 6e36b033-bd79-44a6-882f-a664567a69ff | stress-111 | ERROR | None | NOSTATE | |
| cc294eec-093f-4564-9ab9-e4f28fc3ac99 | stress-112 | ACTIVE | None | Running | netInt216=192.168.216.47 |
| 85892e9e-9e58-44ab-8a41-ef9532a54983 | stress-113 | ACTIVE | None | Running | netInt216=192.168.216.41 |
| 6e917473-ca22-4bf2-8ba8-b1e3761e3aba | stress-114 | ACTIVE | None | Running | netInt216=192.168.216.65 |
| af9e8c49-8d53-4bf8-b96b-82d8a89cf84a | stress-115 | ACTIVE | None | Running | netInt216=192.168.216.58 |
| 51afb7d3-4f3f-4182-bc75-6608dbf2894c | stress-116 | ACTIVE | None | Running | netInt216=192.168.216.92, 192.168.216.98 |
| 6c222d79-1d4d-4d32-aad7-10155a07f5c8 | stress-117 | ACTIVE | None | Running | netInt216=192.168.216.33 |
| 81c1f8ec-b3cf-40ac-a1ae-44d043ab6cc9 | stress-118 | ACTIVE | None | Running | netInt216=192.168.216.51 |
| c77b7e7f-2c48-4709-a64b-58ba4ef4f282 | stress-119 | ERROR | None | NOSTATE | |
| fd1f192a-dd61-4ad8-8fb8-6cf49abe3e16 | stress-12 | ACTIVE | None | Running | netInt216=192.168.216.89 |
| 5405c600-04af-457e-8a0b-560c26b4267b | stress-120 | ACTIVE | None | Running | netInt216=192.168.216.18 |
| a97c1833-9634-4a4d-bf9f-ffa915c33bdc | stress-121 | ACTIVE | None | Running | netInt216=192.168.216.39 |
| 6577d161-7311-4058-a596-e6684906d238 | stress-122 | ACTIVE | None | Running | netInt216=192.168.216.83 |
| 0c06ed9e-52d2-4140-ac05-6fac7dc3b1f8 | stress-123 | ERROR | None | NOSTATE | |
| 4529ad2c-4e10-47e7-8f28-4629e27b58c9 | stress-124 | ERROR | None | NOSTATE | |
| 4acc9354-a54d-4da6-8637-fb870f38996a | stress-125 | ACTIVE | None | Running | netInt216=192.168.216.4 |
| 371f0084-ee5b-4c0c-bfdf-2bcae240dde9 | stress-126 | ERROR | None | NOSTATE | |
| 84867daf-7686-4b60-aa04-c9b29e6fc726 | stress-127 | ACTIVE | None | Running | netInt216=192.168.216.5 |
| 5ce00c24-35d9-4891-9eea-c9df3c661657 | stress-128 | ACTIVE | None | Running | netInt216=192.168.216.2 |
| d160e306-0542-4d63-86df-7c26a9010a1d | stress-13 | ERROR | None | NOSTATE | |
| c37eac35-9940-4ed2-a672-d746093cc7ee | stress-14 | ACTIVE | None | Running | netInt216=192.168.216.21 |
| b294ba72-bb0f-4135-9755-2e2dcc0638c3 | stress-15 | ERROR | None | NOSTATE | |
| 4c74c2bc-4c8c-47c3-8d23-87a56263343e | stress-16 | ACTIVE | None | Running | netInt216=192.168.216.22 |
| 8e4bd38d-6ce0-4789-aa2c-bbabd1759b2c | stress-17 | ERROR | None | NOSTATE | |
| 97cb57bb-4b16-4a88-a6ba-f15b199b9315 | stress-18 | ERROR | None | NOSTATE | |
| 9a3d6467-5a52-4fe9-b96e-c5ac1915e080 | stress-19 | ERROR | None | NOSTATE | |
| ef6bfafd-7063-49b6-ad8a-59b0c41637ea | stress-2 | ACTIVE | None | Running | netInt216=192.168.216.82 |
| 05e5bd38-aa22-46a6-9408-09c536f4892a | stress-20 | ACTIVE | None | Running | netInt216=192.168.216.95 |
| 7767d71c-febf-4811-b173-7f9f51eccbe1 | stress-21 | ACTIVE | None | Running | netInt216=192.168.216.42 |
| 2f9f48d9-0a4b-424d-97fd-2cf678e7b1ec | stress-22 | ACTIVE | None | Running | netInt216=192.168.216.8 |
| b7287ea4-c5c5-41da-b3df-187d4452727d | stress-23 | ACTIVE | None | Running | netInt216=192.168.216.60 |
| 07f06fbf-c464-4742-abdf-2034d9f3c620 | stress-24 | ACTIVE | None | Running | netInt216=192.168.216.35 |
| f12232e4-a8e3-4c47-bb62-75d02237ae3a | stress-25 | ACTIVE | None | Running | netInt216=192.168.216.7 |
| 070c9da4-6ad4-49f8-b8c9-287bab2e53fd | stress-26 | ACTIVE | None | Running | netInt216=192.168.216.25 |
| d6f731a2-c9dc-45ee-9cee-06741057c54c | stress-27 | ACTIVE | None | Running | netInt216=192.168.216.101 |
| fdaefdae-25f1-4613-8ec6-1f8a38977f00 | stress-28 | ERROR | None | NOSTATE | |
| 54fffd41-38c3-440a-8001-826954a5bfcc | stress-29 | ACTIVE | None | Running | netInt216=192.168.216.81 |
| 70ac6840-7e6f-40d5-94cf-47973c6eca72 | stress-3 | ACTIVE | None | Running | netInt216=192.168.216.54 |
| 2ea5c839-fdbc-48a2-9faa-1e810e5f64aa | stress-30 | ACTIVE | None | Running | netInt216=192.168.216.64 |
| be29518e-d61a-4c74-9a0b-a34fbed1e214 | stress-31 | ACTIVE | None | Running | netInt216=192.168.216.94, 192.168.216.97 |
| 712c44e6-762b-493f-bced-74347f734327 | stress-32 | ACTIVE | None | Running | netInt216=192.168.216.56 |
| b2308805-edef-4993-bf55-c94902064c31 | stress-33 | ACTIVE | None | Running | netInt216=192.168.216.85 |
| 651943b7-2d95-43d3-97e8-15099dedf5e0 | stress-34 | ACTIVE | None | Running | netInt216=192.168.216.15 |
| e0753858-91db-4406-b62d-504ddbaff205 | stress-35 | ERROR | None | NOSTATE | |
| 79fafd25-7145-49b5-9356-e39ebdabfb09 | stress-36 | ACTIVE | None | Running | netInt216=192.168.216.19 |
| 0dac104a-dd95-4b89-bc6b-f9e1f9bcdaa6 | stress-37 | ERROR | None | NOSTATE | |
| 056ac4e7-913b-4c22-a6b6-20a2ac43e6ab | stress-38 | ACTIVE | None | Running | netInt216=192.168.216.66 |
| 52897e28-9116-43a2-9b9b-25d6e26700eb | stress-39 | ACTIVE | None | Running | netInt216=192.168.216.87 |
| 0d6a0a24-7e4a-45e9-b9bc-26e66d73748b | stress-4 | ERROR | None | NOSTATE | |
| 6adc30cf-a427-47fb-a488-1185b5edac9f | stress-40 | ACTIVE | None | Running | netInt216=192.168.216.75 |
| 4c03dbef-beb6-478f-a276-330b311d9a41 | stress-41 | ERROR | None | NOSTATE | |
| 018d629b-f536-42a1-987f-5b144506db38 | stress-42 | ERROR | None | NOSTATE | |
| 480a0abe-40af-4441-8aff-d1589fa81b91 | stress-43 | ACTIVE | None | Running | netInt216=192.168.216.77 |
| d443d482-df5d-495c-a540-8deae6c6c83d | stress-44 | ERROR | None | NOSTATE | |
| c6796141-56ad-4557-b9c7-78107fad88c2 | stress-45 | ACTIVE | None | Running | netInt216=192.168.216.59 |
| 0300d4ce-84de-4d1f-a9c1-9704279caa21 | stress-46 | ACTIVE | None | Running | netInt216=192.168.216.50 |
| b50d6f76-bd07-4214-9be9-30e8d5bda048 | stress-47 | ACTIVE | None | Running | netInt216=192.168.216.23 |
| 3eaa71ee-4abd-49a5-acb5-8d8e13b6a5a1 | stress-48 | ERROR | None | NOSTATE | |
| ed6d8564-53e3-4fd1-bf0b-2de1fb41d616 | stress-49 | ERROR | None | NOSTATE | |
| eefd4149-0384-49a3-a44c-b323ae202176 | stress-5 | ERROR | None | NOSTATE | |
| c4025f26-3763-49a6-8aac-893c57ed0f44 | stress-50 | ACTIVE | None | Running | netInt216=192.168.216.11 |
| 4afa17e0-b3ea-4dce-a70c-b94639c65851 | stress-51 | ERROR | None | NOSTATE | |
| 8a8d2451-5fe9-41b3-8c0d-62a2e0f710c8 | stress-52 | ACTIVE | None | Running | netInt216=192.168.216.29 |
| 830f8f01-b301-41e7-a068-7d453d46b4e7 | stress-53 | ACTIVE | None | Running | netInt216=192.168.216.28 |
| 54fff99d-2c5b-418b-9466-bced2035bf9a | stress-54 | ERROR | None | NOSTATE | |
| 16f14493-4789-4393-a5b4-91b4bf9649b1 | stress-55 | ACTIVE | None | Running | netInt216=192.168.216.103 |
| eb13ac53-7115-45bd-9036-100332ae2c58 | stress-56 | ACTIVE | None | Running | netInt216=192.168.216.44 |
| eb1e5ba3-c837-48d5-9ad3-af4891a31e42 | stress-57 | ACTIVE | None | Running | netInt216=192.168.216.63 |
| f4c0396a-4c1e-47ae-8d0e-91bbba09ddc4 | stress-58 | ACTIVE | None | Running | netInt216=192.168.216.73 |
| 9aa69efc-87a6-454d-bdf2-145096c1cc25 | stress-59 | ACTIVE | None | Running | netInt216=192.168.216.49 |
| 108b282f-db42-48ea-ba30-a0463286ca06 | stress-6 | ACTIVE | None | Running | netInt216=192.168.216.68 |
| 58c070b2-767d-41a9-93b3-a2617adb0438 | stress-60 | ERROR | None | NOSTATE | |
| 94de6ddb-c21a-4c2b-9cd2-cbbfa2baf11b | stress-61 | ACTIVE | None | Running | netInt216=192.168.216.72 |
| 61969c14-c743-4f41-b9ab-2f9fbb930894 | stress-62 | ERROR | None | NOSTATE | |
| 20993cd3-d32e-4d0f-8407-8eb2d551e739 | stress-63 | ACTIVE | None | Running | netInt216=192.168.216.32 |
| 38a6638f-c891-4745-a4db-6bbc764cbe92 | stress-64 | ERROR | None | NOSTATE | |
| 0f5d72e1-fdb6-4d55-b302-dc98d4532bb4 | stress-65 | ERROR | None | NOSTATE | |
| 424401ac-cc9a-499e-a982-049fa189442b | stress-66 | ACTIVE | None | Running | netInt216=192.168.216.20 |
| 939c1831-83cd-42ee-a130-b1c0051b01c2 | stress-67 | ACTIVE | None | Running | netInt216=192.168.216.70 |
| be0e2a7f-a6a9-41ac-80f8-3a7064bde2cc | stress-68 | ACTIVE | None | Running | netInt216=192.168.216.48 |
| 5e7426d3-0e9b-47df-9e54-24e16f03c93d | stress-69 | ERROR | None | NOSTATE | |
| fb3525d1-93fe-45b3-96d5-d49fde648d3e | stress-7 | ACTIVE | None | Running | netInt216=192.168.216.57 |
| 368090b9-60e3-440b-8148-08a065ebc52d | stress-70 | ACTIVE | None | Running | netInt216=192.168.216.43 |
| 92e4d020-a684-442a-a958-aafba050458d | stress-71 | ACTIVE | None | Running | netInt216=192.168.216.24 |
| 8da69b9f-266f-4e38-a02b-f15c3918b0ce | stress-72 | ACTIVE | None | Running | netInt216=192.168.216.34 |
| 7aacb6b2-ed1b-4b45-96e9-e96dbc0332d7 | stress-73 | ACTIVE | None | Running | netInt216=192.168.216.17 |
| 27f7f98d-d639-4222-9c30-acb570bcd6d6 | stress-74 | ACTIVE | None | Running | netInt216=192.168.216.86 |
| 5476f11d-be1b-4751-b0c3-9baef2d4b9ea | stress-75 | ACTIVE | None | Running | netInt216=192.168.216.46 |
| aa7775db-51c1-48da-b867-f587fc7c57ca | stress-76 | ACTIVE | None | Running | netInt216=192.168.216.53 |
| 4a5ee968-78f0-463f-8eab-de4453593735 | stress-77 | ACTIVE | None | Running | netInt216=192.168.216.13 |
| 8bf170b3-7f41-4f24-8668-7a3475ac1c3a | stress-78 | ACTIVE | None | Running | netInt216=192.168.216.10 |
| 097aa22a-3d14-4dbc-9437-e8aad4483b1d | stress-79 | ACTIVE | None | Running | netInt216=192.168.216.90, 192.168.216.96 |
| 29b10080-8a84-42d6-9133-51ec99cbbab1 | stress-8 | ACTIVE | None | Running | netInt216=192.168.216.69 |
| 8a7cf911-da93-4844-84a1-5086fe35ed1c | stress-80 | ACTIVE | None | Running | netInt216=192.168.216.67 |
| 49568ddc-3669-456d-a5e6-05e0c8fbf6ee | stress-81 | ACTIVE | None | Running | netInt216=192.168.216.16 |
| a0de5750-ed0c-4b73-8e70-ec7f29fd49ca | stress-82 | ACTIVE | None | Running | netInt216=192.168.216.88 |
| 6d691758-007a-4115-876b-31de2f2fc3ba | stress-83 | ERROR | None | NOSTATE | |
| 62d25278-cf13-45a0-a7bd-331d3de4ce0a | stress-84 | ACTIVE | None | Running | netInt216=192.168.216.62 |
| dfb3fb07-bbd8-4535-8d33-300137b69ced | stress-85 | ACTIVE | None | Running | netInt216=192.168.216.52 |
| cc1321a8-71b5-4308-aab6-63a6182c0759 | stress-86 | ACTIVE | None | Running | netInt216=192.168.216.36 |
| 01da3a4b-aa91-4e44-8e4d-6ae56ecbf6a7 | stress-87 | ACTIVE | None | Running | netInt216=192.168.216.37 |
| ca2bc085-2104-492c-8d79-173b7f1bbbaa | stress-88 | ACTIVE | None | Running | netInt216=192.168.216.100 |
| 9831e389-3a25-4f80-97ae-8d525a60ec09 | stress-89 | ACTIVE | None | Running | netInt216=192.168.216.26 |
| b3ad6ae0-f880-46e9-be2e-0bae80775c15 | stress-9 | ERROR | None | NOSTATE | |
| e00a5d97-ed70-4636-a921-54b204b1f55a | stress-90 | ACTIVE | None | Running | netInt216=192.168.216.93, 192.168.216.102 |
| cc46531d-6733-4458-9da1-5b896574077f | stress-91 | ERROR | None | NOSTATE | |
| 7afcabe1-395d-4e08-ac9a-fd0daf6005c5 | stress-92 | ACTIVE | None | Running | netInt216=192.168.216.99, 192.168.216.91 |
| b93b5b78-00bf-41de-acd3-c0d38823da14 | stress-93 | ACTIVE | None | Running | netInt216=192.168.216.79 |
| 060fece3-ec3e-415e-b74a-c8c4ad1860fc | stress-94 | ACTIVE | None | Running | netInt216=192.168.216.71 |
| fbb6bbc4-d977-47c8-a5fe-45c4c38006a0 | stress-95 | ACTIVE | None | Running | netInt216=192.168.216.84 |
| 642ad6db-91f2-4ade-be57-cd53597d8abb | stress-96 | ACTIVE | None | Running | netInt216=192.168.216.14 |
| 70716764-84f3-4ff5-b9c3-a10f9d760730 | stress-97 | ACTIVE | None | Running | netInt216=192.168.216.40 |
| 38f4b32a-5860-4df3-95cd-bf77cbfa11a9 | stress-98 | ERROR | None | NOSTATE | |
| d243d756-b6b2-432f-831b-15f743ce3eb3 | stress-99 | ACTIVE | None | Running | netInt216=192.168.216.12 |
+--------------------------------------+------------+--------+------------+-------------+-------------------------------------------+

Setup status:
----------------

[root@cougar16 ~(keystone_admin)]# nova image-list
+--------------------------------------+--------------+--------+--------+
| ID | Name | Status | Server |
+--------------------------------------+--------------+--------+--------+
| fd375460-5380-4dda-a71b-4a616064126e | cirros-0.3.1 | ACTIVE | |
+--------------------------------------+--------------+--------+--------+

[root@cougar16 ~(keystone_admin)]# nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
| 0 | mini | 64 | 0 | 0 | | 1 | 1.0 | True |
| 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True |
| 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True |
| 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True |
| 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True |
| 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+

[root@cougar16 ~(keystone_admin)]# nova quota-show
+-----------------------------+-------+
| Quota | Limit |
+-----------------------------+-------+
| instances | 1024 |
| cores | 1024 |
| ram | 51200 |
| floating_ips | 10 |
| fixed_ips | -1 |
| metadata_items | 128 |
| injected_files | 5 |
| injected_file_content_bytes | 10240 |
| injected_file_path_bytes | 255 |
| key_pairs | 100 |
| security_groups | 10 |
| security_group_rules | 20 |
+-----------------------------+-------+

[root@cougar16 ~(keystone_admin)]# neutron quota-show
+---------------------+-------+
| Field | Value |
+---------------------+-------+
| floatingip | 50 |
| network | 10 |
| port | 1024 |
| router | 10 |
| security_group | 10 |
| security_group_rule | 100 |
| subnet | 10 |
+---------------------+-------+

Tags: compute quotas
Revision history for this message
ofer blaut (oblaut) wrote :

I think the API (CLI/GUI) should detailed the Insufficient resource in the VM error state and not only in the logs

no longer affects: neutron
Tracy Jones (tjones-i)
tags: added: api
tags: added: compute
removed: api
melanie witt (melwitt)
Changed in nova:
importance: Undecided → Wishlist
status: New → Confirmed
lizheming (lizheming-li)
Changed in nova:
assignee: nobody → lizheming (lizheming-li)
lizheming (lizheming-li)
Changed in nova:
assignee: lizheming (lizheming-li) → nobody
Joe Gordon (jogo)
tags: added: quotas
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

This spec is targeting that: https://review.openstack.org/#/c/306647/

Changed in nova:
status: Confirmed → Opinion
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.