concurrent snapshot and restore operations can result in duplicate / orphan vm's
Bug #1412952 reported by
Patrick Crews
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Heat |
Incomplete
|
Medium
|
Qiming Teng |
Bug Description
Testing with 2 concurrent users performing random snapshot and restore operations on a single stack can result in one or more 'ghost' / orphan / duplicate vm's using devstack.
I have not isolated the exact combination of events yet, but it seems that while the system is restoring a stack from a snapshot that it is possible for the 'main' / current vm in the stack to be snapshotted. This then might prevent the auto-deletion of this vm once the restored vm is ready (just a theory).
Command line for test + further info follows.
Changed in heat: | |
assignee: | nobody → Qiming Teng (tengqim) |
Changed in heat: | |
milestone: | none → no-priority-tag-bugs |
To post a comment you must log in.
NOTE: the test stack can be anything. This is a basic test stack created from this template: git.openstack. org/cgit/ openstack/ heat-templates/ plain/hot/ F20/WordPress_ Native. yaml
http://
$ heat stack-list ------- ------- ------- ------- ----+-- ------- -----+- ------- ------- --+---- ------- ------- ----+ ------- ------- ------- ------- ----+-- ------- -----+- ------- ------- --+---- ------- ------- ----+ bb00-4b72- 883a-8c84fc7526 4e | test-stack-0 | CREATE_COMPLETE | 2015-01- 20T20:01: 29Z | ------- ------- ------- ------- ----+-- ------- -----+- ------- ------- --+---- ------- ------- ----+ ------- ------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- --+---- ----+-- ------- ---+--- ------- ---+--- ------- ------- -+ ------- ------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- --+---- ----+-- ------- ---+--- ------- ---+--- ------- ------- -+ bb27-4536- 916c-685ed686ea 59 | test-stack- 0-wordpress_ instance- zxjcxj3xvvlz | ACTIVE | - | Running | private=10.0.0.2 | ------- ------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- --+---- ----+-- ------- ---+--- ------- ---+--- ------- ------- -+
+------
| id | stack_name | stack_status | creation_time |
+------
| 1a83b2e8-
+------
$ nova list
+------
| ID | Name | Status | Task State | Power State | Networks |
+------
| be5f8c3d-
+------
# START RANDOM TESTING /github. com/pcrews/ rannsaka 192.168. 0.5 --requests 500 -w 2 --test- file=locust_ files/heat_ basic_stress. py ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- d4974a0e5ce6130 859aea/ stacks 235 0(0.00%) 951 32 11595 | 45 0.80 [name]/ [id]/resources 46 0(0.00%) 396 40 15616 | 47 0.10 [name]/ [id]/snapshots 99 0(0.00%) 461 39 13461 | 91 0.50 [name]/ [id]/snapshots 29 15(34.09%) 1986 120 8277 | 470 0.10 [name]/ [id]/snapshots/ [restore_ snapshot] 53 11(17.19%) 596 76 5653 | 180 0.10 ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -------
# clone https:/
# cd rannsaka/rannsake and run the command line
# 2 'users', 500 total requests - simple set of randomized heat api calls
$ time python rannsaka.py --host=http://
# POST TESTING
Name # reqs # fails Avg Min Max | Median req/s
-------
GET /v1/81e7409a431
POST /v2.0/tokens 12 0(0.00%) 373 167 2098 | 180 0.00
GET stacks/
GET stacks/
POST stacks/
POST stacks/
-------
Total 474 26(5.49%) 1.60
$ heat stack-list ------- ------- ------- ------- ----+-- ------- -----+- ------- ------- ----+-- ------- ------- ------+
+------
| id | stack_name ...