Instance Termination delays in updating port list
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| OpenStack Compute (nova) |
Invalid
|
Undecided
|
Unassigned | ||
| neutron |
Invalid
|
Undecided
|
Unassigned | ||
Bug Description
When deleting an instance, the port list is not updated immediately. As a result - deleting net/subnet fails with error:
409-{u'
(*) Happens only on automated scripts, since manual input isn't quick enough to catch this.
(**) Happens only when Floating IP is attached - error doesn't happen when Floating IP isn't used.
my guess: Nova delays in deleting the VM while checking with neutron DB that Floating IP was deleted.
(***) nova delete command returns BEFORE instance is deleted:
> nova delete $instance_id ; nova show $instance_id
will return the instance without failure
(*****) might also affect Tempest during tearDown execution
Version:
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
python-
openstack-
openstack-
How to reproduce:
**** script attached ****
assumes:
1. external network exists with floating ip range available ("public")
2. image exists
Setup:
1 create network "private"
2. create subnet
3. create router:
3.1 set router gateway to "public"
3.2 set router interface to "private"
4. create VM
5. assign Floating IP to VM
TearDown
1. Delete / Disassociate Floating IP
2. Delete VM
3. detach router interface from subnet (router-
4. Delete subnet/net
Expected Result:
subnet/net should be successfully deleted.
Actual Results:
"Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet"
409-{u'
script log:
line 101 - VM port still in port list even though VM was deleted
line 105 - subnet fails to delete
line 117 - network successfully deleted after enough time passed for port list to update
1 + EXT_NET_NAME=public
2 + NET_NAME=my_net
3 + SUBNET_
4 + ROUTER_
5 + SERVER_
6 + IMAGE_NAME=
7 + MASK=54.0.0
8 + SERVER_IP=54.0.0.6
9 ++ neutron net-list
10 ++ grep public
11 ++ awk '{print $2;}'
12 + EXT_NET_
13 ++ nova image-list
14 ++ grep -w ' cirros-
15 ++ awk '{print $2;}'
16 + IMAGE_ID=
17 ++ grep -w id
18 ++ neutron net-create my_net
19 ++ awk '{print $4;}'
20 + NET_ID=
21 ++ neutron subnet-create --name my_subnet 6ec5ef65-
22 ++ grep -w id
23 ++ awk '{print $4;}'
24 + SUBNET_
25 ++ neutron router-create my_router
26 ++ awk '{print $4;}'
27 ++ grep -w id
28 + ROUTER_
29 + neutron router-gateway-set df211133-
30 Set gateway for router df211133-
31 + neutron router-
32 Added interface cafd4161-
33 + nova boot --flavor 2 --image 1f16b297-
34 +------
35 | Property | Value |
36 +------
37 | OS-DCF:diskConfig | MANUAL |
38 | OS-EXT-
39 | OS-EXT-
40 | OS-EXT-
41 | OS-EXT-STS:vm_state | building |
42 | OS-SRV-
43 | OS-SRV-
44 | accessIPv4 | |
45 | accessIPv6 | |
46 | adminPass | 9VJ9XMrtFghW |
47 | config_drive | |
48 | created | 2014-01-
49 | flavor | m1.small |
50 | hostId | |
51 | id | d09bbee1-
52 | image | cirros-
53 | key_name | None |
54 | metadata | {} |
55 | name | my_server |
56 | os-extended-
57 | progress | 0 |
58 | security_groups | [{"name": "default"}] |
59 | status | BUILD |
60 | tenant_id | 66293febf7164c8
61 | updated | 2014-01-
62 | user_id | e895d5bf0dc64d4
63 +------
64 ++ nova list
65 ++ grep my_server
66 ++ awk '{print $2;}'
67 + SERVER_
68 ++ neutron port-list
69 ++ grep 54.0.0.6
70 ++ awk '{print $2;}'
71 + SERVER_
72 ++ neutron floatingip-create 200a91cf-
73 ++ grep floating_ip_address
74 ++ awk '{print $4;}'
75 + FLOATING_
76 ++ neutron floatingip-list
77 ++ awk '{print $2;}'
78 ++ grep 172.24.4.4
79 + FLOATING_
80 + neutron port-list
81 +------
82 | id | name | mac_address | fixed_ips |
83 +------
84 | 06b3022f-
85 | 13630e5f-
86 | 21981e00-
87 | 2a0a4134-
88 | cafd4161-
89 +------
90 + neutron floatingip-delete 1d77d791-
91 Deleted floatingip: 1d77d791-
92 + nova delete d09bbee1-
93 + neutron router-
94 Removed interface from router df211133-
95 + neutron port-list
96 +------
97 | id | name | mac_address | fixed_ips |
98 +------
99 | 06b3022f-
100 | 13630e5f-
101 | 21981e00-
102 | 2a0a4134-
103 +------
104 + neutron subnet-delete 76abfa0f-
105 409-{u'
106 + neutron port-list
107 +------
108 | id | name | mac_address | fixed_ips |
109 +------
110 | 06b3022f-
111 | 13630e5f-
112 | 2a0a4134-
113 +------
114 + neutron router-delete df211133-
115 Deleted router: df211133-
116 + neutron net-delete 6ec5ef65-
117 Deleted network: 6ec5ef65-
118 + set +x
| description: | updated |
| Mauro S M Rodrigues (maurorodrigues) wrote | #2 |
| Yair Fried (yfried) wrote | #3 |
| Salvatore Orlando (salvatore-orlando) wrote | #4 |
| Yair Fried (yfried) wrote | #5 |
| Mauro S M Rodrigues (maurorodrigues) wrote | #6 |
| Yair Fried (yfried) wrote | #7 |
| no longer affects: | tempest |
| tags: | added: network |
| Changed in nova: | |
| assignee: | nobody → Aaron Rosen (arosen) |
| Changed in neutron: | |
| assignee: | nobody → Aaron Rosen (arosen) |
| Aaron Rosen (arosen) wrote | #8 |
| Changed in nova: | |
| status: | New → Incomplete |
| Changed in neutron: | |
| status: | New → Invalid |
| Changed in nova: | |
| status: | Incomplete → Invalid |
| Changed in neutron: | |
| assignee: | Aaron Rosen (arosen) → nobody |
| Changed in nova: | |
| assignee: | Aaron Rosen (arosen) → nobody |
I tried several times and I could not reproduce it with the provided script, maybe some configuration is missing in my devstack environment, although I could reproduce the nova delete/list sequence issue, but when the nova show is called it has task state 'deleting' and status ACTIVE (and that is why it is showed), so at least task state is changed, maybe we could change the behavior to not show/list servers in deleting state, IMHO it would be better.
Anyway, I don't think this is a bug but the natural downside of choose to do some resource operations asynchronously ( and I'm ok with it).
About tempest I don't think that is a problem cause we can use wait_for_
Thanks