| 2014-01-15 13:04:27 |
Yair Fried |
bug |
|
|
added bug |
| 2014-01-15 13:04:27 |
Yair Fried |
attachment added |
|
bash script to reproduce bug via CLI https://bugs.launchpad.net/bugs/1269407/+attachment/3949453/+files/devstack_script.sh |
|
| 2014-01-15 13:04:43 |
Yair Fried |
bug task added |
|
neutron |
|
| 2014-01-15 13:05:02 |
Yair Fried |
bug task added |
|
tempest |
|
| 2014-01-15 13:06:47 |
Yair Fried |
description |
When deleting an instance, the port list is not updated immediately. As a result - deleting net/subnet fails with error:
409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
(*) Happens only on automated scripts, since manual input isn't quick enough to catch this.
(**) Happens only when Floating IP is attached - error doesn't happen when Floating IP isn't used.
my guess: Nova delays in deleting the VM while checking with neutron DB that Floating IP was deleted.
(***) nova delete command returns BEFORE instance is deleted:
> nova delete $instance_id ; nova show $instance_id
will return the instance without failure
How to reproduce:
assumes:
1. external network exists with floating ip range available ("public")
2. image exists
Setup:
1 create network "private"
2. create subnet
3. create router:
3.1 set router gateway to "public"
3.2 set router interface to "private"
4. create VM
5. assign Floating IP to VM
TearDown
1. Delete / Disassociate Floating IP
2. Delete VM
3. detach router interface from subnet (router-interface-delete)
4. Delete subnet/net
Expected Result:
subnet/net should be successfully deleted.
Actual Results:
"Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet"
409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
script log:
line 101 - VM port still in port list even though VM was deleted
line 105 - subnet fails to delete
line 117 - network successfully deleted after enough time passed for port list to update
1 + EXT_NET_NAME=public
2 + NET_NAME=my_net
3 + SUBNET_NAME=my_subnet
4 + ROUTER_NAME=my_router
5 + SERVER_NAME=my_server
6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
7 + MASK=54.0.0
8 + SERVER_IP=54.0.0.6
9 ++ neutron net-list
10 ++ grep public
11 ++ awk '{print $2;}'
12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
13 ++ nova image-list
14 ++ grep -w ' cirros-0.3.1-x86_64-uec '
15 ++ awk '{print $2;}'
16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
17 ++ grep -w id
18 ++ neutron net-create my_net
19 ++ awk '{print $4;}'
20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
22 ++ grep -w id
23 ++ awk '{print $4;}'
24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
25 ++ neutron router-create my_router
26 ++ awk '{print $4;}'
27 ++ grep -w id
28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
34 +--------------------------------------+--------------------------------------+
35 | Property | Value |
36 +--------------------------------------+--------------------------------------+
37 | OS-DCF:diskConfig | MANUAL |
38 | OS-EXT-AZ:availability_zone | nova |
39 | OS-EXT-STS:power_state | 0 |
40 | OS-EXT-STS:task_state | scheduling |
41 | OS-EXT-STS:vm_state | building |
42 | OS-SRV-USG:launched_at | None |
43 | OS-SRV-USG:terminated_at | None |
44 | accessIPv4 | |
45 | accessIPv6 | |
46 | adminPass | 9VJ9XMrtFghW |
47 | config_drive | |
48 | created | 2014-01-15T12:56:30Z |
49 | flavor | m1.small |
50 | hostId | |
51 | id | d09bbee1-599c-4042-a409-a90e0ae80e3c |
52 | image | cirros-0.3.1-x86_64-uec |
53 | key_name | None |
54 | metadata | {} |
55 | name | my_server |
56 | os-extended-volumes:volumes_attached | [] |
57 | progress | 0 |
58 | security_groups | [{"name": "default"}] |
59 | status | BUILD |
60 | tenant_id | 66293febf7164c849b694a8d3f14cc1a |
61 | updated | 2014-01-15T12:56:30Z |
62 | user_id | e895d5bf0dc64d44b707b9fbf5cbf6e0 |
63 +--------------------------------------+--------------------------------------+
64 ++ nova list
65 ++ grep my_server
66 ++ awk '{print $2;}'
67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
68 ++ neutron port-list
69 ++ grep 54.0.0.6
70 ++ awk '{print $2;}'
71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
73 ++ grep floating_ip_address
74 ++ awk '{print $4;}'
75 + FLOATING_IP=172.24.4.4
76 ++ neutron floatingip-list
77 ++ awk '{print $2;}'
78 ++ grep 172.24.4.4
79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
80 + neutron port-list
81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
82 | id | name | mac_address | fixed_ips |
83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
85 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
88 | cafd4161-f840-4c87-a80b-71b0ef374b9e | | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
95 + neutron port-list
96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
97 | id | name | mac_address | fixed_ips |
98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
100 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
101 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
102 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
103 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
104 + neutron subnet-delete 76abfa0f-938a-4be1-abd5-804af306fa2d
105 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 76abfa0f-938a-4be1-abd5-804af306fa2d. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
106 + neutron port-list
107 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
108 | id | name | mac_address | fixed_ips |
109 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
110 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
111 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
112 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
113 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
114 + neutron router-delete df211133-0513-44fc-bec5-38f9bca74025
115 Deleted router: df211133-0513-44fc-bec5-38f9bca74025
116 + neutron net-delete 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
117 Deleted network: 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
118 + set +x |
When deleting an instance, the port list is not updated immediately. As a result - deleting net/subnet fails with error:
409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
(*) Happens only on automated scripts, since manual input isn't quick enough to catch this.
(**) Happens only when Floating IP is attached - error doesn't happen when Floating IP isn't used.
my guess: Nova delays in deleting the VM while checking with neutron DB that Floating IP was deleted.
(***) nova delete command returns BEFORE instance is deleted:
> nova delete $instance_id ; nova show $instance_id
will return the instance without failure
(*****) might also affect Tempest during tearDown execution
How to reproduce:
**** script attached ****
assumes:
1. external network exists with floating ip range available ("public")
2. image exists
Setup:
1 create network "private"
2. create subnet
3. create router:
3.1 set router gateway to "public"
3.2 set router interface to "private"
4. create VM
5. assign Floating IP to VM
TearDown
1. Delete / Disassociate Floating IP
2. Delete VM
3. detach router interface from subnet (router-interface-delete)
4. Delete subnet/net
Expected Result:
subnet/net should be successfully deleted.
Actual Results:
"Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet"
409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
script log:
line 101 - VM port still in port list even though VM was deleted
line 105 - subnet fails to delete
line 117 - network successfully deleted after enough time passed for port list to update
1 + EXT_NET_NAME=public
2 + NET_NAME=my_net
3 + SUBNET_NAME=my_subnet
4 + ROUTER_NAME=my_router
5 + SERVER_NAME=my_server
6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
7 + MASK=54.0.0
8 + SERVER_IP=54.0.0.6
9 ++ neutron net-list
10 ++ grep public
11 ++ awk '{print $2;}'
12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
13 ++ nova image-list
14 ++ grep -w ' cirros-0.3.1-x86_64-uec '
15 ++ awk '{print $2;}'
16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
17 ++ grep -w id
18 ++ neutron net-create my_net
19 ++ awk '{print $4;}'
20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
22 ++ grep -w id
23 ++ awk '{print $4;}'
24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
25 ++ neutron router-create my_router
26 ++ awk '{print $4;}'
27 ++ grep -w id
28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
34 +--------------------------------------+--------------------------------------+
35 | Property | Value |
36 +--------------------------------------+--------------------------------------+
37 | OS-DCF:diskConfig | MANUAL |
38 | OS-EXT-AZ:availability_zone | nova |
39 | OS-EXT-STS:power_state | 0 |
40 | OS-EXT-STS:task_state | scheduling |
41 | OS-EXT-STS:vm_state | building |
42 | OS-SRV-USG:launched_at | None |
43 | OS-SRV-USG:terminated_at | None |
44 | accessIPv4 | |
45 | accessIPv6 | |
46 | adminPass | 9VJ9XMrtFghW |
47 | config_drive | |
48 | created | 2014-01-15T12:56:30Z |
49 | flavor | m1.small |
50 | hostId | |
51 | id | d09bbee1-599c-4042-a409-a90e0ae80e3c |
52 | image | cirros-0.3.1-x86_64-uec |
53 | key_name | None |
54 | metadata | {} |
55 | name | my_server |
56 | os-extended-volumes:volumes_attached | [] |
57 | progress | 0 |
58 | security_groups | [{"name": "default"}] |
59 | status | BUILD |
60 | tenant_id | 66293febf7164c849b694a8d3f14cc1a |
61 | updated | 2014-01-15T12:56:30Z |
62 | user_id | e895d5bf0dc64d44b707b9fbf5cbf6e0 |
63 +--------------------------------------+--------------------------------------+
64 ++ nova list
65 ++ grep my_server
66 ++ awk '{print $2;}'
67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
68 ++ neutron port-list
69 ++ grep 54.0.0.6
70 ++ awk '{print $2;}'
71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
73 ++ grep floating_ip_address
74 ++ awk '{print $4;}'
75 + FLOATING_IP=172.24.4.4
76 ++ neutron floatingip-list
77 ++ awk '{print $2;}'
78 ++ grep 172.24.4.4
79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
80 + neutron port-list
81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
82 | id | name | mac_address | fixed_ips |
83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
85 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
88 | cafd4161-f840-4c87-a80b-71b0ef374b9e | | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
95 + neutron port-list
96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
97 | id | name | mac_address | fixed_ips |
98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
100 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
101 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
102 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
103 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
104 + neutron subnet-delete 76abfa0f-938a-4be1-abd5-804af306fa2d
105 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 76abfa0f-938a-4be1-abd5-804af306fa2d. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
106 + neutron port-list
107 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
108 | id | name | mac_address | fixed_ips |
109 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
110 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
111 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
112 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
113 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
114 + neutron router-delete df211133-0513-44fc-bec5-38f9bca74025
115 Deleted router: df211133-0513-44fc-bec5-38f9bca74025
116 + neutron net-delete 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
117 Deleted network: 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
118 + set +x |
|
| 2014-01-15 13:07:14 |
Yair Fried |
bug |
|
|
added subscriber Assaf Muller |
| 2014-01-15 13:07:49 |
Yair Fried |
bug |
|
|
added subscriber ofer blaut |
| 2014-01-15 13:19:12 |
Yair Fried |
description |
When deleting an instance, the port list is not updated immediately. As a result - deleting net/subnet fails with error:
409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
(*) Happens only on automated scripts, since manual input isn't quick enough to catch this.
(**) Happens only when Floating IP is attached - error doesn't happen when Floating IP isn't used.
my guess: Nova delays in deleting the VM while checking with neutron DB that Floating IP was deleted.
(***) nova delete command returns BEFORE instance is deleted:
> nova delete $instance_id ; nova show $instance_id
will return the instance without failure
(*****) might also affect Tempest during tearDown execution
How to reproduce:
**** script attached ****
assumes:
1. external network exists with floating ip range available ("public")
2. image exists
Setup:
1 create network "private"
2. create subnet
3. create router:
3.1 set router gateway to "public"
3.2 set router interface to "private"
4. create VM
5. assign Floating IP to VM
TearDown
1. Delete / Disassociate Floating IP
2. Delete VM
3. detach router interface from subnet (router-interface-delete)
4. Delete subnet/net
Expected Result:
subnet/net should be successfully deleted.
Actual Results:
"Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet"
409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
script log:
line 101 - VM port still in port list even though VM was deleted
line 105 - subnet fails to delete
line 117 - network successfully deleted after enough time passed for port list to update
1 + EXT_NET_NAME=public
2 + NET_NAME=my_net
3 + SUBNET_NAME=my_subnet
4 + ROUTER_NAME=my_router
5 + SERVER_NAME=my_server
6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
7 + MASK=54.0.0
8 + SERVER_IP=54.0.0.6
9 ++ neutron net-list
10 ++ grep public
11 ++ awk '{print $2;}'
12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
13 ++ nova image-list
14 ++ grep -w ' cirros-0.3.1-x86_64-uec '
15 ++ awk '{print $2;}'
16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
17 ++ grep -w id
18 ++ neutron net-create my_net
19 ++ awk '{print $4;}'
20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
22 ++ grep -w id
23 ++ awk '{print $4;}'
24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
25 ++ neutron router-create my_router
26 ++ awk '{print $4;}'
27 ++ grep -w id
28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
34 +--------------------------------------+--------------------------------------+
35 | Property | Value |
36 +--------------------------------------+--------------------------------------+
37 | OS-DCF:diskConfig | MANUAL |
38 | OS-EXT-AZ:availability_zone | nova |
39 | OS-EXT-STS:power_state | 0 |
40 | OS-EXT-STS:task_state | scheduling |
41 | OS-EXT-STS:vm_state | building |
42 | OS-SRV-USG:launched_at | None |
43 | OS-SRV-USG:terminated_at | None |
44 | accessIPv4 | |
45 | accessIPv6 | |
46 | adminPass | 9VJ9XMrtFghW |
47 | config_drive | |
48 | created | 2014-01-15T12:56:30Z |
49 | flavor | m1.small |
50 | hostId | |
51 | id | d09bbee1-599c-4042-a409-a90e0ae80e3c |
52 | image | cirros-0.3.1-x86_64-uec |
53 | key_name | None |
54 | metadata | {} |
55 | name | my_server |
56 | os-extended-volumes:volumes_attached | [] |
57 | progress | 0 |
58 | security_groups | [{"name": "default"}] |
59 | status | BUILD |
60 | tenant_id | 66293febf7164c849b694a8d3f14cc1a |
61 | updated | 2014-01-15T12:56:30Z |
62 | user_id | e895d5bf0dc64d44b707b9fbf5cbf6e0 |
63 +--------------------------------------+--------------------------------------+
64 ++ nova list
65 ++ grep my_server
66 ++ awk '{print $2;}'
67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
68 ++ neutron port-list
69 ++ grep 54.0.0.6
70 ++ awk '{print $2;}'
71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
73 ++ grep floating_ip_address
74 ++ awk '{print $4;}'
75 + FLOATING_IP=172.24.4.4
76 ++ neutron floatingip-list
77 ++ awk '{print $2;}'
78 ++ grep 172.24.4.4
79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
80 + neutron port-list
81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
82 | id | name | mac_address | fixed_ips |
83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
85 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
88 | cafd4161-f840-4c87-a80b-71b0ef374b9e | | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
95 + neutron port-list
96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
97 | id | name | mac_address | fixed_ips |
98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
100 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
101 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
102 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
103 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
104 + neutron subnet-delete 76abfa0f-938a-4be1-abd5-804af306fa2d
105 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 76abfa0f-938a-4be1-abd5-804af306fa2d. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
106 + neutron port-list
107 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
108 | id | name | mac_address | fixed_ips |
109 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
110 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
111 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
112 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
113 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
114 + neutron router-delete df211133-0513-44fc-bec5-38f9bca74025
115 Deleted router: df211133-0513-44fc-bec5-38f9bca74025
116 + neutron net-delete 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
117 Deleted network: 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
118 + set +x |
When deleting an instance, the port list is not updated immediately. As a result - deleting net/subnet fails with error:
409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
(*) Happens only on automated scripts, since manual input isn't quick enough to catch this.
(**) Happens only when Floating IP is attached - error doesn't happen when Floating IP isn't used.
my guess: Nova delays in deleting the VM while checking with neutron DB that Floating IP was deleted.
(***) nova delete command returns BEFORE instance is deleted:
> nova delete $instance_id ; nova show $instance_id
will return the instance without failure
(*****) might also affect Tempest during tearDown execution
Version:
openstack-nova-api-2014.1-0.5.b1.el6.noarch
openstack-nova-compute-2014.1-0.5.b1.el6.noarch
openstack-nova-scheduler-2014.1-0.5.b1.el6.noarch
openstack-nova-console-2014.1-0.5.b1.el6.noarch
openstack-nova-conductor-2014.1-0.5.b1.el6.noarch
openstack-nova-cert-2014.1-0.5.b1.el6.noarch
python-neutron-2014.1-0.1.b1.el6.noarch
openstack-neutron-2014.1-0.1.b1.el6.noarch
openstack-neutron-openvswitch-2014.1-0.1.b1.el6.noarch
How to reproduce:
**** script attached ****
assumes:
1. external network exists with floating ip range available ("public")
2. image exists
Setup:
1 create network "private"
2. create subnet
3. create router:
3.1 set router gateway to "public"
3.2 set router interface to "private"
4. create VM
5. assign Floating IP to VM
TearDown
1. Delete / Disassociate Floating IP
2. Delete VM
3. detach router interface from subnet (router-interface-delete)
4. Delete subnet/net
Expected Result:
subnet/net should be successfully deleted.
Actual Results:
"Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet"
409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
script log:
line 101 - VM port still in port list even though VM was deleted
line 105 - subnet fails to delete
line 117 - network successfully deleted after enough time passed for port list to update
1 + EXT_NET_NAME=public
2 + NET_NAME=my_net
3 + SUBNET_NAME=my_subnet
4 + ROUTER_NAME=my_router
5 + SERVER_NAME=my_server
6 + IMAGE_NAME='cirros-0.3.1-x86_64-uec '
7 + MASK=54.0.0
8 + SERVER_IP=54.0.0.6
9 ++ neutron net-list
10 ++ grep public
11 ++ awk '{print $2;}'
12 + EXT_NET_ID=200a91cf-5376-4095-8722-2f247ddb01c9
13 ++ nova image-list
14 ++ grep -w ' cirros-0.3.1-x86_64-uec '
15 ++ awk '{print $2;}'
16 + IMAGE_ID=1f16b297-aeaa-4fa9-9640-269695b6eb48
17 ++ grep -w id
18 ++ neutron net-create my_net
19 ++ awk '{print $4;}'
20 + NET_ID=6ec5ef65-5279-4bbd-919a-b45a27bb31cd
21 ++ neutron subnet-create --name my_subnet 6ec5ef65-5279-4bbd-919a-b45a27bb31cd 54.0.0.0/24
22 ++ grep -w id
23 ++ awk '{print $4;}'
24 + SUBNET_ID=76abfa0f-938a-4be1-abd5-804af306fa2d
25 ++ neutron router-create my_router
26 ++ awk '{print $4;}'
27 ++ grep -w id
28 + ROUTER_ID=df211133-0513-44fc-bec5-38f9bca74025
29 + neutron router-gateway-set df211133-0513-44fc-bec5-38f9bca74025 200a91cf-5376-4095-8722-2f247ddb01c9
30 Set gateway for router df211133-0513-44fc-bec5-38f9bca74025
31 + neutron router-interface-add df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
32 Added interface cafd4161-f840-4c87-a80b-71b0ef374b9e to router df211133-0513-44fc-bec5-38f9bca74025.
33 + nova boot --flavor 2 --image 1f16b297-aeaa-4fa9-9640-269695b6eb48 my_server --nic net-id=6ec5ef65-5279-4bbd-919a-b45a27bb31cd,v4-fixed-ip=54.0.0.6
34 +--------------------------------------+--------------------------------------+
35 | Property | Value |
36 +--------------------------------------+--------------------------------------+
37 | OS-DCF:diskConfig | MANUAL |
38 | OS-EXT-AZ:availability_zone | nova |
39 | OS-EXT-STS:power_state | 0 |
40 | OS-EXT-STS:task_state | scheduling |
41 | OS-EXT-STS:vm_state | building |
42 | OS-SRV-USG:launched_at | None |
43 | OS-SRV-USG:terminated_at | None |
44 | accessIPv4 | |
45 | accessIPv6 | |
46 | adminPass | 9VJ9XMrtFghW |
47 | config_drive | |
48 | created | 2014-01-15T12:56:30Z |
49 | flavor | m1.small |
50 | hostId | |
51 | id | d09bbee1-599c-4042-a409-a90e0ae80e3c |
52 | image | cirros-0.3.1-x86_64-uec |
53 | key_name | None |
54 | metadata | {} |
55 | name | my_server |
56 | os-extended-volumes:volumes_attached | [] |
57 | progress | 0 |
58 | security_groups | [{"name": "default"}] |
59 | status | BUILD |
60 | tenant_id | 66293febf7164c849b694a8d3f14cc1a |
61 | updated | 2014-01-15T12:56:30Z |
62 | user_id | e895d5bf0dc64d44b707b9fbf5cbf6e0 |
63 +--------------------------------------+--------------------------------------+
64 ++ nova list
65 ++ grep my_server
66 ++ awk '{print $2;}'
67 + SERVER_ID=d09bbee1-599c-4042-a409-a90e0ae80e3c
68 ++ neutron port-list
69 ++ grep 54.0.0.6
70 ++ awk '{print $2;}'
71 + SERVER_PORT_ID=21981e00-6fe2-4f6e-a265-d0c357a1a449
72 ++ neutron floatingip-create 200a91cf-5376-4095-8722-2f247ddb01c9 --port_id 21981e00-6fe2-4f6e-a265-d0c357a1a449
73 ++ grep floating_ip_address
74 ++ awk '{print $4;}'
75 + FLOATING_IP=172.24.4.4
76 ++ neutron floatingip-list
77 ++ awk '{print $2;}'
78 ++ grep 172.24.4.4
79 + FLOATING_IP_ID=1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
80 + neutron port-list
81 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
82 | id | name | mac_address | fixed_ips |
83 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
84 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
85 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
86 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
87 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
88 | cafd4161-f840-4c87-a80b-71b0ef374b9e | | fa:16:3e:e3:75:36 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.1"} |
89 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
90 + neutron floatingip-delete 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
91 Deleted floatingip: 1d77d791-9205-4ee4-8ae4-1a1dcf8dd153
92 + nova delete d09bbee1-599c-4042-a409-a90e0ae80e3c
93 + neutron router-interface-delete df211133-0513-44fc-bec5-38f9bca74025 76abfa0f-938a-4be1-abd5-804af306fa2d
94 Removed interface from router df211133-0513-44fc-bec5-38f9bca74025.
95 + neutron port-list
96 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
97 | id | name | mac_address | fixed_ips |
98 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
99 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
100 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
101 | 21981e00-6fe2-4f6e-a265-d0c357a1a449 | | fa:16:3e:38:5f:74 | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.6"} |
102 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
103 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
104 + neutron subnet-delete 76abfa0f-938a-4be1-abd5-804af306fa2d
105 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet 76abfa0f-938a-4be1-abd5-804af306fa2d. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
106 + neutron port-list
107 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
108 | id | name | mac_address | fixed_ips |
109 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
110 | 06b3022f-5492-4baf-b7d9-43f0259337e3 | | fa:16:3e:70:9e:bb | {"subnet_id": "76abfa0f-938a-4be1-abd5-804af306fa2d", "ip_address": "54.0.0.2"} |
111 | 13630e5f-51c2-42b9-901a-2a208e5f986a | | fa:16:3e:08:63:0b | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.2"} |
112 | 2a0a4134-37f2-4582-86ad-de0ce04b8f7e | | fa:16:3e:aa:0d:b1 | {"subnet_id": "2f2bc3c9-2daa-477e-a38c-e7a02b03e06e", "ip_address": "10.0.0.1"} |
113 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
114 + neutron router-delete df211133-0513-44fc-bec5-38f9bca74025
115 Deleted router: df211133-0513-44fc-bec5-38f9bca74025
116 + neutron net-delete 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
117 Deleted network: 6ec5ef65-5279-4bbd-919a-b45a27bb31cd
118 + set +x |
|
| 2014-02-23 16:11:12 |
Yair Fried |
bug task deleted |
tempest |
|
|
| 2014-03-05 16:42:50 |
Tracy Jones |
tags |
|
network |
|
| 2014-03-19 07:24:26 |
Aaron Rosen |
nova: assignee |
|
Aaron Rosen (arosen) |
|
| 2014-03-19 07:24:29 |
Aaron Rosen |
neutron: assignee |
|
Aaron Rosen (arosen) |
|
| 2014-03-19 17:29:59 |
Aaron Rosen |
nova: status |
New |
Incomplete |
|
| 2014-03-19 17:30:04 |
Aaron Rosen |
neutron: status |
New |
Invalid |
|
| 2014-03-19 17:30:41 |
Aaron Rosen |
nova: status |
Incomplete |
Invalid |
|
| 2014-03-19 17:30:53 |
Aaron Rosen |
neutron: assignee |
Aaron Rosen (arosen) |
|
|
| 2014-03-19 17:30:57 |
Aaron Rosen |
nova: assignee |
Aaron Rosen (arosen) |
|
|
