I am doing simultaneous heat stack-update operations.
As a part of stack-update, we create a policy and attach policy to left-network and right-network.
But around 5-10 policies are not getting associated to right-network during 60 heat stack-update operations.
As per HOT template, left-net is a newly created resource and right-net is a parameter (already existing resource).
root@nfv-server04:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 2.21-102 102
contrail-config 2.21-102 102
root@nfv-server04:~# heat resource-list jsm-45273b9b-583f-4e15-be8e-4205e1a4c937
+--------------------+-------------------------------+-----------------+----------------------+
| resource_name | resource_type | resource_status | updated_time |
+--------------------+-------------------------------+-----------------+----------------------+
| left-net | OS::Contrail::VirtualNetwork | CREATE_COMPLETE | 2015-10-19T05:37:44Z |
| vFF-0-vnf-flavor | OS::JSM::GetFlavor | CREATE_COMPLETE | 2015-10-19T05:37:44Z |
| vFF-0-vnf-template | OS::Contrail::ServiceTemplate | CREATE_COMPLETE | 2015-10-19T05:37:45Z |
| left-subnet | OS::Neutron::Subnet | CREATE_COMPLETE | 2015-10-19T05:37:46Z |
| vFF-0-vnf-instance | OS::Contrail::ServiceInstance | CREATE_COMPLETE | 2015-10-19T05:37:49Z |
| site-access-policy | OS::Contrail::NetworkPolicy | CREATE_COMPLETE | 2015-10-19T05:54:26Z |
| ns-entry | OS::Contrail::AttachPolicy | CREATE_COMPLETE | 2015-10-19T05:54:27Z |
| ns-exit | OS::Contrail::AttachPolicy | CREATE_COMPLETE | 2015-10-19T05:54:27Z |
+--------------------+-------------------------------+-----------------+----------------------+
ns-entry: <<<<<< attaches policy to left-net
=================================
depends_on: [left-net, site-access-policy]
properties:
network: {get_resource: left-net}
policy:
get_attr: [site-access-policy, fq_name]
type: OS::Contrail::AttachPolicy
ns-exit: <<<<<< attaches policy to right-net
========================== =======
depends_on: [site-access-policy]
properties:
network: {get_param: right-net-id}
policy:
get_attr: [site-access-policy, fq_name]
type: OS::Contrail::AttachPolicy
site-access-policy:
=================================
depends_on: [left-net, vFF-0-vnf-instance]
properties:
entries:
policy_rule:
- action_list:
apply_service:
- {get_resource: vFF-0-vnf-instance}
direction: <>
dst_addresses:
- virtual_network: {get_param: right-net-id}
dst_ports:
- {end_port: -1, start_port: -1}
protocol: any
src_addresses:
- virtual_network: {get_resource: left-net}
src_ports:
- {end_port: -1, start_port: -1}
name:
Fn::Join:
- '-'
- - {get_param: nsid}
- cb9e2ff7-0683-4e8f-8c44-d3b23c055ffe-site-access-policy
type: OS::Contrail::NetworkPolicy
In Contrail GUI:
==============
For attached-failed policy:
=======================
Display Name
0689a692-7f45-47af-8c68-ce342339c4b9-7f2ecc86-0472-4ed4-bc6f-ff95aad6c60a-site-access-policy
UUID
5122d08c-fce2-42f4-8f6a-6f7239f7aac3
Associated Networks
45273b9b-583f-4e15-be8e-4205e1a4c937-left-net
Rules
pass protocol any network 0689a692-7f45-47af-8c68-ce342339c4b9-left-net ports any <> network right-net ports any services 0689a692-7f45-47af-8c68-ce342339c4b9-vFF-0-vnf-instance
For attached-success policy:
=======================
Display Name
e4c0e411-2dc7-4440-af16-cf813f3f1df2-49eac946-5f5b-43df-8223-1244e9ae9356-site-access-policy
UUID
4ec75a6f-e300-43e9-b626-9fab0117adbb
Associated Networks
right-net
e4c0e411-2dc7-4440-af16-cf813f3f1df2-left-net
Rules
pass protocol any network e4c0e411-2dc7-4440-af16-cf813f3f1df2-left-net ports any <> network right-net ports any services e4c0e411-2dc7-4440-af16-cf813f3f1df2-vFF-0-vnf-instance
From contrail-logs:
==================
VNC config API server trying to add 0689a692-7f45-47af-8c68-ce342339c4b9-7f2ecc86-0472-4ed4-bc6f-ff95aad6c60a-site-access-policy to 29th sequence.
But e9f19e97-0596-43d6-b9b9-59327552b0f7-be4ae337-5068-4e60-8dc9-ad10530bba85-site-access-policy got added to 29th sequence.
So my suspicion is 0689a692-7f45-47af-8c68-ce342339c4b9-7f2ecc86-0472-4ed4-bc6f-ff95aad6c60a-site-access-policy not getting attched to right-netwok.
2015 Oct 18 20:27:42.536410 nfv-server04 [:contrail-api:] : VncApiConfigLog: [VncApiCommon: identifier_uuid = 904b3ebd-bc94-4fd3-8087-69a728b02c4c, object_type = virtual_network, identifier_name = default-domain:admin:right-net, url = http://127.0.0.1:9100/virtual-network/904b3ebd-bc94-4fd3-8087-69a728b02c4c, operation = put, useragent = nfv-server04:/usr/bin/contrail-api, remote_ip = 127.0.0.1:9100, body = {'virtual-network': {'uuid': '904b3ebd-bc94-4fd3-8087-69a728b02c4c', 'network_policy_refs': [{'to': ['default-domain', 'admin', 'a507be3a-9279-4122-8720-30e9cb3091c4-9f81d4e3-eb19-4faa-bccd-6b6cacd654ea-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 0, 'minor': 0}}}, {'to': ['default-domain', 'admin', '8a73c4db-2387-4e0e-b558-86d37676a96e-ba115934-c181-4a07-ac21-eef57efec135-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 1, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'ede97471-94e3-4534-8bbf-e836c762d4ef-ca8cad49-5225-4628-842c-913e9a337ec4-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 2, 'minor': 0}}}, {'to': ['default-domain', 'admin', '91eeca46-6fb7-4581-86a6-d474dd98b088-95dc9444-f7d6-4271-9e13-d51382f26926-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 3, 'minor': 0}}}, {'to': ['default-domain', 'admin', '5245dcc0-c356-47a5-ba28-8820e240fa69-cc999275-6864-4de9-98ec-a92c7cf556ba-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 4, 'minor': 0}}}, {'to': ['default-domain', 'admin', '7c9f9a19-1ab4-4849-96d8-7eb6f0b7beeb-cb1ae85f-1bf6-48f8-9f22-773844cf3369-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 5, 'minor': 0}}}, {'to': ['default-domain', 'admin', '1c601eaa-617d-45e9-9f67-a8dbf3123d4b-c9ea864e-556d-4ae0-ba48-a50cba3d8607-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 6, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'b438e2ce-b125-45c4-ae8f-8a4d8e1d0f33-75bd8e5f-8d09-49df-bab2-ceee8a7189b6-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 7, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'ebcaff3a-ba94-4ac4-9a7f-ecd161e2e12e-3667a6c6-b671-42b5-a2ce-d215ebcbc063-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 8, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'd1318baa-3449-43e9-831d-d498015b9322-65672917-bd2b-400d-bb32-a21c46f941d8-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 9, 'minor': 0}}}, {'to': ['default-domain', 'admin', '848ebb34-9c26-462c-ab07-76939704ac9b-19b3448b-44af-41a9-998b-42c2cdcbbc22-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 10, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'e9d5f2cd-67bb-4322-82dc-c5fcde4a5ae8-00efd5b1-4953-4559-a5b2-8e43b55edde5-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 11, 'minor': 0}}}, {'to': ['default-domain', 'admin', '170808fc-ca22-4248-a001-280cb0161faf-26dce628-77a9-44a1-8bf7-9c8db689a491-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 12, 'minor': 0}}}, {'to': ['default-domain', 'admin', '4738af44-302a-4411-9ac8-fc9d444a6b21-668a4e7c-9908-4940-ab5c-8044739f40e2-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 13, 'minor': 0}}}, {'to': ['default-domain', 'admin', '2077770f-a75b-4f13-9c96-93ee4bab8922-745b058d-9f0b-4c10-9622-b6d27821bab8-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 14, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'b1d33544-8bf7-4556-801e-cd5250955822-21d60a88-da72-4bcd-9bcc-b0d658dbd592-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 15, 'minor': 0}}}, {'to': ['default-domain', 'admin', '4d678791-7a3a-4bdb-add9-a5e5cefa981c-1036475d-2136-4766-b9b7-f6de74d0e503-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 16, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'f487d81c-544d-4fda-aa7e-bdbc0bbbb03d-bf15ca3e-8e6e-4b29-9105-b4062ac429a7-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 17, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'fe6a9e51-d7ef-42ce-bfd3-8ff4aa2225af-3a59a4e3-5cf1-4e34-b5ef-dc7e531ddbf5-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 18, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'd81b8f8d-016a-44ff-960b-7741ae39cf38-2b1f45ca-de45-4919-8d65-6d7c119d083c-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 19, 'minor': 0}}}, {'to': ['default-domain', 'admin', '4dea9c68-6ca7-4056-aad3-36a1aacd4566-f7e3aeba-49b0-47cd-ab22-7dcd36983719-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 20, 'minor': 0}}}, {'to': ['default-domain', 'admin', '2d811420-066a-4aad-9b51-1b57024e2d96-7b9fa042-8914-42c8-80e8-50771672a301-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 21, 'minor': 0}}}, {'to': ['default-domain', 'admin', '6339d193-e42b-4c3f-aa80-d5706af8724a-889b15a7-056b-4180-9b45-b77eb40736bb-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 22, 'minor': 0}}}, {'to': ['default-domain', 'admin', '89950e6c-0ec7-4f07-b66c-a814d561af2e-27b82441-b60c-4151-921f-7fb8f1c482b8-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 23, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'aa0a6fc1-da56-4276-89e8-dc481113bdcf-0234a275-7629-46a4-aec8-e8360203c2fd-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 24, 'minor': 0}}}, {'to': ['default-domain', 'admin', '2745378c-2de4-4396-95a2-bf347978115f-b4593ce8-4d79-4202-afb6-bea45b1a685f-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 25, 'minor': 0}}}, {'to': ['default-domain', 'admin', '3993195f-27d3-445f-a4c4-098af5fc5a17-601c1fec-627b-414c-98b7-16a6544b9fbf-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 26, 'minor': 0}}}, {'to': ['default-domain', 'admin', '67e023eb-5175-4d62-9aae-37b29a4c6c43-519ca04f-7493-48ce-ad21-6e1fc3918344-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 27, 'minor': 0}}}, {'to': ['default-domain', 'admin', '044b3b2d-add8-47ab-8991-4d6ffd7de92c-98e8fe46-22d6-4e8c-8ebf-707ae0486fb4-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 28, 'minor': 0}}}, {'to': ['default-domain', 'admin', '0689a692-7f45-47af-8c68-ce342339c4b9-7f2ecc86-0472-4ed4-bc6f-ff95aad6c60a-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 29, 'minor': 0}}}]}}, domain = default-domain, project = admin, user = admin]
2015 Oct 18 20:28:46.753706 nfv-server04 [:contrail-api:] : VncApiConfigLog: [VncApiCommon: identifier_uuid = 904b3ebd-bc94-4fd3-8087-69a728b02c4c, object_type = virtual_network, identifier_name = default-domain:admin:right-net, url = http://127.0.0.1:9100/virtual-network/904b3ebd-bc94-4fd3-8087-69a728b02c4c, operation = put, useragent = nfv-server04:/usr/bin/contrail-api, remote_ip = 127.0.0.1:9100, body = {'virtual-network': {'uuid': '904b3ebd-bc94-4fd3-8087-69a728b02c4c', 'network_policy_refs': [{'to': ['default-domain', 'admin', 'a507be3a-9279-4122-8720-30e9cb3091c4-9f81d4e3-eb19-4faa-bccd-6b6cacd654ea-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 0, 'minor': 0}}}, {'to': ['default-domain', 'admin', '8a73c4db-2387-4e0e-b558-86d37676a96e-ba115934-c181-4a07-ac21-eef57efec135-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 1, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'ede97471-94e3-4534-8bbf-e836c762d4ef-ca8cad49-5225-4628-842c-913e9a337ec4-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 2, 'minor': 0}}}, {'to': ['default-domain', 'admin', '91eeca46-6fb7-4581-86a6-d474dd98b088-95dc9444-f7d6-4271-9e13-d51382f26926-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 3, 'minor': 0}}}, {'to': ['default-domain', 'admin', '5245dcc0-c356-47a5-ba28-8820e240fa69-cc999275-6864-4de9-98ec-a92c7cf556ba-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 4, 'minor': 0}}}, {'to': ['default-domain', 'admin', '7c9f9a19-1ab4-4849-96d8-7eb6f0b7beeb-cb1ae85f-1bf6-48f8-9f22-773844cf3369-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 5, 'minor': 0}}}, {'to': ['default-domain', 'admin', '1c601eaa-617d-45e9-9f67-a8dbf3123d4b-c9ea864e-556d-4ae0-ba48-a50cba3d8607-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 6, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'b438e2ce-b125-45c4-ae8f-8a4d8e1d0f33-75bd8e5f-8d09-49df-bab2-ceee8a7189b6-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 7, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'ebcaff3a-ba94-4ac4-9a7f-ecd161e2e12e-3667a6c6-b671-42b5-a2ce-d215ebcbc063-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 8, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'd1318baa-3449-43e9-831d-d498015b9322-65672917-bd2b-400d-bb32-a21c46f941d8-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 9, 'minor': 0}}}, {'to': ['default-domain', 'admin', '848ebb34-9c26-462c-ab07-76939704ac9b-19b3448b-44af-41a9-998b-42c2cdcbbc22-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 10, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'e9d5f2cd-67bb-4322-82dc-c5fcde4a5ae8-00efd5b1-4953-4559-a5b2-8e43b55edde5-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 11, 'minor': 0}}}, {'to': ['default-domain', 'admin', '170808fc-ca22-4248-a001-280cb0161faf-26dce628-77a9-44a1-8bf7-9c8db689a491-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 12, 'minor': 0}}}, {'to': ['default-domain', 'admin', '4738af44-302a-4411-9ac8-fc9d444a6b21-668a4e7c-9908-4940-ab5c-8044739f40e2-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 13, 'minor': 0}}}, {'to': ['default-domain', 'admin', '2077770f-a75b-4f13-9c96-93ee4bab8922-745b058d-9f0b-4c10-9622-b6d27821bab8-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 14, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'b1d33544-8bf7-4556-801e-cd5250955822-21d60a88-da72-4bcd-9bcc-b0d658dbd592-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 15, 'minor': 0}}}, {'to': ['default-domain', 'admin', '4d678791-7a3a-4bdb-add9-a5e5cefa981c-1036475d-2136-4766-b9b7-f6de74d0e503-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 16, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'f487d81c-544d-4fda-aa7e-bdbc0bbbb03d-bf15ca3e-8e6e-4b29-9105-b4062ac429a7-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 17, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'fe6a9e51-d7ef-42ce-bfd3-8ff4aa2225af-3a59a4e3-5cf1-4e34-b5ef-dc7e531ddbf5-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 18, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'd81b8f8d-016a-44ff-960b-7741ae39cf38-2b1f45ca-de45-4919-8d65-6d7c119d083c-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 19, 'minor': 0}}}, {'to': ['default-domain', 'admin', '4dea9c68-6ca7-4056-aad3-36a1aacd4566-f7e3aeba-49b0-47cd-ab22-7dcd36983719-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 20, 'minor': 0}}}, {'to': ['default-domain', 'admin', '2d811420-066a-4aad-9b51-1b57024e2d96-7b9fa042-8914-42c8-80e8-50771672a301-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 21, 'minor': 0}}}, {'to': ['default-domain', 'admin', '6339d193-e42b-4c3f-aa80-d5706af8724a-889b15a7-056b-4180-9b45-b77eb40736bb-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 22, 'minor': 0}}}, {'to': ['default-domain', 'admin', '89950e6c-0ec7-4f07-b66c-a814d561af2e-27b82441-b60c-4151-921f-7fb8f1c482b8-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 23, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'aa0a6fc1-da56-4276-89e8-dc481113bdcf-0234a275-7629-46a4-aec8-e8360203c2fd-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 24, 'minor': 0}}}, {'to': ['default-domain', 'admin', '2745378c-2de4-4396-95a2-bf347978115f-b4593ce8-4d79-4202-afb6-bea45b1a685f-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 25, 'minor': 0}}}, {'to': ['default-domain', 'admin', '3993195f-27d3-445f-a4c4-098af5fc5a17-601c1fec-627b-414c-98b7-16a6544b9fbf-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 26, 'minor': 0}}}, {'to': ['default-domain', 'admin', '67e023eb-5175-4d62-9aae-37b29a4c6c43-519ca04f-7493-48ce-ad21-6e1fc3918344-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 27, 'minor': 0}}}, {'to': ['default-domain', 'admin', '044b3b2d-add8-47ab-8991-4d6ffd7de92c-98e8fe46-22d6-4e8c-8ebf-707ae0486fb4-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 28, 'minor': 0}}}, {'to': ['default-domain', 'admin', 'e9f19e97-0596-43d6-b9b9-59327552b0f7-be4ae337-5068-4e60-8dc9-ad10530bba85-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 29, 'minor': 0}}}, {'to': ['default-domain', 'admin', '88108b5a-cf7a-4538-83a8-e967699a412f-8c24dfd7-0b0e-4e9d-80f9-385ecb9294c1-site-access-policy'], 'attr': {'timer': None, 'sequence': {'major': 30, 'minor': 0}}}]}}, domain = default-domain, project = admin, user = admin]
Heat resource implementation for attach_policy fetches the existing policy and appends the new policy.
In race condition, attach_policy overwrites the previously added network policy on VN.
Solution: Use ref_update ADD/DELETE to add new network policy or remove the network policy.