By using jcore-cumulus01 POD, I executed the following scale testing with the result:
User | # of VM each | Total VM | Result
====================================================
1 | 16 | 16 | PASSED
3 | 16 | 48 | PASSED
5 | 16 | 80 | PASSED
7 | 16 | 112 | PASSED
10 | 16 | 160 | FAILED
PASSED: Each user is able to perform binding/unbinding successfully without any error
Note: if the system run into resources issues, the system should not run into non-recoverable error where the VM is failed to delete. (report in detail below)
Topology used:
My topology that I used for testing is a 16VM with the combination of unix host, VMX Phase2, Phase 3 and FPC images. As below
-------------------
#
# TOPOLOGY
#
#
# [ Unix1]----[ RT1 ]--[ RT2 ]--[ RT3 ]--[ RT4 ]--[unix2]
#
# Unix1: cirros unix host
# RT1: VMXPhase 2 ( 2 x RE , 1 x FPC )
# RT2: VMXPhase 3 ( 2 x RE, 2 x FPC ) Use 1 FPC first. config issue
# RT3: VMXPhase 3 ( 2 x RE, 2 x FPC ) Use 1 FPC first, config issue
# RT4: VMXPhase 2 ( 2 x RE , 1 x FPC )
# Unix1: ubuntu unix host
#
# Image List: VMXphase 2: 15.1F-1221-Occam -> junos-x86-32-15.1F-20151221.0.vmdk
# VMXPhase 3 RE: vmxp3-re -> jinstall64-vmx-15.1F-20160301.0-domestic.img
# VMXPhase 3 FPC: vmxp3-FPC -> vFPC-20151211.vmdk
-------------------
How to Reproduce
=================
# Trail run for 10 users cu-01 for binding-testing
./perf_script_main.sh 10 1 ssslau 86000 /opt/juniper/cumulus/perf/slau_script/Occam_151F_2VMXp2_2VMXp3_2Unix.conf /opt/juniper/cumulus/perf/slau_script/result/10userbind-129.log /opt/juniper/cumulus/perf/slau_script/result/10userbind-129.csv BINDING-TIME 10 10 "YES"
Problem
========
- By execute the script, each of the user will report binding error as below:
--------
ERROR : stack creation failed Reason:Resource CREATE failed: Error: Server r2_re1 delete failed: (500) Build of instance 3aa7f410-ccf3-4ce3-a45b-d7537b0481b5 aborted: Could not clean up failed build, not rescheduling
-------
- Given this, each user will run into non-recoverable error where the VM is failed to delete
-------------------
root@jcore-cumulus01:/opt/juniper/cumulus/perf/slau_script/result/129-build# nova list --all_tenants | grep ERROR
| d73dc98a-fd18-46db-a955-9b0f69b4becc | RT1_fpc0 | ERROR | deleting | NOSTATE | |
| 760003e4-a6d3-4ac2-84ac-945c71d813a0 | RT1_fpc1 | ERROR | deleting | NOSTATE | |
| 5c81ef23-ec72-4004-9096-bac99337fad4 | RT2_fpc0 | ERROR | deleting | NOSTATE | |
| e27c8a30-fddc-46ad-995c-caa13afa5949 | RT2_fpc0 | ERROR | deleting | NOSTATE | |
| 237b7ef7-e6c2-4069-846c-53fad244f00d | RT3_FPC0 | ERROR | deleting | NOSTATE | |
| 2737cd87-48a8-4150-b6ec-193a124eaf1c | RT3_FPC0 | ERROR | deleting | NOSTATE | |
| 403886ad-4b6c-4614-9604-7a85c772c4c6 | RT3_FPC0 | ERROR | deleting | NOSTATE | |
| 5365ca18-69a2-4390-b9f0-bb14cef5617d | RT3_FPC0 | ERROR | deleting | NOSTATE | |
| 92513169-1999-40da-ac7f-db58bc96e681 | RT3_FPC0 | ERROR | deleting | NOSTATE | |
| ca4c4649-44ff-44a2-85cf-27d950f7d044 | RT3_FPC0 | ERROR | deleting | NOSTATE | |
| 355b3f3c-7463-4b9b-a8da-9c73e7d5a97f | RT3_FPC1 | ERROR | deleting | NOSTATE | |
| b4a2902f-d828-4d7e-834b-c55269b568d0 | RT3_FPC1 | ERROR | deleting | NOSTATE | |
| d4da449c-ac5e-4076-bf9e-cf7576881e13 | RT3_FPC1 | ERROR | deleting | NOSTATE | |
| 0210d57f-d210-4ca5-ab39-cddb66442a67 | RT4_FPC0 | ERROR | deleting | NOSTATE | |
| 0c79769e-5d3a-485c-976f-0a1faea9f8b1 | RT4_FPC0 | ERROR | deleting | NOSTATE | |
| 4ae310f4-cb44-4632-939b-6fea62ed7918 | RT4_FPC0 | ERROR | deleting | NOSTATE | |
| 7ff31890-aa31-4e5a-9da4-dbb253db198c | RT4_FPC0 | ERROR | deleting | NOSTATE | |
| 0a362468-2c22-46f9-aba3-4ff56703851a | RT4_FPC1 | ERROR | deleting | NOSTATE | |
| 3b197a88-c29b-40a4-89de-eba34eec79e4 | RT4_FPC1 | ERROR | deleting | NOSTATE | |
| 66b59d5d-8cc1-404f-99d4-24cef8412687 | RT4_FPC1 | ERROR | deleting | NOSTATE | |
| cae1e66d-6104-4def-9b8c-e1a9bf985e85 | RT4_FPC1 | ERROR | deleting | NOSTATE | |
| dab1f1e5-a82a-4b18-bd4d-38cd1b952e41 | RT4_FPC1 | ERROR | deleting | NOSTATE | |
| ed214a28-3c25-466a-a8ad-57e05fc20924 | RT4_FPC1 | ERROR | deleting | NOSTATE | |
| 60be3b24-c972-4440-8081-9d73d1a22321 | RT5_fpc0 | ERROR | deleting | NOSTATE | |
| b8a00fe8-09a8-4a07-bee8-e5feb979e1a4 | RT5_fpc0 | ERROR | deleting | NOSTATE | |
| 0da8bc1f-42f2-4f86-b77c-276b16a9037b | lax_fpc0 | ERROR | deleting | Running | r3_fabric_bridge-1523d4aa-cf98-49b8-976e-d7d31ef8f2bf=20.10.1.3; tkekan-net=192.168.1.10, 10.92.14.40; private3-ceecd3d3-69e4-43ee-9470-e3340cb6e7de=20.10.1.3; private2-932e5308-6e04-417e-8048-b183fc97b100=20.10.1.3; r3_re_ctrl_bridge-84964ece-af47-4a6e-8359-78389b21a810=20.10.1.3 |
| 4b1f4f5e-b793-4afe-80a5-3e8a83631f80 | lax_fpc0 | ERROR | deleting | NOSTATE | |
| 9e216805-a428-42dd-88e9-7e78d83c5a27 | lax_fpc0 | ERROR | deleting | NOSTATE | |
| a3c70069-4c67-47f3-9321-3eff38ef7ebc | lax_fpc0 | ERROR | deleting | NOSTATE | |
| a954d7b5-d2a4-42fb-b467-a53f01ff0cb7 | lax_fpc0 | ERROR | deleting | NOSTATE | |
| 04dfccae-4cea-473f-98a4-b43ff2e1504f | nyc_fpc0 | ERROR | deleting | NOSTATE | |
| 90a44bb4-36fd-4388-a54e-14f420eac7a5 | nyc_fpc0 | ERROR | deleting | NOSTATE | |
| 2f92a288-06d2-4219-8ecb-1ec6e50d3fa2 | r1_re0 | ERROR | deleting | NOSTATE | |
| 0f9c7b1c-17a9-4a78-a2d1-7b558e832511 | r1_re1 | ERROR | deleting | Running | tkekan-net=192.168.1.14, 10.92.14.36; r1_re_ctrl_bridge-7ec460dc-219d-48e6-9a35-c72a3995141d=20.10.1.5 |
| e678446f-c810-4bc7-811d-86761c25c086 | r1_re1 | ERROR | deleting | NOSTATE | |
| 0ff33569-f99d-423d-b0af-6d2ff351c38e | r2_re0 | ERROR | deleting | NOSTATE | |
| 226efa25-cc78-4904-b610-7c492791b42c | r2_re0 | ERROR | deleting | NOSTATE | |
| a8af9577-91ae-4e62-9d9a-0921637d4b23 | r2_re0 | ERROR | deleting | NOSTATE | |
| ae41231e-3ca4-43d0-bc60-2243a70c9938 | r2_re0 | ERROR | deleting | NOSTATE | |
| c12db20f-5631-43f2-8828-5e2f81a3718d | r2_re0 | ERROR | deleting | NOSTATE | |
| c5dfcf7c-9b8f-4472-bc6d-56ea7d15d5ff | r2_re0 | ERROR | deleting | NOSTATE | |
| e9e57126-f364-49cb-b720-17546660996e | r2_re0 | ERROR | deleting | NOSTATE | |
| 3aa7f410-ccf3-4ce3-a45b-d7537b0481b5 | r2_re1 | ERROR | deleting | NOSTATE | |
| 65951d1b-7e06-4fb7-a7fa-f7f265789651 | r2_re1 | ERROR | deleting | NOSTATE | |
| 763e556b-7073-4c17-a621-e2a3a157e53d | r2_re1 | ERROR | deleting | NOSTATE | |
| 7bb389df-ceb0-47fb-aae3-5881ccc53c66 | r2_re1 | ERROR | deleting | NOSTATE | |
| 8993e15b-5747-4341-8a62-172a642f4fae | r2_re1 | ERROR | deleting | NOSTATE | |
| bb32be54-93e3-430a-83ba-ad8f157814a2 | r2_re1 | ERROR | deleting | NOSTATE | r2_re_ctrl_bridge-c2bc7168-1fe1-4bdc-af2e-8b9f505262e3=20.10.1.3; tkekan-net=192.168.1.19, 10.92.14.42 |
| fc49ec18-846b-49c8-b3c4-0375a4e6de49 | r2_re1 | ERROR | deleting | NOSTATE | |
| 8c8ea88f-1726-46d0-91d1-2074bc10750c | r3_re0 | ERROR | deleting | NOSTATE | |
| 83d7b14d-2364-43fe-9298-52a38abba099 | r3_re1 | ERROR | deleting | NOSTATE | |
| c41bb1f2-73a7-4959-a89c-fe302dc2b8c3 | r3_re1 | ERROR | deleting | NOSTATE | |
| d9f377b6-4f3e-459d-9053-1cd81295358b | r3_re1 | ERROR | deleting | NOSTATE | |
| f555e466-e3b5-4cf3-ab0f-00e23b85ff12 | r3_re1 | ERROR | deleting | NOSTATE | |
| 047a6f42-e522-4917-a80f-119f47255acc | r4_re0 | ERROR | deleting | NOSTATE | |
| 33a7f4b8-e787-492d-8441-487f9dc84779 | r4_re0 | ERROR | deleting | NOSTATE | |
| 87dba8c3-3000-4799-9c33-0ea6f8bd7b59 | r4_re0 | ERROR | deleting | NOSTATE | |
| d4e99ca3-481a-4533-b671-536efc09cfe7 | r4_re0 | ERROR | deleting | NOSTATE | |
| f7135203-6bf4-4390-a021-66dde3302a27 | r4_re0 | ERROR | deleting | NOSTATE | |
| ffcc7a4c-5a94-4ef6-8398-d53799623989 | r4_re0 | ERROR | deleting | NOSTATE | |
| 62f5208f-5887-459d-9dfa-e3a1eaf1c9d0 | r4_re1 | ERROR | deleting | NOSTATE | |
| 66052ac3-5a2a-4057-8711-6f581f0f0edb | r4_re1 | ERROR | deleting | NOSTATE | |
| 96380fe8-7b3a-4478-bc9c-f32ca49cdf79 | r4_re1 | ERROR | deleting | NOSTATE | |
| adca6927-1cbb-475b-bc7e-a35aaddd2f3f | r4_re1 | ERROR | deleting | NOSTATE | |
| d1dcac1e-1cb4-4620-b763-aa7434da77cb | r4_re1 | ERROR | deleting | NOSTATE | |
| fbb3b801-595d-4089-a94c-5da8501742ed | r4_re1 | ERROR | deleting | NOSTATE | |
| 3449dcf4-9d8a-4c1d-ae40-4acc0f533a3e | r5_re0 | ERROR | deleting | NOSTATE | |
| 96f3460b-c10e-4092-942c-56be714cd2e7 | r5_re0 | ERROR | deleting | NOSTATE | |
| 984d70af-c957-492f-9813-4118de9d185f | r5_re0 | ERROR | deleting | NOSTATE | |
| daae1918-ee92-4fca-abee-8a1c2c9b8a16 | r5_re0 | ERROR | deleting | NOSTATE | |
| 001f34f2-63f0-4ab3-9f55-05299a83d756 | r5_re1 | ERROR | deleting | NOSTATE | |
| 6514dc39-d95a-4a2b-920a-d6d7a603a48c | r5_re1 | ERROR | deleting | NOSTATE | |
| 8466db19-e407-4509-85b8-c2b06fe47193 | r5_re1 | ERROR | deleting | NOSTATE | |
| a4a32cb2-92c8-4bd9-9ae4-aa7c1125794d | r5_re1 | ERROR | deleting | NOSTATE | |
| df197f67-e922-43ee-abb1-f8f9372dc3f3 | r5_re1 | ERROR | deleting | NOSTATE | |
| f56b0c15-2eaf-49f0-869c-f9bf25040a94 | r5_re1 | ERROR | deleting | NOSTATE | |
| 1537af77-224b-42fa-bf2d-2eaad6421921 | sfo_fpc0 | ERROR | deleting | NOSTATE | |
| 27de726c-9cbd-4f43-84d0-84f08013ebb0 | sfo_fpc0 | ERROR | deleting | NOSTATE | |
| d81d2532-c841-416b-bca5-2f3095144cd8 | sfo_fpc0 | ERROR | deleting | NOSTATE | |
| ff95e3f8-d136-46ef-9836-5040b3060e1a | sfo_fpc0 | ERROR | deleting | NOSTATE | |
| 295256d6-f05c-4fc4-b628-51e189010155 | sjc_fpc0 | ERROR | deleting | NOSTATE | |
| 44f2da9f-85df-458b-92ca-b8c3e21dfd71 | sjc_fpc0 | ERROR | deleting | NOSTATE | |
| 134f33b8-913d-41b1-8f05-65067cc5a862 | unix1 | ERROR | deleting | NOSTATE | |
| a8b19205-4430-4525-9c74-08bf6eeb985d | unix1 | ERROR | deleting | NOSTATE | |
| fad8eb97-cd3d-489c-9beb-ac9e028383fe | unix1 | ERROR | deleting | NOSTATE | |
| 557b6929-82a9-4a38-a3b9-a9de4cb72883 | unix2 | ERROR | deleting | NOSTATE | |
| 72533221-185d-4b87-8677-83ac15835b9f | unix2 | ERROR | deleting | NOSTATE | |
| 78bfee4c-258e-4f16-9706-4df0882f45c1 | unix2 | ERROR | deleting | NOSTATE | |
| a07247a5-f8bd-4ced-901b-5b95b33c2cd4 | unix2 | ERROR | deleting | NOSTATE | |
root@jcore-cumulus01:/opt/juniper/cumulus/perf/slau_script/result/129-build#
--------------------
root@jcore-cumulus01:/opt/juniper/cumulus/perf/slau_script/result/129-build# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 2.23-129 129
contrail-config 2.23-129 129
contrail-config-openstack 2.23-129 129
contrail-control 2.23-129 129
contrail-dns 2.23-129 129
contrail-f5 2.23-129 129
contrail-fabric-utils 2.23-129 129
contrail-heat 2.23-129 129
contrail-install-packages 2.23-129~juno 129
contrail-lib 2.23-129 129
contrail-nodemgr 2.23-129 129
contrail-nova-networkapi 2.23-129 129
contrail-openstack 2.23-129 129
contrail-openstack-analytics 2.23-129 129
contrail-openstack-config 2.23-129 129
contrail-openstack-control 2.23-129 129
contrail-openstack-dashboard 2.23-129 129
contrail-openstack-database 2.23-129 129
contrail-openstack-ha 2.23-129 129
contrail-openstack-webui 2.23-129 129
contrail-setup 2.23-129 129
contrail-utils 2.23-129 129
contrail-web-controller 2.23-129 129
contrail-web-core 2.23-129 129
ifmap-python-client 0.1-2 129
ifmap-server 0.3.2-1contrail1 129
neutron-plugin-contrail 2.23-129 129
nova-api 1:2014.2.3-0ubuntu1~cloud0.3contrail129
nova-common 1:2014.2.3-0ubuntu1~cloud0.3contrail129
nova-conductor 1:2014.2.3-0ubuntu1~cloud0.3contrail129
nova-console 1:2014.2.3-0ubuntu1~cloud0.3contrail129
nova-consoleauth 1:2014.2.3-0ubuntu1~cloud0.3contrail129
nova-novncproxy 1:2014.2.3-0ubuntu1~cloud0.3contrail129
nova-objectstore 1:2014.2.3-0ubuntu1~cloud0.3contrail129
nova-scheduler 1:2014.2.3-0ubuntu1~cloud0.3contrail129
python-contrail 2.23-129 129
python-neutronclient 1:2.3.8-0ubuntu1~cloud0.2contrail129
python-nova 1:2014.2.3-0ubuntu1~cloud0.3contrail129
root@jcore-cumulus01:/opt/juniper/cumulus/perf/slau_script/result/129-build#
