OSTF tests start before deployment finished

[Iryna Vovk]

Steps to reproduce:
1. Prepare env with iso 301 7.0 and Neutron with tunneling segmentation
2. Create cluster
3. Add 1 node with “contrail-db”, 1 node "contarail-config" and 1 node "contrail-control",
 1 node with controller and 1 node with compute roles
4. Enable Contrail plugin v3.0.0
5. Deploy cluster with plugin
6. After the end of deploy run OSTF tests

Actual result:

Such OSTF tests scenario fail:

Create volume and boot instance from it:
Instance creation failed.

Create volume and attach it to instance:
Instance creation failed.

Launch instance
Creating instance using the new security group has failed.

Launch instance, create snapshot, launch instance from snapshot
Instance can not be launched from snapshot.

Addapi: '1.0'
astute_sha: 6c5b73f93e24cc781c809db9159927655ced5012
auth_required: true
build_id: '301'
build_number: '301'
feature_groups:
- mirantis
fuel-agent_sha: 50e90af6e3d560e9085ff71d2950cfbcca91af67
fuel-library_sha: 5d50055aeca1dd0dc53b43825dc4c8f7780be9dd
fuel-nailgun-agent_sha: d7027952870a35db8dc52f185bb1158cdd3d1ebd
fuel-ostf_sha: 2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c
fuelmain_sha: a65d453215edb0284a2e4761be7a156bb5627677
nailgun_sha: 4162b0c15adb425b37608c787944d1983f543aa8
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 486bde57cda1badb68f915f66c61b544108606f3
release: '7.0'
release_versions:
  2015.1.0-7.0:
    VERSION:
      api: '1.0'
      astute_sha: 6c5b73f93e24cc781c809db9159927655ced5012
      build_id: '301'
      build_number: '301'
      feature_groups:
      - mirantis
      fuel-agent_sha: 50e90af6e3d560e9085ff71d2950cfbcca91af67
      fuel-library_sha: 5d50055aeca1dd0dc53b43825dc4c8f7780be9dd
      fuel-nailgun-agent_sha: d7027952870a35db8dc52f185bb1158cdd3d1ebd
      fuel-ostf_sha: 2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c
      fuelmain_sha: a65d453215edb0284a2e4761be7a156bb5627677
      nailgun_sha: 4162b0c15adb425b37608c787944d1983f543aa8
      openstack_version: 2015.1.0-7.0
      production: docker
      python-fuelclient_sha: 486bde57cda1badb68f915f66c61b544108606f3
      release: '7.0'itional info:

[Iryna Vovk (ivovk)]

This issue is still reproducing with the latest plugin package. Such OSTF tests scenario fail:

Check network connectivity from instance via floating IP
Server can not be created(Step: Create an instance using the new security group)

Launch instance
Creating instance using the new security group has failed.(Step: 2. Create an instance using the new security group.)

Launch instance with file injection
Creating instance using the new security group has failed.(Step: 2. Create an instance with injected file.)

[Iryna Vovk (ivovk)]

This issue is still present on CI:
https://jenkins-tpi.bud.mirantis.net/view/7.0.contrail.3.0/job/7.0.contrail.3.0.contrail_bvt.juniper/15/console
And there is problem with one additional OSTF case:
"Check that required services are running (failure)"

[Iryna Vovk (ivovk)]

Also this problem occured on lab:
scenario:
            1. Create an environment with "Neutron with tunneling segmentation"
               as a network configuration and CEPH storage
            2. Enable and configure Contrail plugin
            3. Add 3 nodes with controller role
            4. Add 2 nodes with "compute" and "Ceph-OSD" roles
            5. Add a node with MongoDB role
            6. Add a node with "contrail-config", "contrail-control" and "contrail-db" roles
            7. Deploy cluster with plugin
            8. Run OSTF tests

Actual result:

AssertionError: WARNING! Unexpected fail: expected ['Check network connectivity from instance via floating IP', 'Launch instance with file injection'], actual [u'Check create, update and delete image actions using Glance v1', u'Check create, update and delete image actions using Glance v2', u'Check network connectivity from instance via floating IP', u'Launch instance', u'Launch instance, create snapshot, launch instance from snapshot', u'Typical stack actions: create, delete, show details, etc.', u'Advanced stack actions: suspend, resume and check', u'Check stack rollback', u'Update stack actions: inplace, replace and update whole template']

[Iryna Vovk (ivovk)]

This issue is still present on my lab with ha scenario:
AssertionError: Failed 10 OSTF tests; should fail 1 tests. Names of failed tests: [
 {
  "Check create, update and delete image actions using Glance v1 (failure)": "Error creating image. Please refer to Openstack logs for more information. Please refer to OpenStack logs for more details."
 },
 {
  "Check create, update and delete image actions using Glance v2 (failure)": "Error creating image. Please refer to Openstack logs for more information. Please refer to OpenStack logs for more details."
 },
 {
  "Check network connectivity from instance via floating IP (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 },
 {
  "Create security group (failure)": "Failed to create rule in security group. Please refer to OpenStack logs for more details."
 },
 {
  "Launch instance (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 },
 {
  "Launch instance, create snapshot, launch instance from snapshot (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 },
 {
  "Typical stack actions: create, delete, show details, etc. (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 },
 {
  "Advanced stack actions: suspend, resume and check (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 },
 {
  "Check stack rollback (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 },
 {
  "Update stack actions: inplace, replace and update whole template (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 }
]

[Przemysław Szypowicz (pszypowicz)]

But still it looks like it lacks TestVM.

Could you do this:

"TestVM image not found. Please, download
http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image
and register it in Glance with name 'TestVM' as 'admin' tenant."

in open stack horizon? Then rerun OSTF

On Thu, Dec 24, 2015 at 11:04 AM, Iryna Vovk <email address hidden> wrote:

> This issue is still present on my lab with ha scenario:
> AssertionError: Failed 10 OSTF tests; should fail 1 tests. Names of failed
> tests: [
> {
> "Check create, update and delete image actions using Glance v1
> (failure)": "Error creating image. Please refer to Openstack logs for more
> information. Please refer to OpenStack logs for more details."
> },
> {
> "Check create, update and delete image actions using Glance v2
> (failure)": "Error creating image. Please refer to Openstack logs for more
> information. Please refer to OpenStack logs for more details."
> },
> {
> "Check network connectivity from instance via floating IP (failure)":
> "TestVM image not found. Please, download
> http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image
> and register it in Glance with name 'TestVM' as 'admin' tenant."
> },
> {
> "Create security group (failure)": "Failed to create rule in security
> group. Please refer to OpenStack logs for more details."
> },
> {
> "Launch instance (failure)": "TestVM image not found. Please, download
> http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image
> and register it in Glance with name 'TestVM' as 'admin' tenant."
> },
> {
> "Launch instance, create snapshot, launch instance from snapshot
> (failure)": "TestVM image not found. Please, download
> http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image
> and register it in Glance with name 'TestVM' as 'admin' tenant."
> },
> {
> "Typical stack actions: create, delete, show details, etc. (failure)":
> "TestVM image not found. Please, download
> http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image
> and register it in Glance with name 'TestVM' as 'admin' tenant."
> },
> {
> "Advanced stack actions: suspend, resume and check (failure)": "TestVM
> image not found. Please, download
> http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image
> and register it in Glance with name 'TestVM' as 'admin' tenant."
> },
> {
> "Check stack rollback (failure)": "TestVM image not found. Please,
> download
> http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image
> and register it in Glance with name 'TestVM' as 'admin' tenant."
> },
> {
> "Update stack actions: inplace, replace and update whole template
> (failure)": "TestVM image not found. Please, download
> http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image
> and register it in Glance with name 'TestVM' as 'admin' tenant."
> }
> ]
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1518348
>
> Title:
> [contrail] Some OSTF test scenario fail if Contrail roles locates on
> separated nodes.
>
> Status in Fuel Plugins:
> In Progress
>
> Bug ...

[Iryna Vovk (ivovk)]

Looks like the same issue occurred during such test scenario:
            1. Create an environment with "Neutron with tunneling segmentation" as a network configuration
            2. Enable and configure Contrail plugin
            3. Add 3 nodes with "controller" + "storage-cinder" multirole
            4. Add 1 node with "compute" role
            5. Add 2 nodes with "contrail-config", "contrail-control" and "contrail-db" roles
            6. Deploy cluster with plugin
            8. Run OSTF tests

Actual result:
AssertionError: Failed 6 OSTF tests; should fail 0 tests. Names of failed tests: [
 {
  "Create volume and boot instance from it (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 },
 {
  "Create volume and attach it to instance (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 },
 {
  "Check network connectivity from instance via floating IP (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 },
 {
  "Launch instance (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 },
 {
  "Launch instance with file injection (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 },
 {
  "Launch instance, create snapshot, launch instance from snapshot (failure)": "TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant."
 }
]

[Iryna Vovk (ivovk)]

Workaround:
If additional time is added(approximately 20 min) - ostf will passed.
While post deploy tasks are still in progress, ostf has already started.

[Vitalii Kovalchuk (vkovalchuk)]

I repeat Irina test scenario but added at the last executable manifest (service_token_off.pp) fail string "fail(' fail ')"

1. Create an environment with "Neutron with tunneling segmentation" as a network configuration
2. Enable and configure Contrail plugin
3. Add 3 nodes with "controller" + "storage-cinder" multirole
4. Add 1 node with "compute" role
5. Add 2 nodes with "contrail-config", "contrail-control" and "contrail-db" roles
6. Deploy cluster with plugin
8. Run OSTF tests

after deploy crash tests were still running

2016-02-12 11:28:01,751 - INFO fuel_web_client.py:301 -- Assert task {u'status': u'running', u'name': u'deploy', u'cluster': 1, u'result': {}, u'progress': 0, u'message': None, u'id': 2, u'uuid': u'2dbaa469-71c1-4e27-af09-c5c582f9d6ae'} is success
2016-02-12 11:28:01,751 - INFO fuel_web_client.py:1177 -- Wait for task 7800 seconds: {
 "status": "running",
 "name": "deploy",
 "cluster": 1,
 "result": {},
 "progress": 0,
 "message": null,
 "id": 2,
 "uuid": "2dbaa469-71c1-4e27-af09-c5c582f9d6ae"
}
2016-02-12 13:22:41,661 - INFO fuel_web_client.py:1194 -- Task finished. Took 6879 seconds. {
 "status": "error",
 "name": "deploy",
 "cluster": 1,
 "result": {},
 "progress": 100,
 "message": "Deployment has failed. Method granular_deploy. Failed to execute hook 'puppet' Puppet run failed. Check puppet logs for details\n\n---\npriority: 2100\nfail_on_error: true\ntype: puppet\nuids:\n- '1'\n- '2'\n- '4'\nparameters:\n puppet_modules: \"/etc/puppet/modules\"\n puppet_manifest: \"/etc/puppet/modules/osnailyfacter/modular/astute/service_token_off.pp\"\n timeout: 180\n cwd: \"/\"\n.\nInspect Astute logs for the details",
 "id": 2,
 "uuid": "2dbaa469-71c1-4e27-af09-c5c582f9d6ae"
}
2016-02-12 13:22:41,661 - INFO regenerate_repo.py:278 -- Check puppet logs for packages with unmet dependences.
2016-02-12 13:22:41,725 - INFO regenerate_repo.py:289 -- Check puppet logs completed.
2016-02-12 13:22:41,995 - INFO fuel_web_client.py:948 -- Get ID of a last created cluster
2016-02-12 13:30:03,552 - INFO openstack.py:62 -- ip is 10.109.5.4
2016-02-12 13:30:11,068 - INFO openstack.py:62 -- ip is 10.109.5.8
2016-02-12 13:30:31,877 - INFO openstack.py:62 -- ip is 10.109.5.6
2016-02-12 13:30:39,078 - INFO openstack.py:62 -- ip is 10.109.5.7
2016-02-12 13:31:00,139 - INFO openstack.py:62 -- ip is 10.109.5.5
2016-02-12 13:31:21,101 - INFO openstack.py:62 -- ip is 10.109.5.3
2016-02-12 13:31:21,102 - INFO nailgun_client.py:253 -- Run OSTF tests at cluster #1: ['smoke', 'sanity']
2016-02-12 13:31:22,019 - INFO fuel_web_client.py:1137 -- Try to run assert ostf with expected fail name None
2016-02-12 13:31:22,019 - INFO fuel_web_client.py:213 -- Assert OSTF run at cluster #1. Should fail 0 tests named None
2016-02-12 13:31:22,020 - INFO fuel_web_client.py:106 -- Wait OSTF tests at cluster #1 for 1800 seconds

[Alexandra Morozova (astepanchuk)]

isn't this bug related to this one? https://bugs.launchpad.net/fuel/+bug/1538577

[Bug Checker Bot (esikachev-l)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Tatyanka (tatyana-leontovich)]

Sergey, sorry but problem is not in ostf. There are several variants - 1. Guys has wrong test scenario - so do not check if all task finishes before ostf test invocation or some issues in task graph with enabled plugin. So remove tag area-ostf here

[okosse (okosse)]

Hi, everyone.
This bug hasn't reproduced already (we are using Fuel 8.0 and fuel plug-in contrail 4.0.0).

[okosse (okosse)]

I also cannot reproduce this bug on Fuel 7.0 with current MU