Fuel for OpenStack

Package testing job fails

Bug #1423500 reported by Victor Ryzhenkin on 2015-02-19

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Released	Critical	Alexander Tsamutali	Fuel for OpenStack 6.1
	6.0.x	Won't Fix	Undecided	Unassigned	Fuel for OpenStack 6.0-updates

Bug Description

Link to failed job build: http://osci-jenkins.srt.mirantis.net:8080/job/package_testing_deb/2553/

For Murano platform tests required at least 2048 Mb of free RAM at compute node, but we have less RAM that we needs.

Tracebacks:

======================================================================
FAIL: Deploy cluster in HA mode with Murano and Neutron GRE
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/package_testing_deb/fuel-main/fuelweb_test/helpers/decorators.py", line 64, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/package_testing_deb/fuel-main/fuelweb_test/tests/test_services.py", line 356, in deploy_murano_ha_one_controller_gre
    test_name=test_name, timeout=60 * 36)
  File "/home/jenkins/workspace/package_testing_deb/fuel-main/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/package_testing_deb/fuel-main/fuelweb_test/models/fuel_web_client.py", line 621, in run_single_ostf_test
    timeout=timeout)
  File "/home/jenkins/workspace/package_testing_deb/fuel-main/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/package_testing_deb/fuel-main/fuelweb_test/models/fuel_web_client.py", line 148, in assert_ostf_run_certain
    'must have passed: %s' % fail_details)
AssertionError: The following tests have not succeeded, while they must have passed: [u'Check that user can deploy Apache service in Murano environment (fuel_health.tests.platform_tests.test_murano_linux.MuranoDeployLinuxServicesTests.test_deploy_apache_service). Test status: skipped, message: This test requires more hardware resources of your OpenStack cluster: at least one of the compute nodes must have >= 2048 MB of free RAM, but you have only 1993 MB on most appropriate compute node.']

======================================================================
FAIL: Deploy cluster in ha mode with Murano and Neutron GRE
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/package_testing_deb/fuel-main/fuelweb_test/helpers/decorators.py", line 64, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/package_testing_deb/fuel-main/fuelweb_test/tests/test_services.py", line 478, in deploy_murano_ha_with_gre
    test_name=test_name, timeout=60 * 36)
  File "/home/jenkins/workspace/package_testing_deb/fuel-main/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/package_testing_deb/fuel-main/fuelweb_test/models/fuel_web_client.py", line 621, in run_single_ostf_test
    timeout=timeout)
  File "/home/jenkins/workspace/package_testing_deb/fuel-main/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/package_testing_deb/fuel-main/fuelweb_test/models/fuel_web_client.py", line 148, in assert_ostf_run_certain
    'must have passed: %s' % fail_details)
AssertionError: The following tests have not succeeded, while they must have passed: [u'Check that user can deploy Apache service in Murano environment (fuel_health.tests.platform_tests.test_murano_linux.MuranoDeployLinuxServicesTests.test_deploy_apache_service). Test status: skipped, message: This test requires more hardware resources of your OpenStack cluster: at least one of the compute nodes must have >= 2048 MB of free RAM, but you have only 0 MB on most appropriate compute node.']

Tags:

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2015-02-19:

We need to use only correct slave nodes for MOS system tests

Changed in fuel:
importance:	Undecided → Critical
assignee:	nobody → Fuel DevOps (fuel-devops)
milestone:	none → 6.1.1
status:	New → Confirmed
milestone:	6.1.1 → 6.1
tags:	added: devops

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2015-02-19:

Looks like we need to use VMs with more RAM size to avoid such failed tests

Sergey Kulanov (skulanov) on 2015-02-19

Changed in fuel:
assignee:	Fuel DevOps (fuel-devops) → Fuel OSCI Team (fuel-osci)

Roman Vyalov (r0mikiam) on 2015-02-19

Changed in fuel:
assignee:	Fuel OSCI Team (fuel-osci) → Alexander Tsamutali (astsmtl)

Revision history for this message

Alexander Tsamutali (astsmtl) wrote on 2015-02-19:

Actually you can put any environment veriable in integration_tests.conf. So in this case you can add:

SLAVE_NODE_MEMORY=3072

Alexander Tsamutali (astsmtl) on 2015-02-19

Changed in fuel:
status:	Confirmed → In Progress

Alexander Tsamutali (astsmtl) on 2015-02-19

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

Alexander Tsamutali (astsmtl) wrote on 2015-02-19:

To be precise, you need to export this varible, like this:

export SLAVE_NODE_MEMORY=3072

Revision history for this message

OSCI Robot (oscirobot) wrote on 2015-02-19:

RPM package murano has been built for project openstack/murano
Package version == 2015.1.0b2, package release == fuel6.1.mira7.git.52b43e6.7e134ca

Changeset: https://review.fuel-infra.org/3743
project: openstack-build/murano-build
branch: master
author: Viktor Ryzhenkin
committer: Viktor Ryzhenkin
subject: Add more RAM to slave nodes
status: patchset-created

Files placed on repository:
openstack-murano-api-2015.1.0b2-fuel6.1.mira7.git.52b43e6.7e134ca.noarch.rpm
openstack-murano-common-2015.1.0b2-fuel6.1.mira7.git.52b43e6.7e134ca.noarch.rpm
openstack-murano-engine-2015.1.0b2-fuel6.1.mira7.git.52b43e6.7e134ca.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-3743/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2015-02-19:

DEB package murano has been built for project openstack/murano
Package version == 2015.1.0b2, package release == fuel6.1~mira7+git.52b43e6.7e134ca

Files placed on repository:
murano-api_2015.1.0b2-fuel6.1~mira7+git.52b43e6.7e134ca_all.deb
murano-common_2015.1.0b2-fuel6.1~mira7+git.52b43e6.7e134ca_all.deb
murano-engine_2015.1.0b2-fuel6.1~mira7+git.52b43e6.7e134ca_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-master-3743/ubuntu

Revision history for this message

OSCI Robot (oscirobot) wrote on 2015-02-19:

RPM package murano has been built for project openstack/murano
Package version == 2015.1.0b2, package release == fuel6.1.mira7.git.52b43e6.7e134ca

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-3743/centos

Revision history for this message

OSCI Robot (oscirobot) wrote on 2015-02-19:

DEB package murano has been built for project openstack/murano
Package version == 2015.1.0b2, package release == fuel6.1~mira7+git.52b43e6.7e134ca

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-master-3743/ubuntu

Revision history for this message

OSCI Robot (oscirobot) wrote on 2015-02-19:

RPM package murano has been built for project openstack/murano
Package version == 2015.1.0b2, package release == fuel6.1.mira7.git.6720b51.7e134ca

Files placed on repository:
openstack-murano-api-2015.1.0b2-fuel6.1.mira7.git.6720b51.7e134ca.noarch.rpm
openstack-murano-common-2015.1.0b2-fuel6.1.mira7.git.6720b51.7e134ca.noarch.rpm
openstack-murano-engine-2015.1.0b2-fuel6.1.mira7.git.6720b51.7e134ca.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-master-3743/centos

Revision history for this message

Alexander Tsamutali (astsmtl) wrote on 2015-02-20:

#10

After increasing SLAVE_NODE_MEMORY to 6144 testing still fails with timeout. http://osci-jenkins.srt.mirantis.net:8080/job/package_testing_rpm/2957/console

Alexander Tsamutali (astsmtl) on 2015-02-20

Changed in fuel:
assignee:	Alexander Tsamutali (astsmtl) → Fuel QA Team (fuel-qa)

Nastya Urlapova (aurlapova) on 2015-02-25

Changed in fuel:
assignee:	Fuel QA Team (fuel-qa) → Timur Nurlygayanov (tnurlygayanov)
status:	Fix Committed → Confirmed

Timur Nurlygayanov (tnurlygayanov) on 2015-02-25

Changed in fuel:
assignee:	Timur Nurlygayanov (tnurlygayanov) → Viktor Ryzhenkin (vryzhenkin)

Igor Shishkin (teran) on 2015-03-06

tags:

added: osci
removed: devops

Victor Ryzhenkin (vryzhenkin) on 2015-03-17

Changed in fuel:
status:	Confirmed → In Progress
tags:	added: murano

Revision history for this message

Victor Ryzhenkin (vryzhenkin) wrote on 2015-03-17:

#11

Now, we are waiting for Murano fixes in Fuel isos. It is nessesary to verify, that job running correctly.

Revision history for this message

Ekaterina Chernova (efedorova) wrote on 2015-03-25:

#12

RAM on compute node is enough and it's not the root cause, need to investigate more on that.

summary:

- Package testing job fails because of insufficiently RAM on compute node
+ Package testing job fails

Ekaterina Chernova (efedorova) on 2015-03-25

Changed in fuel:
assignee:	Victor Ryzhenkin (vryzhenkin) → nobody

Serg Melikyan (smelikyan) on 2015-03-25

Changed in fuel:
assignee:	nobody → Fuel OSCI Team (fuel-osci)

Roman Vyalov (r0mikiam) on 2015-03-25

Changed in fuel:
assignee:	Fuel OSCI Team (fuel-osci) → Alexander Tsamutali (astsmtl)

Revision history for this message

Serg Melikyan (smelikyan) wrote on 2015-03-25:

#13

Summary:

1. First failure of this job was detected as issue with insufficient amout RAM on compute node.
    OSCI Team increased RAM up to 6GB on slave nodes.
    Result: Issue was not fixed, we still got timeout on these jobs
2. Snapshot was reverted, new image with murano-agent and latest updates from canonical was uploaded, OSTF tests passed on new snapshot. We commited changes to fuel-qa repo with new image md5 sum and image name, and contacted with Fuel DevOps Team and they replaced old image to new.
    Result: issue was not fixed

3. OSCI team contacted with us and report about problem with dev-ops scripts. After a while, OSCI team reported, that issue was fixed, and job was failed with new error. New error was caused by unstable Fuel ISOs and in this isos Murano couldn't be installed.
Result: Issue in ISO was fixed, jobs are still failing, we still get timeout on these jobs.

Revision history for this message

Artem Panchenko (apanchenko-8) wrote on 2015-03-26:

#14

Guys,

I examined logs and VMs snapshots of failed tests and discovered following things:

1) the issue #1 with lack of RAM on slaves wasn't resolved. You added export of variable SLAVE_NODE_MEMORY before starting tests, but tests are still using old environments (CI job has KEEP_BEFORE=yes parameter). So the first thing you need to do is run 'dos.py erase ${ENV_NAME}' on all slave nodes with 'package-testing' label or unset KEEP_BEFORE variable;

2) system tests fail with TimeoutError while waiting for Nova instance becomes 'ACTIVE', because scheduler can't boot VM with 2GB RAM (compute node still has only 1.5 GB RAM free), so it is also caused by issue #1;

3) system tests on Ubuntu are unable to create snapshot (ready_with_5_slaves) due to internal QEMU error (unfortunately we are still fighting with it and haven't a good workaround yet) and it will be also resolved after environment re-creation (help in 100% cases)

Revision history for this message

Ekaterina Chernova (efedorova) wrote on 2015-03-26:

#15

Another patch that blocked by this issue https://review.fuel-infra.org/#/c/4965/

Revision history for this message

Dmitry Borodaenko (angdraug) wrote on 2015-03-30:

#16

It's been 2 business days since last update on this Critical bug. Alexander, are you currently working on this? Please provide current status.

Revision history for this message

Alexander Tsamutali (astsmtl) wrote on 2015-03-30:

#17

Victor is trying to fine the exact reason that causes test failure. Today we tried to run tests on ISO that was known to work before with no success: http://osci-jenkins.srt.mirantis.net:8080/job/6.1.mos.deploy-deb-test/5/console . Also we tried to retrigger one test on more recent ISO, it's currently running: http://osci-jenkins.srt.mirantis.net:8080/job/6.1.mos.deploy-deb/477/console . Both servers are disconnected from Jenkins now and will be available to Viktor and Serg (I hope he will help us tommorow) to debug this issue. On a positive side, we found that sometimes tests work as expected: http://osci-jenkins.srt.mirantis.net:8080/job/6.1.mos.deploy-rpm/349/

Revision history for this message

Victor Ryzhenkin (vryzhenkin) wrote on 2015-03-30:

#18

Hi Dmitry! Today we merged into fuel-ostf new test for Murano, which don't require Murano-image, internet connection and imported applications. Also we merged patch to fuel-qa, which replaces old test execution in system tests.
Now, we need to wait a little bit, when new Fuel ISO was builded and passed BVT tests.

Revision history for this message

Alexander Tsamutali (astsmtl) wrote on 2015-04-01:

#19

Finally we have +2: https://review.fuel-infra.org/#/c/4877/

Revision history for this message

Victor Ryzhenkin (vryzhenkin) wrote on 2015-04-01:

#20

All jobs works successfully.
Thanks OSCI team.
Now, move bug to Fix Released.

Changed in fuel:
status:	In Progress → Fix Released

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-07-16: Change abandoned on openstack-build/murano-build (openstack-ci/fuel-6.1/2014.2)

#21

Change abandoned by Viktor Ryzhenkin <email address hidden> on branch: openstack-ci/fuel-6.1/2014.2
Review: https://review.fuel-infra.org/3764

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2015-07-16: Change abandoned on openstack-build/murano-build (master)

#22

Change abandoned by Viktor Ryzhenkin <email address hidden> on branch: master
Review: https://review.fuel-infra.org/3743

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.