master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy failed on checking Ironic steps

Bug #1576881 reported by Artem Hrechanychenko
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Critical
Tatyanka
Mirantis OpenStack
Fix Committed
Critical
Andrey Shestakov

Bug Description

https://ci.fuel-infra.org/job/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/79/console

2016-04-29 21:34:34,540 - ERROR decorators.py:126 -- Traceback (most recent call last):
  File "/home/jenkins/workspace/systest/master/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/systest/master/gates_tests/tests/test_review_in_fuel_agent.py", line 146, in gate_patch_fuel_agent
    ironic_conn.wait_for_vms(ironic_conn)
  File "/home/jenkins/workspace/systest/master/fuelweb_test/helpers/ironic_actions.py", line 100, in wait_for_vms
    timeout=60 * 30, timeout_msg='Server didn\'t became active')
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 100, in wait
    raise TimeoutError(timeout_msg)
TimeoutError: Server didn't became active

Looks like we have some CI environment differences with CI slaves where https://ci.fuel-infra.org/job/mitaka.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/ passed last 5 times

fuel-qa code for this 2 jobs are similar.
https://github.com/openstack/fuel-qa/blob/master/gates_tests/tests/test_review_in_fuel_agent.py
https://github.com/openstack/fuel-qa/blob/stable/mitaka/gates_tests/tests/test_review_in_fuel_agent.py

On custom CI - https://custom-ci.infra.mirantis.net/view/9.0/job/9.0.custom.packages_test.ubuntu/47/console passed too.

Need to set 'non-voiting' mode for master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy until root cause of issues will be finded

description: updated
Revision history for this message
Dmitry Kaigarodеsev (dkaiharodsev) wrote :

please provide more info regarding 'environment differences' - what should be checked and where could be a problem

Changed in fuel:
status: New → Incomplete
assignee: Fuel CI (fuel-ci) → Artem Grechanichenko (agrechanichenko)
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

Folks, on the latest master job fails I see next error in ironic conductor http://paste.openstack.org/show/496054/, could you please confirm that slaves where test are executed contain fix for https://bugs.launchpad.net/fuel/+bug/1576243

Revision history for this message
Dmitry Kaigarodеsev (dkaiharodsev) wrote :
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

Looks like run puppet agent with latests manifest fix the libvirt problem, so there are not errors related to libvirt. But we fail on next step (created server do not became active in defined timeout). This should be investigated

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

so actually instance is in error state:
root@node-1:~# nova list
+--------------------------------------+-----------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------------+--------+------------+-------------+----------+
| a3eb1b19-b395-46ce-961d-54c4c8ae1ad1 | ironic-slave-01 | ERROR | - | NOSTATE | |

So at least we should make improvement in tests waiter like if instance in error state - stop to wait for Active and fail

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Download full text (5.1 KiB)

so the insatnce failed on attempts to binging port in neutron :
6 14:54:25 GMT Connection: close Content-Length: 0 X-Openstack-Request-Id: req-350e0a02-1465-4b2d-91f9-eb8a5bb71328 _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:277
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager [req-a74b8c35-1db9-410d-bb82-3e6b518200e3 b237c20cf2ab4b5da151bec429acd229 8ef5461db02845a7a01f69a30abbd0e5 - - -] Instance failed network setup after 1 attempt(s)
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager Traceback (most recent call last):
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1570, in _allocate_network_async
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager bind_host_id=bind_host_id)
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 666, in allocate_for_instance
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager self._delete_ports(neutron, instance, created_port_ids)
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager self.force_reraise()
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 655, in allocate_for_instance
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager security_group_ids, available_macs, dhcp_opts)
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 259, in _create_port
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager raise exception.PortBindingFailed(port_id=port_id)
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager PortBindingFailed: Binding failed for port 7c039fd7-f06f-481f-a4a1-812efda89376, please check neutron logs for more information.
2016-05-04 14:54:25.249 26452 ERROR nova.compute.manager
2016-05-04 14:54:25.257 26452 ERROR nova.virt.ironic.driver [req-a74b8c35-1db9-410d-bb82-3e6b518200e3 b237c20cf2ab4b5da151bec429acd229 8ef5461db02845a7a01f69a30abbd0e5 - - -] Error preparing deploy for instance a3eb1b19-b395-46ce-961d-54c4c8ae1ad1 on baremetal node 495ffe14-3550-4abb-babf-db2803cf8dd6.

in neutron :
<164>May 4 14:54:24 node-1 neutron-server: 2016-05-04 14:54:24.636 14117 WARNING neutron.plugins.ml2.drivers.mech_agent [req-0f415803-9fb9-46ef-8572-4f003555600e ed1692e212d74842a159a5a27269c1ba 60e91c44f1cb4222b78dd1fd56fb52f4 - - -] Port 7c039fd7-f06f-481f-a4a1-812efda89376 on network 4dba4fa6-545c-4b42-a630-3a31f387194a not bound, no agent registered on host ironic-compute
<163>May 4 14:54:24 node-1 neutron-server: 2016-05-04 14:54:24.637 14117 ERROR neutron.plugins.ml2.managers [req-0f415803-9fb9...

Read more...

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

Looks like related upstream issue - https://bugs.launchpad.net/neutron/+bug/1492254

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

so passing the ball to ironic team

Changed in fuel:
assignee: Artem Grechanichenko (agrechanichenko) → MOS Ironic (mos-ironic)
Changed in fuel:
milestone: 9.0 → 10.0
Revision history for this message
Andrey Shestakov (ashestakov) wrote :

This issue occurred because bug https://bugs.launchpad.net/mos/+bug/1492272 is not fixed in downstream branch for 10.0.
Should works in 9.0.

Revision history for this message
Pavlo Shchelokovskyy (pshchelo) wrote :

Until there are no downstream 10.0 branches, we can not cherry-pick this change to Nova:

https://review.fuel-infra.org/#/c/18323/

So I would suggest to move the failing job to non-voting until branches have been cut.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/314936

Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

Job master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy is for release 10.0
I think this LP issue should not affect 9.0 because for 9.0 we have another job: mitaka.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy.

Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

I remove it from 10.0 scope because 10.0 has similar but another job and it is affected by another bug.

no longer affects: fuel
Changed in fuel:
milestone: none → 9.0
assignee: nobody → Fuel for Openstack (fuel)
status: New → Incomplete
importance: Undecided → Critical
milestone: 9.0 → 10.0
Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

Sorry, i messed it with bug https://bugs.launchpad.net/fuel/+bug/1580204 which is for 9.0.

So current bug is dedicated for release 10.0.

no longer affects: fuel/mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/314936
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=4fa7f92aee53a860955bd7ebadb19838e1d9efb3
Submitter: Jenkins
Branch: master

commit 4fa7f92aee53a860955bd7ebadb19838e1d9efb3
Author: Tatyana Leontovich <email address hidden>
Date: Wed May 11 12:43:13 2016 +0300

    Disable step14-15 in fuel-agent gate tests

    We should to disable vm creation on ironic node
    until LP1576881 become fixed.
    According to It happens only after stable branches
    creation for 10.0, we disable this steps and
    turn on test into voating mode to get feedback for the fuel-agent patches.

    Change-Id: I1b2c8666f605f4d3ec29138e021dd419c87d3c78
    Related-Bug: #1576881

Changed in fuel:
status: Incomplete → Fix Committed
Changed in mos:
importance: Undecided → Critical
milestone: none → 10.0
assignee: nobody → MOS Ironic (mos-ironic)
status: New → Confirmed
Changed in fuel:
assignee: Fuel for Openstack (fuel) → Tatyanka (tatyana-leontovich)
Revision history for this message
Serge Kovaleff (serge-kovaleff) wrote :

Andrey, please merge the fix to MOS 10 branch when it's available.

Changed in mos:
assignee: MOS Ironic (mos-ironic) → Andrey Shestakov (ashestakov)
Revision history for this message
Serge Kovaleff (serge-kovaleff) wrote :

There is no branch for 10.
9.x branch should have the fix from 9.0

Changed in mos:
status: Confirmed → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.