[9.2] [Tempest] Functional OSTF fail to start instances

Bug #1646833 reported by Yury Tregubov on 2016-12-02
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Critical
Nikita Karpin
Nominated for Ocata by Oleksiy Molchanov
Mitaka
Critical
Nikita Karpin
Newton
Critical
Nikita Karpin

Bug Description

Since 9.2 snapshot #578 the following problem is seen right after deploy of tempest envs:

AssertionError: Failed 8 OSTF tests; should fail 0 tests. Names of failed tests:
  - Create volume and boot instance from it (failure) Instance creation failed. Please refer to OpenStack logs for more details.
  - Create volume and attach it to instance (failure) Instance creation failed. Please refer to OpenStack logs for more details.
  - Check network connectivity from instance via floating IP (failure) Failed to create rule in security group. Please refer to OpenStack logs for more details.
  - Create security group (failure) Failed to create rule in security group. Please refer to OpenStack logs for more details.
  - Check network parameters (error)
  - Launch instance (failure) Failed to create rule in security group. Please refer to OpenStack logs for more details.
  - Launch instance with file injection (failure) Failed to create rule in security group. Please refer to OpenStack logs for more details.
  - Launch instance, create snapshot, launch instance from snapshot (failure) Default private network 'admin_internal_net' isn't present. Please verify it is properly created. Please refer to OpenStack logs for more details.

That blocks Tempest execution.

Diagnostic snapshot is: https://drive.google.com/a/mirantis.com/file/d/0B1Crk-sAvGanUUJPMFFTQXFDc2s/view?usp=sharing

Logs from deploy are available here: http://172.18.173.8:8080/jenkins/view/Tempest_9.%D0%A5/job/9.x_Tempest_Detached_RabbitMQ/190/console

Changed in fuel:
importance: Undecided → Critical
milestone: none → 9.2
description: updated
tags: added: blocker-for-qa
summary: - [9.2] [Tempest] Functional OSTF are failed to start instances
+ [9.2] [Tempest] Functional OSTF fail to start instances
Changed in fuel:
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
status: New → Confirmed
tags: added: area-library
Dmitry Pyzhov (dpyzhov) on 2016-12-06
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Stanislaw Bogatkin (sbogatkin)
Oleksiy Molchanov (omolchanov) wrote :

QA team, it is sending request that contains 'null' value for security group id

2016-12-05 23:55:10.327 28485 DEBUG nova.api.openstack.wsgi [req-1faf753d-6e34-4574-a777-729f2cb5391d 197788e0f8dd49e99259e971474c1b6d e06694adff6a4c7a987996e8e1cc0aea - - -] Action: 'create', calling method: <bound method SecurityGroupRulesController.create of <nova.api.openstack.compute.security_groups.SecurityGroupRulesController object at 0x7f14fcd88810>>, body: {"security_group_rule": {"from_port": -1, "ip_protocol": "icmp", "to_port": -1, "parent_group_id": 6, "cidr": "0.0.0.0/0", "group_id": null}} _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:696

Changed in fuel:
assignee: Stanislaw Bogatkin (sbogatkin) → Fuel QA Team (fuel-qa)
Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Vladimir Khlyunev (vkhlyunev)
Vladimir Khlyunev (vkhlyunev) wrote :

OSTF if working fine using received from novaclient parameters; however there is really strange behavior in nova:

https://paste.mirantis.net/show/2843/

NovaClient creates secgroup with "int" id and then fails at validation step; BUT after several ostf debug restarts I got reversed situation - secgroup id was "uuid" and nova traces with "Security group id should be integer"

Really weird that I got following situation once and it was not reproduced:
While debugging I tries "nova secgroup-list" on controller and it shows 4 created by ostf testgroup BUT they does not matches with secgroup showed in horizon. Then I created secgroup using dashboard and it WAS NOT presented in "nova secgroup-list". Then I created secgroup using "nova secgroup-create" -- weird place -- output of the command was completely changed and show all new groups (no one old was present in new output) and here id became uuid type. It looks like we have 2 databases with secgroups.

Changed in fuel:
assignee: Vladimir Khlyunev (vkhlyunev) → MOS Nova (mos-nova)
Roman Podoliaka (rpodolyaka) wrote :

The inconsistent behaviour Vladimir is seeing is caused by the fact that part of nova services are configured to use nova-network, while the other part is configured (correctly) to use neutron:

http://paste.openstack.org/show/591960/

This explains the issues with missing admin_internal_net, as well as validation failures.

MOS Puppet, please take a look at this one. I'm ready to assist with debugging further.

Changed in fuel:
assignee: MOS Nova (mos-nova) → Nikita Karpin (mkarpin)
Nikita Karpin (mkarpin) wrote :

There is a problem with ordering of fuel-library tasks - both ironic-compute and server-nova tasks are declaring the same parameters, and if ironic-compute task goes before server-nova task there will be no refresh event for nova-api service, the first draft for the fix is https://review.openstack.org/409332.

But I have a question - was this bug observed in test cases which don't contain ironic?

Changed in fuel:
status: Confirmed → In Progress
Nikita Karpin (mkarpin) wrote :

I mean they are declaring ::nova::network::neutron class with same parameters

Reviewed: https://review.openstack.org/409685
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a5a7f1d6330d0699d7f63da7bbbe176bfeab0ff5
Submitter: Jenkins
Branch: master

commit a5a7f1d6330d0699d7f63da7bbbe176bfeab0ff5
Author: Mykyta Karpin <email address hidden>
Date: Mon Dec 12 11:23:18 2016 +0200

    Fix ironic-compute task ordering

    Both openstack-network/server-nova task and ironic/ironic-compute task
    declare ::nova::network::neutron class with same parameters,
    in case when ironic-compute task is applied before server-nova task,
    nova-api service will not be refreshed, because nova_config resources
    from ::nova::network::neutron were not updated.

    This patch ensures that ironic-compute task will be applied
    after openstack network configuration.

    Change-Id: Ie13df7d2ceaa204bdd1958b79949487c2dbd24fe
    Closes-Bug: #1646833

Changed in fuel:
status: In Progress → Fix Committed
Nikita Karpin (mkarpin) on 2016-12-13
Changed in fuel:
status: Fix Committed → In Progress
Nikita Karpin (mkarpin) wrote :

ostf passed here https://custom-ci.infra.mirantis.net/view/9.x/job/9.x.custom.ubuntu.system_test/724, but it is failed on glance image upload, after env revert image I uploaded image successfully, so I think it was infra issue, so I am rebuilding it - https://custom-ci.infra.mirantis.net/view/9.x/job/9.x.custom.ubuntu.system_test/726

Roman Vyalov (r0mikiam) wrote :

there was the problem with ubuntu mirrors or internet and did not related to our Infra

Reviewed: https://review.openstack.org/409687
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1ac714e70ca5845f7fe91dea00672ab63727601e
Submitter: Jenkins
Branch: stable/newton

commit 1ac714e70ca5845f7fe91dea00672ab63727601e
Author: Mykyta Karpin <email address hidden>
Date: Mon Dec 12 11:23:18 2016 +0200

    Fix ironic-compute task ordering

    Both openstack-network/server-nova task and ironic/ironic-compute task
    declare ::nova::network::neutron class with same parameters,
    in case when ironic-compute task is applied before server-nova task,
    nova-api service will not be refreshed, because nova_config resources
    from ::nova::network::neutron were not updated.

    This patch ensures that ironic-compute task will be applied
    after openstack network configuration.

    Change-Id: Ie13df7d2ceaa204bdd1958b79949487c2dbd24fe
    Closes-Bug: #1646833
    (cherry picked from commit a5a7f1d6330d0699d7f63da7bbbe176bfeab0ff5)

tags: added: in-stable-newton
tags: added: in-stable-mitaka

Reviewed: https://review.openstack.org/409332
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6b5d44f4ecd79281ab484363cd68c97eedb6a5c8
Submitter: Jenkins
Branch: stable/mitaka

commit 6b5d44f4ecd79281ab484363cd68c97eedb6a5c8
Author: Mykyta Karpin <email address hidden>
Date: Fri Dec 9 22:44:02 2016 +0200

    Fix ironic-compute task ordering

    Both openstack-network/server-nova task and
    ironic/ironic-compute task declare ::nova::network::neutron
    class with same parameters, in case when ironic-compute task
    is applied before server-nova task, nova-api service will not
    be refreshed, because nova_config resources from
    ::nova::network::neutron were not updated.

    This patch ensures that ironic-compute task will be applied
    after openstack network configuration.

    Change-Id: Ie13df7d2ceaa204bdd1958b79949487c2dbd24fe
    Closes-Bug: #1646833
    (cherry picked from commit a5a7f1d6330d0699d7f63da7bbbe176bfeab0ff5)

Dmitry Pyzhov (dpyzhov) on 2016-12-15
Changed in fuel:
status: In Progress → Fix Committed
tags: added: on-verification
tags: removed: on-verification
Changed in fuel:
status: Fix Committed → Fix Released
Changed in fuel:
milestone: 9.2 → 11.0
status: Fix Released → Fix Committed

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

Not reproduced on Newton.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers