[9.2] [Tempest] Functional OSTF fail to start instances

[Yury Tregubov]

Since 9.2 snapshot #578 the following problem is seen right after deploy of tempest envs:

AssertionError: Failed 8 OSTF tests; should fail 0 tests. Names of failed tests:
  - Create volume and boot instance from it (failure) Instance creation failed. Please refer to OpenStack logs for more details.
  - Create volume and attach it to instance (failure) Instance creation failed. Please refer to OpenStack logs for more details.
  - Check network connectivity from instance via floating IP (failure) Failed to create rule in security group. Please refer to OpenStack logs for more details.
  - Create security group (failure) Failed to create rule in security group. Please refer to OpenStack logs for more details.
  - Check network parameters (error)
  - Launch instance (failure) Failed to create rule in security group. Please refer to OpenStack logs for more details.
  - Launch instance with file injection (failure) Failed to create rule in security group. Please refer to OpenStack logs for more details.
  - Launch instance, create snapshot, launch instance from snapshot (failure) Default private network 'admin_internal_net' isn't present. Please verify it is properly created. Please refer to OpenStack logs for more details.

That blocks Tempest execution.

Diagnostic snapshot is: https://drive.google.com/a/mirantis.com/file/d/0B1Crk-sAvGanUUJPMFFTQXFDc2s/view?usp=sharing

Logs from deploy are available here: http://172.18.173.8:8080/jenkins/view/Tempest_9.%D0%A5/job/9.x_Tempest_Detached_RabbitMQ/190/console

[Timur Nurlygayanov (tnurlygayanov)]

The issue affects SWARM pass rate as well:

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ironic_deploy_swift/144/testReport/(root)/ironic_deploy_swift/ironic_deploy_swift/

[Oleksiy Molchanov (omolchanov)]

QA team, it is sending request that contains 'null' value for security group id

2016-12-05 23:55:10.327 28485 DEBUG nova.api.openstack.wsgi [req-1faf753d-6e34-4574-a777-729f2cb5391d 197788e0f8dd49e99259e971474c1b6d e06694adff6a4c7a987996e8e1cc0aea - - -] Action: 'create', calling method: <bound method SecurityGroupRulesController.create of <nova.api.openstack.compute.security_groups.SecurityGroupRulesController object at 0x7f14fcd88810>>, body: {"security_group_rule": {"from_port": -1, "ip_protocol": "icmp", "to_port": -1, "parent_group_id": 6, "cidr": "0.0.0.0/0", "group_id": null}} _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:696

[Vladimir Khlyunev (vkhlyunev)]

OSTF if working fine using received from novaclient parameters; however there is really strange behavior in nova:

https://paste.mirantis.net/show/2843/

NovaClient creates secgroup with "int" id and then fails at validation step; BUT after several ostf debug restarts I got reversed situation - secgroup id was "uuid" and nova traces with "Security group id should be integer"

Really weird that I got following situation once and it was not reproduced:
While debugging I tries "nova secgroup-list" on controller and it shows 4 created by ostf testgroup BUT they does not matches with secgroup showed in horizon. Then I created secgroup using dashboard and it WAS NOT presented in "nova secgroup-list". Then I created secgroup using "nova secgroup-create" -- weird place -- output of the command was completely changed and show all new groups (no one old was present in new output) and here id became uuid type. It looks like we have 2 databases with secgroups.

[Roman Podoliaka (rpodolyaka)]

The inconsistent behaviour Vladimir is seeing is caused by the fact that part of nova services are configured to use nova-network, while the other part is configured (correctly) to use neutron:

http://paste.openstack.org/show/591960/

This explains the issues with missing admin_internal_net, as well as validation failures.

MOS Puppet, please take a look at this one. I'm ready to assist with debugging further.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/409332

[Nikita Karpin (mkarpin)]

There is a problem with ordering of fuel-library tasks - both ironic-compute and server-nova tasks are declaring the same parameters, and if ironic-compute task goes before server-nova task there will be no refresh event for nova-api service, the first draft for the fix is https://review.openstack.org/409332.

But I have a question - was this bug observed in test cases which don't contain ironic?

[Nikita Karpin (mkarpin)]

I mean they are declaring ::nova::network::neutron class with same parameters

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/409685

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/409687

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409685
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a5a7f1d6330d0699d7f63da7bbbe176bfeab0ff5
Submitter: Jenkins
Branch: master

commit a5a7f1d6330d0699d7f63da7bbbe176bfeab0ff5
Author: Mykyta Karpin <email address hidden>
Date: Mon Dec 12 11:23:18 2016 +0200

    Fix ironic-compute task ordering

    Both openstack-network/server-nova task and ironic/ironic-compute task
    declare ::nova::network::neutron class with same parameters,
    in case when ironic-compute task is applied before server-nova task,
    nova-api service will not be refreshed, because nova_config resources
    from ::nova::network::neutron were not updated.

    This patch ensures that ironic-compute task will be applied
    after openstack network configuration.

    Change-Id: Ie13df7d2ceaa204bdd1958b79949487c2dbd24fe
    Closes-Bug: #1646833

[Nikita Karpin (mkarpin)]

waiting for custom tests results - https://custom-ci.infra.mirantis.net/view/9.x/job/9.x.custom.ubuntu.system_test/724/

[Nikita Karpin (mkarpin)]

ostf passed here https://custom-ci.infra.mirantis.net/view/9.x/job/9.x.custom.ubuntu.system_test/724, but it is failed on glance image upload, after env revert image I uploaded image successfully, so I think it was infra issue, so I am rebuilding it - https://custom-ci.infra.mirantis.net/view/9.x/job/9.x.custom.ubuntu.system_test/726

[Roman Vyalov (r0mikiam)]

there was the problem with ubuntu mirrors or internet and did not related to our Infra

[Nikita Karpin (mkarpin)]

https://custom-ci.infra.mirantis.net/job/9.x.custom.ubuntu.system_test/730/console is successful finally

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409687
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1ac714e70ca5845f7fe91dea00672ab63727601e
Submitter: Jenkins
Branch: stable/newton

commit 1ac714e70ca5845f7fe91dea00672ab63727601e
Author: Mykyta Karpin <email address hidden>
Date: Mon Dec 12 11:23:18 2016 +0200

    Fix ironic-compute task ordering

    Both openstack-network/server-nova task and ironic/ironic-compute task
    declare ::nova::network::neutron class with same parameters,
    in case when ironic-compute task is applied before server-nova task,
    nova-api service will not be refreshed, because nova_config resources
    from ::nova::network::neutron were not updated.

    This patch ensures that ironic-compute task will be applied
    after openstack network configuration.

    Change-Id: Ie13df7d2ceaa204bdd1958b79949487c2dbd24fe
    Closes-Bug: #1646833
    (cherry picked from commit a5a7f1d6330d0699d7f63da7bbbe176bfeab0ff5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409332
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6b5d44f4ecd79281ab484363cd68c97eedb6a5c8
Submitter: Jenkins
Branch: stable/mitaka

commit 6b5d44f4ecd79281ab484363cd68c97eedb6a5c8
Author: Mykyta Karpin <email address hidden>
Date: Fri Dec 9 22:44:02 2016 +0200

    Fix ironic-compute task ordering

    Both openstack-network/server-nova task and
    ironic/ironic-compute task declare ::nova::network::neutron
    class with same parameters, in case when ironic-compute task
    is applied before server-nova task, nova-api service will not
    be refreshed, because nova_config resources from
    ::nova::network::neutron were not updated.

    This patch ensures that ironic-compute task will be applied
    after openstack network configuration.

    Change-Id: Ie13df7d2ceaa204bdd1958b79949487c2dbd24fe
    Closes-Bug: #1646833
    (cherry picked from commit a5a7f1d6330d0699d7f63da7bbbe176bfeab0ff5)

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 snapshot #638.

Currently all OSTF tests are passed in the run:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.ironic_deploy_swift/152/testReport/(root)/ironic_deploy_swift/ironic_deploy_swift/

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

[Dmitry Belyaninov (dbelyaninov)]

Not reproduced on Newton.