tempest orchestration api for neutron resources fails

[Hemanth Nakkina]

In a typical OpenStack environment (Non Devstack environment) the following tempest test cases failed.

tox -eall -- tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON

{0} setUpClass (tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON) [0.000000s] ... FAILED

The reason for the failure is the heat template used as part of setup() class for the above tests - neutron_basic.yaml have resources of type AWS::CloudFormation::WaitCondition, AWS::CloudFormation::WaitConditionHandle which mandates Orchestration API should be reachable from provisioned VM.
In a typical OpenStack environment, there wont be any reachability between Host machines and VM.
(These test cases work in devstack in single host environment because of typical routing on the host machine which is nothing to do with openstack)

The question here is do we need AWS::CloudFormation::WaitCondition, AWS::CloudFormation::WaitConditionHandle in neutron_basic.yaml heat template.

The test cases under NeutronResourcesTestJSON have no relation with the resources (WaitCondition, WaitConditionHandle)

Suggestion is to remove the userdata from OS::Nova::Resource section, AWS::CloudFormation::WaitCondition, AWS::CloudFormation::WaitConditionHandle from neutron_basic.yaml heat template.

Verified on stable/ocata.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/448574

[Attila Fazekas (afazekas)]

The heat-cfn-api service is not expected to be ruining on the hypervisor,
it can be anywhere, as the heat-api service. If the heat-cfn not reachable from your vm's basically your heat-cfn service is not working.

If our assumption about if someone want's heat he also want's cfn was wrong,
we might consider adding a cfn specific feature config flag, and skip the test fully, when cfn is not present.

[Hemanth Nakkina (hemanth-n)]

I understand heat-cfn-api service can run anywhere but I disagree with your statement "If the heat-cfn not reachable from your vm's basically your heat-cfn service is not working" since not all the CloudFormation Resource Types require heat-cfn service reachability from VM.

There can be deployments that does not require VM reachability to heat-cfn-api service but can use heat-cfn-api service for some resource types.
Feel free to correct me if I am wrong.

Following test cases are not executed because of the setupClass failure
tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON.test_created_network
tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON.test_created_resources
tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON.test_created_router
tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON.test_created_router_interface
tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON.test_created_server
tempest.api.orchestration.stacks.test_neutron_resources.NeutronResourcesTestJSON.test_created_subnet

Intention here is to test network orchestration resource types (and nova resource type "OS::Nova::Server" as well) and not AWS::CloudFormation::WaitCondition.
So I don't see why these test cases shouldn't get executed because of more constrained basic heat template used as part of setup which has no relation with test cases.

If the core team still thinks atleast OS::Nova::Server should be mapped with AWS::CloudFormation::WaitCondition for test case verification, I suggest moving test_created_server to a new one like NovaResourcesTestJSON.

[Attila Fazekas (afazekas)]

Maybe the naming and place of test within the tempest hierarchy is not the best.
complete removal of that test was also proposed: https://review.openstack.org/#/c/351426/ .

Basically, tempest is mainly for integration tests,
I do not think it is really good idea to remove vm call back kind of test step.

As I remember the heat members also did not found really useful to have basic resource create tests on the tempest side.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Hemanth Nakkina (<email address hidden>) on branch: master
Review: https://review.openstack.org/448574
Reason: As per comments from afazekas on launchpad, as the tests are mainly for integration but not for resource create tests, the bug can be marked as invalid.

[Attila Fazekas (afazekas)]

Some ipv6 config needs to be better supported, for example.: when
the cfn api is advertised only with an ipv6 address.

[Attila Fazekas (afazekas)]

What caused the issue in your environment ? How generic is the unreachability ? Was it really a design decision or an unexpected consequence ?

[Hemanth Nakkina (hemanth-n)]

As I mentioned in the bug description, typically it is not mandatory in openstack deployments that a provisioned VM will be reachable to OpenStack API. (considering IPv4, I haven't explored IPv6)

Its a design decision.
Typically any VM to reach to OpenStack API requires Floating IP or by using Provider networks. In both cases assuming external routers are configured to route traffic from Floating IP network to OpenStack API network.

[Attila Fazekas (afazekas)]

IMHO using some config option for skipping might be a good idea (default not skipping).

Looks like it is possible to create an ipv6 deployment where the WaitCondition would be able to work, but not with this test case, since the test creates an ipv4 vm and the heat API is ipv6 only,
The possible ipv6 variant(s) of the test are so different to the current one, and it would require a different template to work, so it would be a completely different case(s)..
This ipv6 deployment does not feels like to the today's best practice, but it is possible.

`OpenStack API network` Do you consider swift as part of this network ?

ATM I am unsure about what would be the best `future proof` skip condition and option,
because both cases just fails this testclass.

We do not really want to have too many options in the tempest.conf, and we also do not want to change them frequently (we have to depreciate them first..)

Do we want to go with some WaitCondition specific option, or with some generic property of the default tenant network(s) or the external network routeability ?

Is the design decision expected to stay at tomorrow ? Are you planning to add some NAT or route rule (as many of us does it on test systems) ?

[Launchpad Janitor (janitor)]

[Expired for tempest because there has been no activity for 60 days.]