test_os_wait_condition fails when enabling SSL on keystone

[Guoping Jia]

When enable SSL on keystone, some wait condition tests in Heat integration keep failing with a WaitConditionTimeout. Here is one such test failure & its detailed traceback:

heat_integrationtests.functional.test_os_wait_condition.OSWaitCondition.test_create_stack_with_multi_signal_waitcondition [175.583482s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/stack/guoping/heat-scripts/heat-testcase/hos-qa-tests/heat/testcase/heat_integrations/heat_integrationtests/functional/test_os_wait_condition.py", line 107, in test_create_stack_with_multi_signal_waitcondition
        self.stack_create(template=self.template, parameters=params)
      File "/home/stack/guoping/heat-scripts/heat-testcase/hos-qa-tests/heat/testcase/heat_integrations/heat_integrationtests/common/test.py", line 521, in stack_create
        self._wait_for_stack_status(**kwargs)
      File "/home/stack/guoping/heat-scripts/heat-testcase/hos-qa-tests/heat/testcase/heat_integrations/heat_integrationtests/common/test.py", line 334, in _wait_for_stack_status
        fail_regexp):
      File "/home/stack/guoping/heat-scripts/heat-testcase/hos-qa-tests/heat/testcase/heat_integrations/heat_integrationtests/common/test.py", line 300, in _verify_status
        stack_status_reason=stack.stack_status_reason)
    heat_integrationtests.common.exceptions.StackBuildErrorException: Stack OSWaitCondition-1576040942/b3369043-64f9-48aa-97a1-3fec92ed8e78 is in CREATE_FAILED status due to 'Resource CREATE failed: WaitConditionTimeout: resources.wait_condition: 0 of 25 received'

[Peter Razumovsky (prazumovsky)]

Hello! What version of Heat you used?

[Guoping Jia (guoping-jia)]

The Heat integration is running against Mitaka branch. Here is the heat version:

$ heat --version
1.1.1

[Guoping Jia (guoping-jia)]

By the way, I saw the same timeout when running Heat integration against Liberty branch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/334962

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/334962
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=c8ed9d28244aac4f0c245064e0f6866eb5535ede
Submitter: Jenkins
Branch: stable/mitaka

commit c8ed9d28244aac4f0c245064e0f6866eb5535ede
Author: danny <email address hidden>
Date: Sat Apr 9 19:33:09 2016 +0800

    Add --insecure in CURL if set True in client option

    The CURL attribute is offen used as part of
    the user data to signal back to heat in wait
    handle. If the insecure option is set True in
    the clients_heat option, we need to add
    --insecure to the CURL url to make it work.

    Change-Id: I153a9c71837ee61632e0cf39254bbbc66427b1de
    Closes-Bug: #1594441
    (cherry picked from commit 8f98d34f3649d884b6bfc79e1e99e2aec7a4e137)

[Guoping Jia (guoping-jia)]

After heat reconfiguration by adding 'insecure = True' to the section clients_heat (see below), I've re-run the heat integration in Mitaka.

[clients_heat]
endpoint_type = publicURL
insecure = True

However, the test_os_wait_condition still kept failing with the same error:

{19} heat_integrationtests.functional.test_os_wait_condition.OSWaitCondition.test_create_stack_with_multi_signal_waitcondition [151.113822s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/stack/guoping/heat-scripts/heat-testcase/hos-qa-tests/heat/testcase/heat_integrations/heat_integrationtests/functional/test_os_wait_condition.py", line 107, in test_create_stack_with_multi_signal_waitcondition
        self.stack_create(template=self.template, parameters=params)
      File "/home/stack/guoping/heat-scripts/heat-testcase/hos-qa-tests/heat/testcase/heat_integrations/heat_integrationtests/common/test.py", line 521, in stack_create
        self._wait_for_stack_status(**kwargs)
      File "/home/stack/guoping/heat-scripts/heat-testcase/hos-qa-tests/heat/testcase/heat_integrations/heat_integrationtests/common/test.py", line 334, in _wait_for_stack_status
        fail_regexp):
      File "/home/stack/guoping/heat-scripts/heat-testcase/hos-qa-tests/heat/testcase/heat_integrations/heat_integrationtests/common/test.py", line 300, in _verify_status
        stack_status_reason=stack.stack_status_reason)
    heat_integrationtests.common.exceptions.StackBuildErrorException: Stack OSWaitCondition-2056615565/104f4255-7ad3-487a-b226-2559ed44806d is in CREATE_FAILED status due to 'Resource CREATE failed: WaitConditionTimeout: resources.wait_condition: 0 of 25 received'

[Peter Razumovsky (prazumovsky)]

I cannot reproduce your issue. Could you share your config file heat.conf?

[Guoping Jia (guoping-jia)]

Sorry for getting back to you late. Here is the heat.conf file we are using.

[DEFAULT]
verbose = True
debug = True
use_syslog = False
log_dir = /var/log/heat
region_name_for_services = xxx
rpc_backend = rabbit
auth_encryption_key = xxx
deferred_auth_method = trusts
stack_user_domain_id = xxx
stack_domain_admin = xxx
stack_domain_admin_password = xxx
heat_watch_server_url = https://xxx:8003
heat_metadata_server_url = https://xxx:8000
heat_waitcondition_server_url = https://xxx:8000/v1/waitcondition
notification_topic = notifications
notification_driver = messaging
environment_dir = /opt/stack/service/heat-engine/etc/heat/environment.d
plugin_dirs = /opt/stack/service/heat-engine/venv/share/lib/heat
heat_stack_user_role = heat_stack_user
encrypt_parameters_and_properties = True

[oslo_messaging_rabbit]
rabbit_userid = xxx
rabbit_hosts = xxx:5672,xxx:5672
rabbit_password = xxx
rabbit_use_ssl = False

[database]
connection = mysql://heat:xxxx@xxxx/heat?charset=utf8

[keystone_authtoken]
auth_uri = https://x.x.x.x:5000/v2.0
admin_tenant_name = services
admin_password = xxxx
admin_user = heat
identity_uri = https://x.x.x.x:5000

[ec2authtoken]
auth_uri = https://x.x.x.x:5000/v2.0

[audit_middleware_notifications]
driver = log

[heat_api]
bind_port = 8004
workers = 2
bind_host = x.x.x.x

[heat_api_cfn]
bind_port = 8000
bind_host = x.x.x.x

[heat_api_cloudwatch]
bind_port = 8003
bind_host = x.x.x.x

[paste_deploy]
api_paste_config = /opt/stack/service/heat-engine/etc/heat/api-paste.ini

[clients]
endpoint_type = internalURL

[clients_heat]
endpoint_type = publicURL
ca_file = /etc/ssl/certs/ca-certificates.crt
insecure = True

### End of File ###
## Do NOT put anything after this line ##

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 6.1.0 release.

[Peter Razumovsky (prazumovsky)]

OK, then, if it's still reproducible, can you share heat_integrationtests.conf file?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 6.1.0 release.