All OSTF tests failed with enabled SSL

[Vasily Gorin]

ISO: 478

Steps to reproduce:
   1. Disable plain http for nailgun
          Add to the /etc/fuel/astute.yaml
         "SSL":
             "force_https": "true"

             restart nginx container using command 'dockerctl restart nginx'
  2. create cluster :
       Ubuntu
       neutron tun
       1 controller
       1 compute
       1 cinder

        and deploy it
  3 Run OSTF tests

Expected result:
  All OSTF tests passed

Actual result:
 All OSTF tests failed

First trace:

(nose_storage_plugin) <nose.suite.ContextSuite context=RabbitSanityTest>:setup
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nose/suite.py", line 209, in run
    self.setUp()
  File "/usr/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp
    self.setupContext(ancestor)
  File "/usr/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext
    try_run(context, names)
  File "/usr/lib/python2.7/site-packages/nose/util.py", line 471, in try_run
    return func()
  File "/usr/lib/python2.7/site-packages/fuel_health/ha_base.py", line 37, in setUpClass
    cls.nodes = cls.config.compute.nodes
AttributeError: 'ConfigGroup' object has no attribute 'nodes'
-------------------- >> begin captured logging << --------------------
fuel_health.config: INFO: INSTANCE {502: <fuel_health.config.NailgunConfig object at 0x509bb50>}
--------------------- >> end captured logging << ---------------------

Actual latest traces:

(nose_storage_plugin) fuel_health.tests.sanity.test_sanity_glance.GlanceSanityTests.test_glance_image_list
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
    yield
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 597, in run
    self.setUp()
  File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 560, in setUp
    self.check_clients_state()
  File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 454, in check_clients_state
    self.fail(self.manager.keystone_error_message)
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.
-------------------- >> begin captured logging << --------------------
fuel_health.config: INFO: INSTANCE {500: <fuel_health.config.NailgunConfig object at 0x509bd90>}
fuel_health.config: INFO: INSTANCE {500: <fuel_health.config.NailgunConfig object at 0x509bd90>}
keystoneclient.auth.identity.v2: DEBUG: Making authentication request to http://localhost/tokens
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): localhost
requests.packages.urllib3.connectionpool: DEBUG: "POST /tokens HTTP/1.1" 301 236
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): localhost
requests.packages.urllib3.connectionpool: DEBUG: "POST /tokens HTTP/1.1" 301 178
requests.packages.urllib3.connectionpool: INFO: Starting new HTTPS connection (1): localhost
requests.packages.urllib3.connectionpool: DEBUG: "POST /tokens HTTP/1.1" 404 162
keystoneclient.session: DEBUG: Request returned failure status: 404
fuel_health.nmanager: DEBUG: Unable to initialize Keystone client:
--------------------- >> end captured logging << ---------------------

(nose_storage_plugin) fuel_health.tests.configuration.test_configuration.SanityConfigurationTest.test_003_check_default_keystone_credential_usage
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
    yield
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 597, in run
    self.setUp()
  File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 1150, in setUp
    self.check_clients_state()
  File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 454, in check_clients_state
    self.fail(self.manager.keystone_error_message)
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.
-------------------- >> begin captured logging << --------------------
fuel_health.nmanager: DEBUG: Unable to initialize Keystone client:
--------------------- >> end captured logging << ---------------------

[Vasily Gorin (vgorin)]

Diagnostic snapshot : https://drive.google.com/a/mirantis.com/file/d/0B1I4oOG20Aa_dXNBOUVOSmpaRTA/view?usp=sharing

[Ivan Ponomarev (ivanzipfer)]

Do this bug block something?
Please provide the Fuel version

[Vasily Gorin (vgorin)]

ISO build 478

[root@nailgun ~]# fuel --version
8.0.0

[Tatyanka (tatyana-leontovich)]

 Looks like related https://bugs.launchpad.net/fuel/+bug/1530318

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/273573

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/273573
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=dcf5047446067e29f5fd577ceab2aabc541f1204
Submitter: Jenkins
Branch: master

commit dcf5047446067e29f5fd577ceab2aabc541f1204
Author: Artem Roma <email address hidden>
Date: Thu Jan 28 16:30:25 2016 +0200

    Disable ssl verification while setting config for fuel_health

    In case 'force_https' is turned on for fuel we must disable ssl cert
    verification for code that performs http requests inside fuel
    infrastructure (e.g. requests to nailgun). The verification is
    meaningless by itself as we use self signed cert.

    Change-Id: Ic47d91d1a59ef2741bfab8709ff0b0985fb1919d
    Closes-Bug: #1538977

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/274590

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/274590
Committed: https://git.openstack.org/cgit/openstack/fuel-ostf/commit/?id=5fe41945c2a49f26c849df1fd46329f6db1ab6b0
Submitter: Jenkins
Branch: stable/8.0

commit 5fe41945c2a49f26c849df1fd46329f6db1ab6b0
Author: Artem Roma <email address hidden>
Date: Thu Jan 28 16:30:25 2016 +0200

    Disable ssl verification while setting config for fuel_health

    In case 'force_https' is turned on for fuel we must disable ssl cert
    verification for code that performs http requests inside fuel
    infrastructure (e.g. requests to nailgun). The verification is
    meaningless by itself as we use self signed cert.

    Change-Id: Ic47d91d1a59ef2741bfab8709ff0b0985fb1919d
    Closes-Bug: #1538977
    (cherry picked from commit dcf5047446067e29f5fd577ceab2aabc541f1204)

[Dmitriy Kruglov (dkruglov)]

Verified on MOS 8.0, build 515. The issue is fixed.

ISO details:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "515"
  build_id: "515"
  fuel-nailgun_sha: "c363c742bf3a7e7881239c8819ff5c6622f90625"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "4a783e271a0c4b889d16b66dd8410766b2b00813"
  fuel-ostf_sha: "5fe41945c2a49f26c849df1fd46329f6db1ab6b0"
  fuel-mirror_sha: "eea143f2c6ed91dcaf55bc0911677f6fdc5a9133"
  fuelmenu_sha: "12227354aec1d38f7f51042df64cca59fa7a95f1"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "c5aac6c0b8d2a37147fdd6e6b78df4ad2ad90e31"

[Mikhail Samoylov (msamoylov)]

Verification failed in fuel version 9.0 (fuel-9.0-45-2016-03-11_18-49-00.iso)

1. Disable plain http for nailgun
          Add to the /etc/fuel/astute.yaml
         "SSL":
             "force_https": "true"

2. Apply puppet job puppet apply /etc/puppet/liberty-9.0/modules/fuel/examples/nginx_services.pp
3. create cluster :
       Ubuntu
       neutron tun
       1 controller
       1 compute
       1 cinder

        and deploy it
  3 Go to ostf tab in UI

Expected result:
We can access to OSTF Tab through UI

Actual result:
Cannot get access to OSTF Tasb

If I enable HTTP log for OSTF:

http://paste.openstack.org/show/491093/
With HTTPS only logs for OSTF:
http://paste.openstack.org/show/491094/

[Artem Roma (aroma-x)]

@Mikhail, could you please provide diagnostic snapshot from the failed environment?

[Mikhail Samoylov (msamoylov)]

https://drive.google.com/a/mirantis.com/file/d/0B2SenDuhfXPlMzZxNFhLcWFySW8 -snapshot.

[Artem Roma (aroma-x)]

I cannot reproduce the issue on my environment with given in comment #10 steps. I use https://product-ci.infra.mirantis.net/job/9.0.all/97/ build. After deployment and enabling of the option for nginx tests are discovered and can be run w/o any issues.

@Mikhail, only those actions, as described in comment #10, had been performed with the environment? If not, I would like to know what additional operations had taken place.

If anything, could you, please, also verify that bug is reproduced on latest iso? Again, if yes, don't hesitate to ping me immediately, as I want to get access to live environment in such case because I don't think that the issue here is with ostf and further debugging of additional components might be needed.

[Mikhail Samoylov (msamoylov)]

Verification passed in fuel version 9.0 (fuel-9.0-97-2016-03-22_08-00-00.iso)

1. Disable plain http for nailgun
          Add to the /etc/fuel/astute.yaml
         "SSL":
             "force_https": "true"

2. Apply puppet job puppet apply /etc/puppet/liberty-9.0/modules/fuel/examples/nginx_services.pp
3. create cluster :
       Ubuntu
       neutron tun
       1 controller
       1 compute
       1 cinder

        and deploy it
3 Go to ostf tab in UI
4. Run OSTF