Task ssl-dns-setup ain't applied in right time

[Maksim Malchuk]

https://product-ci.infra.mirantis.net/job/9.0-liberty.ubuntu.bvt_2/127/console

fuel-9.0-liberty-171-2016-02-02_15-31-07.iso

failed on node-4 with:

2016-02-02 17:18:30 +0000 /Stage[main]/Main/Create_cinder_types[volumes_ceph]/Cinder::Type[volumes_ceph]/Exec[cinder type-create volumes_ceph] (info): Starting to evaluate the resource
2016-02-02 17:18:30 +0000 Exec[cinder type-create volumes_ceph](provider=posix) (debug): Executing check 'cinder type-list | grep -qP '\svolumes_ceph\s''
2016-02-02 17:18:30 +0000 Puppet (debug): Executing 'cinder type-list | grep -qP '\svolumes_ceph\s''
2016-02-02 17:18:31 +0000 /Stage[main]/Main/Create_cinder_types[volumes_ceph]/Cinder::Type[volumes_ceph]/Exec[cinder type-create volumes_ceph]/unless (debug): ERROR: Unable to establish connection to https://public.fuel.local:5000/v2.0/tokens
2016-02-02 17:18:31 +0000 /Stage[main]/Main/Create_cinder_types[volumes_ceph]/Cinder::Type[volumes_ceph]/Exec[cinder type-create volumes_ceph]/returns (debug): Exec try 1/2
2016-02-02 17:18:31 +0000 Exec[cinder type-create volumes_ceph](provider=posix) (debug): Executing 'cinder type-create volumes_ceph'
2016-02-02 17:18:31 +0000 Puppet (debug): Executing 'cinder type-create volumes_ceph'
2016-02-02 17:18:31 +0000 /Stage[main]/Main/Create_cinder_types[volumes_ceph]/Cinder::Type[volumes_ceph]/Exec[cinder type-create volumes_ceph]/returns (debug): Sleeping for 5.0 seconds between tries
2016-02-02 17:18:36 +0000 /Stage[main]/Main/Create_cinder_types[volumes_ceph]/Cinder::Type[volumes_ceph]/Exec[cinder type-create volumes_ceph]/returns (debug): Exec try 2/2
2016-02-02 17:18:36 +0000 Exec[cinder type-create volumes_ceph](provider=posix) (debug): Executing 'cinder type-create volumes_ceph'
2016-02-02 17:18:36 +0000 Puppet (debug): Executing 'cinder type-create volumes_ceph'
2016-02-02 17:18:37 +0000 /Stage[main]/Main/Create_cinder_types[volumes_ceph]/Cinder::Type[volumes_ceph]/Exec[cinder type-create volumes_ceph]/returns (debug): Sleeping for 5.0 seconds between tries
2016-02-02 17:18:42 +0000 /Stage[main]/Main/Create_cinder_types[volumes_ceph]/Cinder::Type[volumes_ceph]/Exec[cinder type-create volumes_ceph]/returns (notice): ERROR: Unable to establish connection to https://public.fuel.local:5000/v2.0/tokens
2016-02-02 17:18:42 +0000 Puppet (err): cinder type-create volumes_ceph returned 1 instead of one of [0]
/usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
/usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:160:in `sync'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:204:in `sync'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `block in perform_changes'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `each'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `perform_changes'
/usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
/usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
/usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
/usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
/usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
/usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
/usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
/usr/bin/puppet:8:in `<main>'
2016-02-02 17:18:42 +0000 /Stage[main]/Main/Create_cinder_types[volumes_ceph]/Cinder::Type[volumes_ceph]/Exec[cinder type-create volumes_ceph]/returns (err): change from notrun to 0 failed: cinder type-create volumes_ceph returned 1 instead of one of [0]
2016-02-02 17:18:42 +0000 /Stage[main]/Main/Create_cinder_types[volumes_ceph]/Cinder::Type[volumes_ceph]/Exec[cinder type-create volumes_ceph] (info): Evaluated in 11.71 seconds

[Dmitry Pyzhov (dpyzhov)]

It is unclear if it could affect 8.0. Marking as Incomplete for 8.0

[Maksim Malchuk (mmalchuk)]

https://product-ci.infra.mirantis.net/job/8.0.ubuntu.bvt_2/465/
same problem

found an interesting info in logfile:
2016-02-03T12:09:55.987899+00:00 notice: (/Stage[main]/Ceph::Osds/Ceph::Osds::Osd[/dev/vdb3]/Exec[ceph-deploy osd prepare node-1:/dev/vdb3]/returns) [node-1][WARNING] Error: Partition(s) 1 on /dev/vdb3 have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes.

[Vladimir Kuklin (vkuklin)]

This message may not be relevant - we need to check whether ceph is health. If it is, then this bug is cinder-specific only.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/275803

[Michael Polenchuk (mpolenchuk)]

ERROR: Unable to establish connection to https://public.fuel.local:5000/v2.0/tokens
This caused that task ssl-dns-setup not applied in right time.
- id: ssl-dns-setup
  groups: [primary-controller, controller]
  requires: [firewall, ssl-add-trust-chain]
  required_for: [deploy_end]

[Michael Polenchuk (mpolenchuk)]

Raised to critical as it could affect overall deployment with SSL enabled.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275803
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4c7894d171c6633e339a1869e8cb4903197b785f
Submitter: Jenkins
Branch: master

commit 4c7894d171c6633e339a1869e8cb4903197b785f
Author: Ivan Ponomarev <email address hidden>
Date: Wed Feb 3 19:22:21 2016 +0300

    change depends for ssl-dns-setup

      - public.fuel.local should be added to hosts file
        before all openstack services

    Closes-bug: #1541054
    Change-Id: I768edb86f9356128c5db00af45cb2e786e9fa9f6

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/275928

[Maksim Malchuk (mmalchuk)]

Custom BVT_2 with committed fix: http://jenkins-product.srt.mirantis.net:8080/job/9.0.custom.ubuntu.bvt_2/178/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275928
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=29829b131ca802830bc5a9a131c83cd0f43f702b
Submitter: Jenkins
Branch: stable/8.0

commit 29829b131ca802830bc5a9a131c83cd0f43f702b
Author: Ivan Ponomarev <email address hidden>
Date: Wed Feb 3 19:22:21 2016 +0300

    change depends for ssl-dns-setup

      - public.fuel.local should be added to hosts file
        before all openstack services

    Closes-bug: #1541054
    Change-Id: I768edb86f9356128c5db00af45cb2e786e9fa9f6
    (cherry picked from commit 4c7894d171c6633e339a1869e8cb4903197b785f)

[Maksim Malchuk (mmalchuk)]

Custom BVT_2 pass all OSTF tests but failed at the end because of https://bugs.launchpad.net/fuel/+bug/1541390, so this is good!

[Andrey Sledzinskiy (asledzinskiy)]

verified by 8.0 and 9.0 bvt tests