Deployment fails because 'swift-rebalance-cron' is performed before Swift is installed: "err: Puppet::Util::FileType::FileTypeCrontab could not write swift: Broken pipe"

[Artem Panchenko]

Environment deployment fails on primary controller:

2016-01-26 00:25:25 ERROR [771] Task '{"priority"=>5600, "type"=>"puppet", "id"=>"swift-rebalance-cron", "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/swift/rebalance_cronjob.pp", "timeout"=>300, "cwd"=>"/"}, "uids"=>["1"]}' failed on node 1
2016-01-26 00:25:25 DEBUG [771] Task time summary: swift-rebalance-cron with status error on node 1 took 00:00:31

node-1 2016-01-26T00:25:18.312871 notice: (Scope(Class[main])) MODULAR: swift/rebalance_cronjob.pp
node-1 2016-01-26T00:25:18.570702 notice: Compiled catalog for node-1.test.domain.local in environment production in 0.28 seconds
node-1 2016-01-26T00:25:19.460039 err: Puppet::Util::FileType::FileTypeCrontab could not write swift: Broken pipe
node-1 2016-01-26T00:25:19.460039 err: /usr/lib/ruby/vendor_ruby/puppet/util/filetype.rb:201:in `write'
node-1 2016-01-26T00:25:19.460039 err: /usr/lib/ruby/vendor_ruby/puppet/util/filetype.rb:201:in `print'
node-1 2016-01-26T00:25:19.460334 err: /usr/lib/ruby/vendor_ruby/puppet/util/filetype.rb:201:in `block in write'
node-1 2016-01-26T00:25:19.460334 err: /usr/lib/ruby/vendor_ruby/puppet/util/filetype.rb:200:in `popen'
node-1 2016-01-26T00:25:19.460334 err: /usr/lib/ruby/vendor_ruby/puppet/util/filetype.rb:200:in `write'
node-1 2016-01-26T00:25:19.460334 err: /usr/lib/ruby/vendor_ruby/puppet/util/filetype.rb:58:in `block (2 levels) in newfiletype'
node-1 2016-01-26T00:25:19.460334 err: /usr/lib/ruby/vendor_ruby/puppet/provider/parsedfile.rb:98:in `flush_target'
node-1 2016-01-26T00:25:19.460334 err: /usr/lib/ruby/vendor_ruby/puppet/provider/parsedfile.rb:71:in `block in flush'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/provider/parsedfile.rb:68:in `each'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/provider/parsedfile.rb:68:in `flush'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/provider/parsedfile.rb:428:in `flush'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/type.rb:990:in `flush'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:24:in `evaluate'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
node-1 2016-01-26T00:25:19.462887 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
node-1 2016-01-26T00:25:19.462887 err: /usr/bin/puppet:8:in `<main>'
node-1 2016-01-26T00:25:19.463982 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) Could not evaluate: Puppet::Util::FileType::FileTypeCrontab could not write swift: Broken pipe
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util/filetype.rb:201:in `write'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util/filetype.rb:201:in `print'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util/filetype.rb:201:in `block in write'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util/filetype.rb:200:in `popen'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util/filetype.rb:200:in `write'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util/filetype.rb:58:in `block (2 levels) in newfiletype'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/provider/parsedfile.rb:98:in `flush_target'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/provider/parsedfile.rb:71:in `block in flush'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/provider/parsedfile.rb:68:in `each'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/provider/parsedfile.rb:68:in `flush'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/provider/parsedfile.rb:428:in `flush'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/type.rb:990:in `flush'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:24:in `evaluate'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:204:in `apply'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:217:in `eval_resource'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `call'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block (2 levels) in evaluate'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util.rb:335:in `block in thinmark'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util.rb:334:in `thinmark'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:147:in `block in evaluate'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:138:in `evaluate'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
node-1 2016-01-26T00:25:19.464997 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
node-1 2016-01-26T00:25:19.465965 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
node-1 2016-01-26T00:25:19.467036 err: (/Stage[main]/Openstack::Swift::Rebalance_cronjob/Cron[swift-rings-rebalance]) /usr/bin/puppet:8:in `<main>'
node-1 2016-01-26T00:25:19.801147 notice: Finished catalog run in 0.68 seconds

Steps to reproduce:

         1. Create cluster in Ha mode with 1 controller
         2. Add 1 node with controller role
         3. Add 1 node with compute and Ceph OSD roles
         4. Add 1 node with Ceph OSD role
         5. Deploy the cluster

Expected result: deployment is successful, environment passes OSTF checks

Actual result: deployment fails on primary controller

Here is the full task list performed on controller during deployment:

http://paste.openstack.org/show/485033/

As you can see swift wasn't installed before adding cronjob for re-balancing.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/272623

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/272623
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9552480b65f2061ec096c3d0fefdd6cf63d29f59
Submitter: Jenkins
Branch: master

commit 9552480b65f2061ec096c3d0fefdd6cf63d29f59
Author: Aleksandr Didenko <email address hidden>
Date: Tue Jan 26 17:23:13 2016 +0100

    Fix task dependencies for swift tasks

    After introducing separate tasks for primary-controller for task
    based deployment feature some swift tasks has wrong dependencies.
    This patch fixes them.

    Change-Id: I920d789396f11c92e2d76d3eaa9df12e5c582259
    Closes-bug: #1538193

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/272953

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/272953
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b1296d44b6b3dcfad938e6c486cd1f0088c65867
Submitter: Jenkins
Branch: stable/8.0

commit b1296d44b6b3dcfad938e6c486cd1f0088c65867
Author: Aleksandr Didenko <email address hidden>
Date: Tue Jan 26 17:23:13 2016 +0100

    Fix task dependencies for swift tasks

    After introducing separate tasks for primary-controller for task
    based deployment feature some swift tasks has wrong dependencies.
    This patch fixes them.

    Change-Id: I920d789396f11c92e2d76d3eaa9df12e5c582259
    Closes-bug: #1538193
    (cherry picked from commit 9552480b65f2061ec096c3d0fefdd6cf63d29f59)

[Tatyanka (tatyana-leontovich)]

verified 493 iso