Smoke nova failed with deployment error

[Nastya Urlapova]

{
build_id: "2015-05-26_05-51-55",
build_number: "467",
api: "1.0",
fuel-library_sha: "64cb2844b6c508778105cb2e3c8aa90e0c38bc8a",
nailgun_sha: "61ef0edfbfe0c457265a62f0eab05af634ec3b91",
feature_groups: [
"mirantis"
],
openstack_version: "2014.2.2-6.1",
production: "docker",
python-fuelclient_sha: "e19f1b65792f84c4a18b5a9473f85ef3ba172fce",
astute_sha: "0bd72c72369e743376864e8e8dabfe873d40450a",
fuel-ostf_sha: "87819878bc0ca572900e1f6933d9b99e666d6f62",
release: "6.1",
fuelmain_sha: "5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93"
}

Scenario:
Deploy cluster in HA mode (one controller) with flat nova-network
        Scenario:
            1. Create cluster in HA mode with 1 controller
            2. Add 1 node with controller role
            3. Add 1 node with compute role
            4. Deploy the cluster
            5. Validate cluster was set up correctly, there are no dead
            services, there are no errors in logs
            6. Verify networks
            7. Verify network configuration on controller
            8. Run OSTF

Deployment has failed. Method granular_deploy. Failed to execute hook 'puppet'
---
priority: 100
fail_on_error: true
type: puppet
uids:
- '1'
parameters:
  puppet_modules: "/etc/puppet/modules"
  puppet_manifest: "/etc/puppet/modules/osnailyfacter/modular/netconfig/configure_default_route.pp"
  timeout: 3600
  cwd: "/"
.
Inspect Astute logs for the details

[Oleksiy Molchanov (omolchanov)]

Nailgun team, can you check?

configure_default_route.pp was executed successfully in few seconds. Also it didn't make any changes to the system.

[Vladimir Kuklin (vkuklin)]

According to the logs there is an astute error with detection of puppet run:

2015-05-26T07:25:03 debug: [647] Waiting for puppet to finish deployment on all
                           nodes (timeout = 5400 sec)...
2015-05-26T07:25:03 debug: [647] Waiting for puppet to finish deployment on node 1 (timeout = 5400 sec)...
2015-05-26T07:25:03 debug: [647] 5bb0a9ae-b42a-4e33-a0f1-97f5202f755b: MC agent 'puppetd', method 'last_run_summary', results: {:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=>{:idling=>0, :status=>"stopped", :stopped=>1, :running=>0, :time=>{"schedule"=>0.001146, "config_retrieval"=>0.080272, "last_run"=>1432625101, "total"=>0.081515, "filebucket"=>9.7e-05}, :resources=>{"changed"=>0, "skipped"=>0, "failed"=>0, "restarted"=>0, "out_of_sync"=>0, "total"=>7, "scheduled"=>0, "failed_to_restart"=>0}, :lastrun=>1432625101, :runtime=>2, :changes=>{"total"=>0}, :events=>{"total"=>0, "failure"=>0, "success"=>0}, :version=>{"puppet"=>"3.4.2", "config"=>1432625096}, :output=>"Currently stopped; last completed run 2 seconds ago", :enabled=>1}}
2015-05-26T07:25:03 debug: [647] 5bb0a9ae-b42a-4e33-a0f1-97f5202f755b: MC agent 'puppetd', method 'last_run_summary', results: {:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=>{:idling=>0, :status=>"stopped", :stopped=>1, :running=>0, :time=>{"schedule"=>0.001146, "config_retrieval"=>0.080272, "last_run"=>1432625101, "total"=>0.081515, "filebucket"=>9.7e-05}, :resources=>{"changed"=>0, "skipped"=>0, "failed"=>0, "restarted"=>0, "out_of_sync"=>0, "total"=>7, "scheduled"=>0, "failed_to_restart"=>0}, :lastrun=>1432625101, :runtime=>2, :changes=>{"total"=>0}, :events=>{"total"=>0, "failure"=>0, "success"=>0}, :version=>{"puppet"=>"3.4.2", "config"=>1432625096}, :output=>"Currently stopped; last completed run 2 seconds ago", :enabled=>1}}
2015-05-26T07:25:03 debug: [647] 5bb0a9ae-b42a-4e33-a0f1-97f5202f755b: MC agent 'puppetd', method 'runonce', results: {:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=>{:idling=>0, :status=>"stopped", :stopped=>1, :running=>0, :lastrun=>1432625101, :runtime=>2, :enabled=>1, :output=>"Called /usr/sbin/daemonize -a -l /tmp/puppetd.lock -p /tmp/puppetd.lock -c / /usr/bin/puppet apply /etc/puppet/modules/osnailyfacter/modular/netconfig/configure_default_route.pp --modulepath=/etc/puppet/modules --logdest syslog --trace --no-report --debug --evaltrace --logdest /var/log/puppet.log, Currently stopped; last completed run 2 seconds ago"}}
2015-05-26T07:25:04 debug: [647] 5bb0a9ae-b42a-4e33-a0f1-97f5202f755b: MC agent 'puppetd', method 'last_run_summary', results: {:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=>{:idling=>0, :status=>"stopped", :stopped=>1, :running=>0, :time=>{"schedule"=>0.001146, "config_retrieval"=>0.080272, "last_run"=>1432625101, "total"=>0.081515, "filebucket"=>9.7e-05}, :resources=>{"changed"=>0, "skipped"=>0, "failed"=>0, "restarted"=>0, "out_of_sync"=>0, "total"=>7, "scheduled"=>0, "failed_to_restart"=>0}, :lastrun=>1432625101, :runtime=>3, :changes=>{"total"=>0}, :events=>{"total"=>0, "failure"=>0, "success"=>0}, :version=>{"puppet"=>"3.4.2", "config"=>1432625096}, :output=>"Currently stopped; last c...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/185915

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/185915
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=d579d95dbfda3ccc19dd52387417bbadd8cf48b9
Submitter: Jenkins
Branch: master

commit d579d95dbfda3ccc19dd52387417bbadd8cf48b9
Author: Łukasz Oleś <email address hidden>
Date: Wed May 27 12:06:54 2015 +0200

    Wait some time before checking puppet state

    This tries to fix really strange bug. Puppet finished
    its run without any error but for some reason last_run_summary.yaml
    was not updated immediately.
    If puppet finishes and last_run_summary.yaml is not updated we assume
    that puppet crushed.

    Change-Id: I81cc57564089dfbe937dd7e447b65fe645c8bae9
    Closes-bug: #1458780

[Lope Beltran II (beltranlope)]

I'm still experiencing this in fuel 6.1 using CentOS. Already submitted a bug report in a separate thread before finding this.