apt-get update failed in the first run and a lock file was not cleaned, blocking following chef reruns.
need a mechanism to clean lock files such as /var/lib/apt/lists/* /var/lib/dpkg/lock /var/cache/apt/archives/lock on exception.
Jun 10 11:52:07 host1.1 [2014-06-10T11:52:07-07:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Jun 10 11:52:07 host1.1 [2014-06-10T11:52:07-07:00] ERROR: execute[apt-get-update-build-essentials] (build-essential::debian line 22) had an error: Mixlib::ShellOut::CommandTimeout: command timed out:
Jun 10 11:52:07 host1.1 ---- Begin output of apt-get update ----
Jun 10 11:52:07 host1.1 STDOUT: Ign http://172.16.0.2 ppa Release.gpg
Jun 10 11:52:07 host1.1 Get:1 http://172.16.0.2 ppa Release [2238 B]
Jun 10 11:52:07 host1.1 Ign http://172.16.0.2 ppa/main TranslationIndex
Jun 10 11:52:07 host1.1 Get:2 http://172.16.0.2 ppa/main amd64 Packages [408 B]
Jun 10 11:52:07 host1.1 Ign http://172.16.0.2 ppa/main Translation-en
Jun 10 11:52:07 host1.1 Err http://archive.ubuntu.com precise Release.gpg
Jun 10 11:52:07 host1.1 Connection failed
Jun 10 11:52:07 host1.1 Err http://archive.ubuntu.com precise-security Release.gpg
Jun 10 11:52:07 host1.1 Connection failed
Jun 10 11:52:07 host1.1 Err http://archive.ubuntu.com precise-updates Release.gpg
Jun 10 11:52:07 host1.1 Connection failed
Jun 10 11:52:07 host1.1 Err http://archive.ubuntu.com precise-proposed Release.gpg
Jun 10 11:52:07 host1.1 Connection failed
Jun 10 11:52:07 host1.1 Err http://archive.ubuntu.com precise-backports Release.gpg
Jun 10 11:52:07 host1.1 Connection failed
Jun 10 11:52:07 host1.1 Ign http://archive.ubuntu.com precise Release
Jun 10 11:52:07 host1.1 Ign http://archive.ubuntu.com precise-security Release
Jun 10 11:52:07 host1.1 Ign http://archive.ubuntu.com precise-updates Release
Jun 10 11:52:07 host1.1 Ign http://archive.ubuntu.com precise-proposed Release
Jun 10 11:52:07 host1.1 Ign http://archive.ubuntu.com precise-backports Release
Jun 10 11:52:07 host1.1 STDERR:
Jun 10 11:52:07 host1.1 ---- End output of apt-get update ----
Jun 10 11:52:07 host1.1 [2014-06-10T11:52:07-07:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Jun 10 11:53:08 host1.1 [2014-06-10T11:53:08-07:00] INFO: Forking chef instance to converge...
Jun 10 11:53:08 host1.1 [2014-06-10T11:53:08-07:00] INFO: *** Chef 11.8.0 ***
Jun 10 11:53:08 host1.1 [2014-06-10T11:53:08-07:00] INFO: Chef-client pid: 1505
Jun 10 11:53:09 host1.1 [2014-06-10T11:53:09-07:00] INFO: Run List is [role[test-synclog], role[os-ops-database], role[os-ops-messaging], role[os-controller], role[os-compute-vncproxy], role[os-image], role[os-block-storage-worker], role[os-network], role[os-compute-worker]]
Jun 10 11:53:09 host1.1 [2014-06-10T11:53:09-07:00] INFO: Run List expands to [rsyslog::client, openstack-common, openstack-common::logging, openstack-ops-database::server, openstack-ops-database::openstack-db, openstack-ops-messaging::server, openstack-dashboard::server, openstack-identity::server, openstack-identity::registration, openstack-block-storage::api, openstack-block-storage::scheduler, openstack-compute::api-ec2, openstack-compute::api-os-compute, openstack-compute::nova-cert, openstack-compute::scheduler, openstack-compute::vncproxy, openstack-compute::conductor, openstack-compute::nova-setup, openstack-network::server, openstack-image::registry, openstack-image::api, openstack-block-storage::volume, openstack-network::openvswitch, openstack-network::l3_agent, openstack-network::dhcp_agent, openstack-network::metadata_agent, openstack-compute::compute]
Jun 10 11:53:09 host1.1 [2014-06-10T11:53:09-07:00] INFO: Starting Chef Run for host1.1
Jun 10 11:53:09 host1.1 [2014-06-10T11:53:09-07:00] INFO: Running start handlers
Jun 10 11:53:09 host1.1 [2014-06-10T11:53:09-07:00] INFO: Start handlers complete.
Jun 10 11:53:10 host1.1 [2014-06-10T11:53:09-07:00] INFO: HTTP Request Returned 404 Object Not Found:
Jun 10 11:53:10 host1.1 [2014-06-10T11:53:10-07:00] INFO: Loading cookbooks [apache2, apt, aws, build-essential, database, erlang, mysql, openssl, openstack-block-storage, openstack-common, openstack-compute, openstack-dashboard, openstack-identity, openstack-image, openstack-network, openstack-ops-database, openstack-ops-messaging, postgresql, python, rabbitmq, rsyslog, selinux, sysctl, xfs, yum]
Jun 10 11:53:12 host1.1 [2014-06-10T11:53:12-07:00] INFO: Processing execute[apt-get-update-build-essentials] action run (build-essential::debian line 22)
Jun 10 11:53:13 host1.1 [2014-06-10T11:53:12-07:00] ERROR: Running exception handlers
Jun 10 11:53:13 host1.1 [2014-06-10T11:53:12-07:00] ERROR: Exception handlers complete
Jun 10 11:53:13 host1.1 [2014-06-10T11:53:12-07:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Jun 10 11:53:13 host1.1 [2014-06-10T11:53:12-07:00] ERROR: execute[apt-get-update-build-essentials] (build-essential::debian line 22) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '100'
Jun 10 11:53:13 host1.1 ---- Begin output of apt-get update ----
Jun 10 11:53:13 host1.1 STDOUT:
Jun 10 11:53:13 host1.1 STDERR: E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
Jun 10 11:53:13 host1.1 E: Unable to lock directory /var/lib/apt/lists/
Jun 10 11:53:13 host1.1 ---- End output of apt-get update ----
Jun 10 11:53:13 host1.1 Ran apt-get update returned 100
Jun 10 11:53:13 host1.1 [2014-06-10T11:53:12-07:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit co
