I think this fix may have just broken puppet running on a non EC2 instance. On real hardware we now get the following output from puppet:
root@uk-sofalive1:/usr/share/doc/facter# puppetd --test
notice: Ignoring --listen on onetime run
info: Retrieving plugin
err: Could not run Puppet configuration client: Could not retrieve local facts: execution expired
Running facter reveals:
root@uk-sofalive1:/usr/share/doc/facter# facter -p
/usr/lib/ruby/1.8/timeout.rb:60:in `open': execution expired (Timeout::Error)
from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
from /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
from /usr/lib/ruby/1.8/timeout.rb:93:in `timeout'
from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
from /usr/lib/ruby/1.8/net/http.rb:542:in `start'
from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'
from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
from /usr/lib/ruby/1.8/facter/ec2.rb:33
from /usr/lib/ruby/1.8/facter/util/loader.rb:72:in `load'
from /usr/lib/ruby/1.8/facter/util/loader.rb:72:in `load_file'
from /usr/lib/ruby/1.8/facter/util/loader.rb:38:in `load_all'
from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `each'
from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `load_all'
from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `each'
from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `load_all'
from /usr/lib/ruby/1.8/facter/util/collection.rb:94:in `load_all'
from /usr/lib/ruby/1.8/facter.rb:91:in `to_hash'
from /usr/bin/facter:138
Stracing this process reveals that's it's attempting to connect to 169.254.169.254 and failing when it times out:
I think this fix may have just broken puppet running on a non EC2 instance. On real hardware we now get the following output from puppet:
root@uk- sofalive1: /usr/share/ doc/facter# puppetd --test
notice: Ignoring --listen on onetime run
info: Retrieving plugin
err: Could not run Puppet configuration client: Could not retrieve local facts: execution expired
Running facter reveals:
root@uk- sofalive1: /usr/share/ doc/facter# facter -p ruby/1. 8/timeout. rb:60:in `open': execution expired (Timeout::Error) ruby/1. 8/net/http. rb:560: in `connect' ruby/1. 8/timeout. rb:53:in `timeout' ruby/1. 8/timeout. rb:93:in `timeout' ruby/1. 8/net/http. rb:560: in `connect' ruby/1. 8/net/http. rb:553: in `do_start' ruby/1. 8/net/http. rb:542: in `start' ruby/1. 8/open- uri.rb: 242:in `open_http' ruby/1. 8/open- uri.rb: 616:in `buffer_open' ruby/1. 8/open- uri.rb: 164:in `open_loop' ruby/1. 8/open- uri.rb: 162:in `catch' ruby/1. 8/open- uri.rb: 162:in `open_loop' ruby/1. 8/open- uri.rb: 132:in `open_uri' ruby/1. 8/open- uri.rb: 518:in `open' ruby/1. 8/open- uri.rb: 30:in `open' ruby/1. 8/facter/ ec2.rb: 10:in `can_connect?' ruby/1. 8/facter/ ec2.rb: 10:in `can_connect?' ruby/1. 8/facter/ ec2.rb: 33 ruby/1. 8/facter/ util/loader. rb:72:in `load' ruby/1. 8/facter/ util/loader. rb:72:in `load_file' ruby/1. 8/facter/ util/loader. rb:38:in `load_all' ruby/1. 8/facter/ util/loader. rb:33:in `each' ruby/1. 8/facter/ util/loader. rb:33:in `load_all' ruby/1. 8/facter/ util/loader. rb:30:in `each' ruby/1. 8/facter/ util/loader. rb:30:in `load_all' ruby/1. 8/facter/ util/collection .rb:94: in `load_all' ruby/1. 8/facter. rb:91:in `to_hash'
/usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/lib/
from /usr/bin/facter:138
Stracing this process reveals that's it's attempting to connect to 169.254.169.254 and failing when it times out:
connect(5, {sa_family=AF_INET, sin_port=htons(80), sin_addr= inet_addr( "169.254. 169.254" )}, 16) = -1 EINPROGRESS (Operation now in progress) PROT_WRITE, MAP_PRIVATE| MAP_ANONYMOUS, -1, 0) = 0x7fb087093000 SIG_BLOCK, NULL, [PIPE], 8) = 0 SIG_SETMASK, [PIPE], NULL, 8) = 0 SIG_BLOCK, NULL, [PIPE], 8) = 0 SIG_SETMASK, [PIPE], NULL, 8) = 0 SIG_SETMASK, [PIPE], NULL, 8) = 0 SIG_SETMASK, [PIPE], NULL, 8) = 0
select(7, [], [5], [5], {1, 976432}) = 0 (Timeout)
select(7, [], [5], [5], {0, 0}) = 0 (Timeout)
mmap(NULL, 167936, PROT_READ|
rt_sigprocmask(
rt_sigprocmask(
select(7, [], [5], [5], {0, 0}) = 0 (Timeout)
rt_sigprocmask(
rt_sigprocmask(
rt_sigprocmask(
rt_sigprocmask(
close(5) = 0
We've seen this on three of our servers now. We expect it to happen on all of them if we allow the facter package to update.