can_connect function inside ec2.rb always return false

[Juan L. Negron]

Binary package hint: facter

the can_connect function in ec2.rb always return false.
I modified the code as follows and it seems to have fixed the issue:
Before (Line 9 in ec2.rb): Timeout::timeout(wait_sec) {open(ip, port)}
After: Timeout::timeout(wait_sec) {open("http://#{ip}:#{port}/latest/meta-data")}

After that change, the module seems to work properly in EC2 instances.

======
IMPACT:
 * Any puppet configuration that relies on EC2 related facts will fail entirely or fail to propagate the correct configuration on nodes running lucid, maverick, and natty.

ADDRESSED:
 * This was fixed upstream and in ubuntu as of oneiric. The proposed patch fixes argument handling in a test of the ec2 metadata service.

REPRODUCE:
 * Launch any AMI of an affected series, 'apt-get install facter' and 'facter | grep ec2' should report none of the ec2 related facts.

REGRESSION POTENTIAL:
 * The potential regression is minimal as the fix enables functionality that was previously broken.
======

[Juan L. Negron (negronjl)]

The meta-data URL also seems to be wrong:
http://169.254.169.254/2008-02-01/meta-data
It should be:
http://169.254.169.254/latest/meta-data

[Chuck Short (zulcss)]

Juan,

Can you provide a patch please?

Thanks
chuck

[Juan L. Negron (negronjl)]

Here is the modified file.

-Juan

[Juan L. Negron (negronjl)]

I attached the modified file to the bug.

I am attaching to this email as well.

--
Thanks,

Juan

> Juan,
>
> Can you provide a patch please?
>
> Thanks
> chuck
>
> ** Changed in: facter (Ubuntu)
> Importance: Undecided => Low
>
> ** Changed in: facter (Ubuntu)
> Status: New => Incomplete

[Launchpad Janitor (janitor)]

This bug was fixed in the package facter - 1.5.8-2ubuntu2

---------------
facter (1.5.8-2ubuntu2) natty; urgency=low

  * lib/facter/ec2.rb: Use the latest-meta api so it always
    does not return false. (LP: #732953)
 -- Chuck Short <email address hidden> Mon, 04 Apr 2011 11:34:30 -0400

[James D. Nurmi (james-abneptis)]

This bug was not fixed, the part that needs fixing is the 'can_connect' function

Patch attached.

[David Lowes (davidl-david)]

the port number in the "can_connect" ruby function should be specified without quotation marks. The attached patch resolved the issue for me.

[Adam Gandelman (gandelman-a)]

Confirming that this bug still exists in natty (facter 1.5.8-2ubuntu2) but has since been fixed upstream. The issue exists in the "can_connect?" method and has little to do with the actual URL elsewhere, though changing the meta-data URL to use "/latest" instead of "/2008-02-01" provides a few more additional facts about network interfaces, security group and host names (as of today, at least).

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "repatch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[David Lowes (davidl-david)]

Hi,
Is there any chance you can backport your fix to lucid or integrate my patch
into lucid?
Please advise me.
David Lowes

On Mon, Oct 17, 2011 at 11:44 PM, Adam Gandelman
<email address hidden>wrote:

> Confirming that this bug still exists in natty (facter 1.5.8-2ubuntu2)
> but has since been fixed upstream. The issue exists in the
> "can_connect?" method and has little to do with the actual URL
> elsewhere, though changing the meta-data URL to use "/latest" instead of
> "/2008-02-01" provides a few more additional facts about network
> interfaces, security group and host names (as of today, at least).
>
> ** Changed in: facter (Ubuntu)
> Status: Fix Released => Confirmed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/732953
>
> Title:
> can_connect function inside ec2.rb always return false
>
> Status in “facter” package in Ubuntu:
> Confirmed
>
> Bug description:
> Binary package hint: facter
>
> the can_connect function in ec2.rb always return false.
> I modified the code as follows and it seems to have fixed the issue:
> Before (Line 9 in ec2.rb): Timeout::timeout(wait_sec) {open(ip, port)}
> After: Timeout::timeout(wait_sec) {open("http://
> #{ip}:#{port}/latest/meta-data")}
>
> After that change, the module seems to work properly in EC2 instances.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/facter/+bug/732953/+subscriptions
>

[Adam Gandelman (gandelman-a)]

The linked branches also resolve Bug #876130

[Martin Pitt (pitti)]

Hello Juan, or anyone else affected,

Accepted facter into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Martin Pitt (pitti)]

Hello Juan, or anyone else affected,

Accepted facter into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Martin Pitt (pitti)]

Hello Juan, or anyone else affected,

Accepted facter into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Jacob Helwig (jhelwig)]

I can confirm that the packages in lucid-proposed, natty-proposed, and mverick-proposed fix this issue for me.

[Launchpad Janitor (janitor)]

This bug was fixed in the package facter - 1.5.6-2ubuntu2.2

---------------
facter (1.5.6-2ubuntu2.2) lucid-proposed; urgency=low

  * lib/facter/ec2.rb: Properly handle ip+port when testing connectivity
    of ec2 metadata service.(LP: #732953)
  * lib/facter/util/collection.rb: Backported fix from upstream commit
    2255abee7bdb9b6478ca228546e3d275dbac0ec3. Reload all facts if the
    requested fact is not found. Ensures consistency after facts
    have been cleared. (LP: #876130)
 -- Adam Gandelman <email address hidden> Tue, 25 Oct 2011 10:28:17 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package facter - 1.5.7-1ubuntu1.2

---------------
facter (1.5.7-1ubuntu1.2) maverick-proposed; urgency=low

  * lib/facter/ec2.rb: Properly handle ip+port when testing connectivity
    of ec2 metadata service.(LP: #732953)
  * lib/facter/util/collection.rb: Backported fix from upstream commit
    2255abee7bdb9b6478ca228546e3d275dbac0ec3. Reload all facts if the
    requested fact is not found. Ensures consistency after facts
    have been cleared. (LP: #876130)
 -- Adam Gandelman <email address hidden> Tue, 25 Oct 2011 10:43:54 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package facter - 1.5.8-2ubuntu2.1

---------------
facter (1.5.8-2ubuntu2.1) natty-proposed; urgency=low

  * debian/patches/reload_all_facts.patch: Reload all facts if the
    requested fact is not found. Ensures consistency after facts
    have been cleared. (LP: #876130)
  * debian/patches/fix_ec2_metadata_facts.patch: Properly handle ip+port
    when testing connectivity of ec2 metadata service.(LP: #732953)
 -- Adam Gandelman <email address hidden> Thu, 20 Oct 2011 10:40:35 -0700

[Paul Hirst (paul-hirst)]

I think this fix may have just broken puppet running on a non EC2 instance. On real hardware we now get the following output from puppet:

root@uk-sofalive1:/usr/share/doc/facter# puppetd --test
notice: Ignoring --listen on onetime run
info: Retrieving plugin
err: Could not run Puppet configuration client: Could not retrieve local facts: execution expired

Running facter reveals:

root@uk-sofalive1:/usr/share/doc/facter# facter -p
/usr/lib/ruby/1.8/timeout.rb:60:in `open': execution expired (Timeout::Error)
 from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 from /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
 from /usr/lib/ruby/1.8/timeout.rb:93:in `timeout'
 from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
 from /usr/lib/ruby/1.8/net/http.rb:542:in `start'
 from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
 from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
 from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
 from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
 from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
 from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
 from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
 from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'
 from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
 from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
 from /usr/lib/ruby/1.8/facter/ec2.rb:33
 from /usr/lib/ruby/1.8/facter/util/loader.rb:72:in `load'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:72:in `load_file'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:38:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `each'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `each'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/collection.rb:94:in `load_all'
 from /usr/lib/ruby/1.8/facter.rb:91:in `to_hash'
 from /usr/bin/facter:138

Stracing this process reveals that's it's attempting to connect to 169.254.169.254 and failing when it times out:

connect(5, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("169.254.169.254")}, 16) = -1 EINPROGRESS (Operation now in progress)
select(7, [], [5], [5], {1, 976432}) = 0 (Timeout)
select(7, [], [5], [5], {0, 0}) = 0 (Timeout)
mmap(NULL, 167936, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb087093000
rt_sigprocmask(SIG_BLOCK, NULL, [PIPE], 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
select(7, [], [5], [5], {0, 0}) = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, NULL, [PIPE], 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
close(5) = 0

We've seen this on three of our servers now. We expect it to happen on all of them if we allow the facter package to update.

[Ben Davis (bendavis78)]

I'm currently getting this error on a Rackspace Cloud instance, using facter 1.5.8-2ubuntu2.1

# facter -p
/usr/lib/ruby/1.8/timeout.rb:64:in `open': execution expired (Timeout::Error)
 from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 from /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
 from /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
 from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
 from /usr/lib/ruby/1.8/net/http.rb:542:in `start'
 from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
 from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
 from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
 from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
 from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
 from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
 from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
 from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'
 from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
 from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
 from /usr/lib/ruby/1.8/facter/ec2.rb:33
 from /usr/lib/ruby/1.8/facter/util/loader.rb:73:in `load'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:73:in `load_file'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:38:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `each'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `each'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/collection.rb:94:in `load_all'
 from /usr/lib/ruby/1.8/facter.rb:95:in `to_hash'
 from /usr/bin/facter:137

[Craig Miskell (3-crjig-7)]

Confirmed here; same symptoms, breaking the puppet client.

[Adam Gandelman (gandelman-a)]

Regression confirmed and currently being tracked in bug #885998.