can_connect function inside ec2.rb always return false

Bug #732953 reported by Juan L. Negron
28
This bug affects 3 people
Affects Status Importance Assigned to Milestone
facter (Ubuntu)
Fix Released
High
Adam Gandelman
Lucid
Fix Released
High
Unassigned
Maverick
Fix Released
High
Unassigned
Natty
Fix Released
High
Unassigned
Oneiric
Fix Released
Undecided
Unassigned
Precise
Fix Released
High
Adam Gandelman

Bug Description

Binary package hint: facter

the can_connect function in ec2.rb always return false.
I modified the code as follows and it seems to have fixed the issue:
Before (Line 9 in ec2.rb): Timeout::timeout(wait_sec) {open(ip, port)}
After: Timeout::timeout(wait_sec) {open("http://#{ip}:#{port}/latest/meta-data")}

After that change, the module seems to work properly in EC2 instances.

======
IMPACT:
 * Any puppet configuration that relies on EC2 related facts will fail entirely or fail to propagate the correct configuration on nodes running lucid, maverick, and natty.

ADDRESSED:
 * This was fixed upstream and in ubuntu as of oneiric. The proposed patch fixes argument handling in a test of the ec2 metadata service.

REPRODUCE:
 * Launch any AMI of an affected series, 'apt-get install facter' and 'facter | grep ec2' should report none of the ec2 related facts.

REGRESSION POTENTIAL:
 * The potential regression is minimal as the fix enables functionality that was previously broken.
======

Related branches

Revision history for this message
Juan L. Negron (negronjl) wrote :

The meta-data URL also seems to be wrong:
http://169.254.169.254/2008-02-01/meta-data
It should be:
http://169.254.169.254/latest/meta-data

Revision history for this message
Chuck Short (zulcss) wrote :

Juan,

Can you provide a patch please?

Thanks
chuck

Changed in facter (Ubuntu):
importance: Undecided → Low
status: New → Incomplete
Revision history for this message
Juan L. Negron (negronjl) wrote :

Here is the modified file.

-Juan

Revision history for this message
Juan L. Negron (negronjl) wrote : Re: [Bug 732953] Re: can_connect function inside ec2.rb always return false
  • ec2.rb Edit (822 bytes, application/x-ruby; name="ec2.rb")

I attached the modified file to the bug.

I am attaching to this email as well.

--
Thanks,

Juan

> Juan,
>
> Can you provide a patch please?
>
> Thanks
> chuck
>
> ** Changed in: facter (Ubuntu)
> Importance: Undecided => Low
>
> ** Changed in: facter (Ubuntu)
> Status: New => Incomplete

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package facter - 1.5.8-2ubuntu2

---------------
facter (1.5.8-2ubuntu2) natty; urgency=low

  * lib/facter/ec2.rb: Use the latest-meta api so it always
    does not return false. (LP: #732953)
 -- Chuck Short <email address hidden> Mon, 04 Apr 2011 11:34:30 -0400

Changed in facter (Ubuntu):
status: Incomplete → Fix Released
Revision history for this message
James D. Nurmi (james-abneptis) wrote :

This bug was not fixed, the part that needs fixing is the 'can_connect' function

Patch attached.

Revision history for this message
David Lowes (davidl-david) wrote :

the port number in the "can_connect" ruby function should be specified without quotation marks. The attached patch resolved the issue for me.

Changed in facter (Ubuntu):
assignee: nobody → David Lowes (davidl-david)
Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Confirming that this bug still exists in natty (facter 1.5.8-2ubuntu2) but has since been fixed upstream. The issue exists in the "can_connect?" method and has little to do with the actual URL elsewhere, though changing the meta-data URL to use "/latest" instead of "/2008-02-01" provides a few more additional facts about network interfaces, security group and host names (as of today, at least).

Changed in facter (Ubuntu):
status: Fix Released → Confirmed
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "repatch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

tags: added: patch
Revision history for this message
David Lowes (davidl-david) wrote : Re: [Bug 732953] Re: can_connect function inside ec2.rb always return false

Hi,
Is there any chance you can backport your fix to lucid or integrate my patch
into lucid?
Please advise me.
David Lowes

On Mon, Oct 17, 2011 at 11:44 PM, Adam Gandelman
<email address hidden>wrote:

> Confirming that this bug still exists in natty (facter 1.5.8-2ubuntu2)
> but has since been fixed upstream. The issue exists in the
> "can_connect?" method and has little to do with the actual URL
> elsewhere, though changing the meta-data URL to use "/latest" instead of
> "/2008-02-01" provides a few more additional facts about network
> interfaces, security group and host names (as of today, at least).
>
> ** Changed in: facter (Ubuntu)
> Status: Fix Released => Confirmed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/732953
>
> Title:
> can_connect function inside ec2.rb always return false
>
> Status in “facter” package in Ubuntu:
> Confirmed
>
> Bug description:
> Binary package hint: facter
>
> the can_connect function in ec2.rb always return false.
> I modified the code as follows and it seems to have fixed the issue:
> Before (Line 9 in ec2.rb): Timeout::timeout(wait_sec) {open(ip, port)}
> After: Timeout::timeout(wait_sec) {open("http://
> #{ip}:#{port}/latest/meta-data")}
>
> After that change, the module seems to work properly in EC2 instances.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/facter/+bug/732953/+subscriptions
>

Changed in facter (Ubuntu):
assignee: David Lowes (davidl-david) → Adam Gandelman (gandelman-a)
importance: Low → High
Changed in facter (Ubuntu):
status: Confirmed → In Progress
description: updated
Changed in facter (Ubuntu Lucid):
importance: Undecided → High
status: New → Triaged
Changed in facter (Ubuntu Maverick):
importance: Undecided → High
status: New → Triaged
Changed in facter (Ubuntu Natty):
importance: Undecided → High
status: New → Triaged
Changed in facter (Ubuntu):
status: In Progress → Fix Released
Changed in facter (Ubuntu Oneiric):
status: New → Fix Released
Revision history for this message
Adam Gandelman (gandelman-a) wrote :

The linked branches also resolve Bug #876130

Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Hello Juan, or anyone else affected,

Accepted facter into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in facter (Ubuntu Lucid):
status: Triaged → Fix Committed
tags: added: verification-needed
Changed in facter (Ubuntu Natty):
status: Triaged → Fix Committed
Revision history for this message
Martin Pitt (pitti) wrote :

Hello Juan, or anyone else affected,

Accepted facter into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Revision history for this message
Martin Pitt (pitti) wrote :

Hello Juan, or anyone else affected,

Accepted facter into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in facter (Ubuntu Maverick):
status: Triaged → Fix Committed
Revision history for this message
Jacob Helwig (jhelwig) wrote :

I can confirm that the packages in lucid-proposed, natty-proposed, and mverick-proposed fix this issue for me.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package facter - 1.5.6-2ubuntu2.2

---------------
facter (1.5.6-2ubuntu2.2) lucid-proposed; urgency=low

  * lib/facter/ec2.rb: Properly handle ip+port when testing connectivity
    of ec2 metadata service.(LP: #732953)
  * lib/facter/util/collection.rb: Backported fix from upstream commit
    2255abee7bdb9b6478ca228546e3d275dbac0ec3. Reload all facts if the
    requested fact is not found. Ensures consistency after facts
    have been cleared. (LP: #876130)
 -- Adam Gandelman <email address hidden> Tue, 25 Oct 2011 10:28:17 -0700

Changed in facter (Ubuntu Lucid):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package facter - 1.5.7-1ubuntu1.2

---------------
facter (1.5.7-1ubuntu1.2) maverick-proposed; urgency=low

  * lib/facter/ec2.rb: Properly handle ip+port when testing connectivity
    of ec2 metadata service.(LP: #732953)
  * lib/facter/util/collection.rb: Backported fix from upstream commit
    2255abee7bdb9b6478ca228546e3d275dbac0ec3. Reload all facts if the
    requested fact is not found. Ensures consistency after facts
    have been cleared. (LP: #876130)
 -- Adam Gandelman <email address hidden> Tue, 25 Oct 2011 10:43:54 -0700

Changed in facter (Ubuntu Maverick):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package facter - 1.5.8-2ubuntu2.1

---------------
facter (1.5.8-2ubuntu2.1) natty-proposed; urgency=low

  * debian/patches/reload_all_facts.patch: Reload all facts if the
    requested fact is not found. Ensures consistency after facts
    have been cleared. (LP: #876130)
  * debian/patches/fix_ec2_metadata_facts.patch: Properly handle ip+port
    when testing connectivity of ec2 metadata service.(LP: #732953)
 -- Adam Gandelman <email address hidden> Thu, 20 Oct 2011 10:40:35 -0700

Changed in facter (Ubuntu Natty):
status: Fix Committed → Fix Released
Revision history for this message
Paul Hirst (paul-hirst) wrote :

I think this fix may have just broken puppet running on a non EC2 instance. On real hardware we now get the following output from puppet:

root@uk-sofalive1:/usr/share/doc/facter# puppetd --test
notice: Ignoring --listen on onetime run
info: Retrieving plugin
err: Could not run Puppet configuration client: Could not retrieve local facts: execution expired

Running facter reveals:

root@uk-sofalive1:/usr/share/doc/facter# facter -p
/usr/lib/ruby/1.8/timeout.rb:60:in `open': execution expired (Timeout::Error)
 from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 from /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
 from /usr/lib/ruby/1.8/timeout.rb:93:in `timeout'
 from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
 from /usr/lib/ruby/1.8/net/http.rb:542:in `start'
 from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
 from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
 from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
 from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
 from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
 from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
 from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
 from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'
 from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
 from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
 from /usr/lib/ruby/1.8/facter/ec2.rb:33
 from /usr/lib/ruby/1.8/facter/util/loader.rb:72:in `load'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:72:in `load_file'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:38:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `each'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `each'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/collection.rb:94:in `load_all'
 from /usr/lib/ruby/1.8/facter.rb:91:in `to_hash'
 from /usr/bin/facter:138

Stracing this process reveals that's it's attempting to connect to 169.254.169.254 and failing when it times out:

connect(5, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("169.254.169.254")}, 16) = -1 EINPROGRESS (Operation now in progress)
select(7, [], [5], [5], {1, 976432}) = 0 (Timeout)
select(7, [], [5], [5], {0, 0}) = 0 (Timeout)
mmap(NULL, 167936, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb087093000
rt_sigprocmask(SIG_BLOCK, NULL, [PIPE], 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
select(7, [], [5], [5], {0, 0}) = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, NULL, [PIPE], 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
close(5) = 0

We've seen this on three of our servers now. We expect it to happen on all of them if we allow the facter package to update.

Revision history for this message
Ben Davis (bendavis78) wrote :

I'm currently getting this error on a Rackspace Cloud instance, using facter 1.5.8-2ubuntu2.1

# facter -p
/usr/lib/ruby/1.8/timeout.rb:64:in `open': execution expired (Timeout::Error)
 from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 from /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
 from /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
 from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
 from /usr/lib/ruby/1.8/net/http.rb:542:in `start'
 from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
 from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
 from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
 from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
 from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
 from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
 from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
 from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'
 from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
 from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
 from /usr/lib/ruby/1.8/facter/ec2.rb:33
 from /usr/lib/ruby/1.8/facter/util/loader.rb:73:in `load'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:73:in `load_file'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:38:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `each'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `each'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/collection.rb:94:in `load_all'
 from /usr/lib/ruby/1.8/facter.rb:95:in `to_hash'
 from /usr/bin/facter:137

Revision history for this message
Craig Miskell (3-crjig-7) wrote :

Confirmed here; same symptoms, breaking the puppet client.

Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Regression confirmed and currently being tracked in bug #885998.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers