can_connect function inside ec2.rb always return false

Bug #732953 reported by Juan L. Negron on 2011-03-10
28
This bug affects 3 people
Affects Status Importance Assigned to Milestone
facter (Ubuntu)
High
Adam Gandelman
Lucid
High
Unassigned
Maverick
High
Unassigned
Natty
High
Unassigned
Oneiric
Undecided
Unassigned
Precise
High
Adam Gandelman

Bug Description

Binary package hint: facter

the can_connect function in ec2.rb always return false.
I modified the code as follows and it seems to have fixed the issue:
Before (Line 9 in ec2.rb): Timeout::timeout(wait_sec) {open(ip, port)}
After: Timeout::timeout(wait_sec) {open("http://#{ip}:#{port}/latest/meta-data")}

After that change, the module seems to work properly in EC2 instances.

======
IMPACT:
 * Any puppet configuration that relies on EC2 related facts will fail entirely or fail to propagate the correct configuration on nodes running lucid, maverick, and natty.

ADDRESSED:
 * This was fixed upstream and in ubuntu as of oneiric. The proposed patch fixes argument handling in a test of the ec2 metadata service.

REPRODUCE:
 * Launch any AMI of an affected series, 'apt-get install facter' and 'facter | grep ec2' should report none of the ec2 related facts.

REGRESSION POTENTIAL:
 * The potential regression is minimal as the fix enables functionality that was previously broken.
======

Related branches

Juan L. Negron (negronjl) wrote :

The meta-data URL also seems to be wrong:
http://169.254.169.254/2008-02-01/meta-data
It should be:
http://169.254.169.254/latest/meta-data

Chuck Short (zulcss) wrote :

Juan,

Can you provide a patch please?

Thanks
chuck

Changed in facter (Ubuntu):
importance: Undecided → Low
status: New → Incomplete
Juan L. Negron (negronjl) wrote :

Here is the modified file.

-Juan

  • ec2.rb Edit (822 bytes, application/x-ruby; name="ec2.rb")

I attached the modified file to the bug.

I am attaching to this email as well.

--
Thanks,

Juan

> Juan,
>
> Can you provide a patch please?
>
> Thanks
> chuck
>
> ** Changed in: facter (Ubuntu)
> Importance: Undecided => Low
>
> ** Changed in: facter (Ubuntu)
> Status: New => Incomplete

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package facter - 1.5.8-2ubuntu2

---------------
facter (1.5.8-2ubuntu2) natty; urgency=low

  * lib/facter/ec2.rb: Use the latest-meta api so it always
    does not return false. (LP: #732953)
 -- Chuck Short <email address hidden> Mon, 04 Apr 2011 11:34:30 -0400

Changed in facter (Ubuntu):
status: Incomplete → Fix Released
James D. Nurmi (james-abneptis) wrote :

This bug was not fixed, the part that needs fixing is the 'can_connect' function

Patch attached.

David Lowes (davidl-david) wrote :

the port number in the "can_connect" ruby function should be specified without quotation marks. The attached patch resolved the issue for me.

Changed in facter (Ubuntu):
assignee: nobody → David Lowes (davidl-david)
Adam Gandelman (gandelman-a) wrote :

Confirming that this bug still exists in natty (facter 1.5.8-2ubuntu2) but has since been fixed upstream. The issue exists in the "can_connect?" method and has little to do with the actual URL elsewhere, though changing the meta-data URL to use "/latest" instead of "/2008-02-01" provides a few more additional facts about network interfaces, security group and host names (as of today, at least).

Changed in facter (Ubuntu):
status: Fix Released → Confirmed

The attachment "repatch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

tags: added: patch

Hi,
Is there any chance you can backport your fix to lucid or integrate my patch
into lucid?
Please advise me.
David Lowes

On Mon, Oct 17, 2011 at 11:44 PM, Adam Gandelman
<email address hidden>wrote:

> Confirming that this bug still exists in natty (facter 1.5.8-2ubuntu2)
> but has since been fixed upstream. The issue exists in the
> "can_connect?" method and has little to do with the actual URL
> elsewhere, though changing the meta-data URL to use "/latest" instead of
> "/2008-02-01" provides a few more additional facts about network
> interfaces, security group and host names (as of today, at least).
>
> ** Changed in: facter (Ubuntu)
> Status: Fix Released => Confirmed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/732953
>
> Title:
> can_connect function inside ec2.rb always return false
>
> Status in “facter” package in Ubuntu:
> Confirmed
>
> Bug description:
> Binary package hint: facter
>
> the can_connect function in ec2.rb always return false.
> I modified the code as follows and it seems to have fixed the issue:
> Before (Line 9 in ec2.rb): Timeout::timeout(wait_sec) {open(ip, port)}
> After: Timeout::timeout(wait_sec) {open("http://
> #{ip}:#{port}/latest/meta-data")}
>
> After that change, the module seems to work properly in EC2 instances.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/facter/+bug/732953/+subscriptions
>

Changed in facter (Ubuntu):
assignee: David Lowes (davidl-david) → Adam Gandelman (gandelman-a)
importance: Low → High
Changed in facter (Ubuntu):
status: Confirmed → In Progress
description: updated
Changed in facter (Ubuntu Lucid):
importance: Undecided → High
status: New → Triaged
Changed in facter (Ubuntu Maverick):
importance: Undecided → High
status: New → Triaged
Changed in facter (Ubuntu Natty):
importance: Undecided → High
status: New → Triaged
Changed in facter (Ubuntu):
status: In Progress → Fix Released
Changed in facter (Ubuntu Oneiric):
status: New → Fix Released
Adam Gandelman (gandelman-a) wrote :

The linked branches also resolve Bug #876130

Hello Juan, or anyone else affected,

Accepted facter into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in facter (Ubuntu Lucid):
status: Triaged → Fix Committed
tags: added: verification-needed
Changed in facter (Ubuntu Natty):
status: Triaged → Fix Committed
Martin Pitt (pitti) wrote :

Hello Juan, or anyone else affected,

Accepted facter into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Martin Pitt (pitti) wrote :

Hello Juan, or anyone else affected,

Accepted facter into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in facter (Ubuntu Maverick):
status: Triaged → Fix Committed
Jacob Helwig (jhelwig) wrote :

I can confirm that the packages in lucid-proposed, natty-proposed, and mverick-proposed fix this issue for me.

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package facter - 1.5.6-2ubuntu2.2

---------------
facter (1.5.6-2ubuntu2.2) lucid-proposed; urgency=low

  * lib/facter/ec2.rb: Properly handle ip+port when testing connectivity
    of ec2 metadata service.(LP: #732953)
  * lib/facter/util/collection.rb: Backported fix from upstream commit
    2255abee7bdb9b6478ca228546e3d275dbac0ec3. Reload all facts if the
    requested fact is not found. Ensures consistency after facts
    have been cleared. (LP: #876130)
 -- Adam Gandelman <email address hidden> Tue, 25 Oct 2011 10:28:17 -0700

Changed in facter (Ubuntu Lucid):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package facter - 1.5.7-1ubuntu1.2

---------------
facter (1.5.7-1ubuntu1.2) maverick-proposed; urgency=low

  * lib/facter/ec2.rb: Properly handle ip+port when testing connectivity
    of ec2 metadata service.(LP: #732953)
  * lib/facter/util/collection.rb: Backported fix from upstream commit
    2255abee7bdb9b6478ca228546e3d275dbac0ec3. Reload all facts if the
    requested fact is not found. Ensures consistency after facts
    have been cleared. (LP: #876130)
 -- Adam Gandelman <email address hidden> Tue, 25 Oct 2011 10:43:54 -0700

Changed in facter (Ubuntu Maverick):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package facter - 1.5.8-2ubuntu2.1

---------------
facter (1.5.8-2ubuntu2.1) natty-proposed; urgency=low

  * debian/patches/reload_all_facts.patch: Reload all facts if the
    requested fact is not found. Ensures consistency after facts
    have been cleared. (LP: #876130)
  * debian/patches/fix_ec2_metadata_facts.patch: Properly handle ip+port
    when testing connectivity of ec2 metadata service.(LP: #732953)
 -- Adam Gandelman <email address hidden> Thu, 20 Oct 2011 10:40:35 -0700

Changed in facter (Ubuntu Natty):
status: Fix Committed → Fix Released
Paul Hirst (paul-hirst) wrote :

I think this fix may have just broken puppet running on a non EC2 instance. On real hardware we now get the following output from puppet:

root@uk-sofalive1:/usr/share/doc/facter# puppetd --test
notice: Ignoring --listen on onetime run
info: Retrieving plugin
err: Could not run Puppet configuration client: Could not retrieve local facts: execution expired

Running facter reveals:

root@uk-sofalive1:/usr/share/doc/facter# facter -p
/usr/lib/ruby/1.8/timeout.rb:60:in `open': execution expired (Timeout::Error)
 from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 from /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
 from /usr/lib/ruby/1.8/timeout.rb:93:in `timeout'
 from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
 from /usr/lib/ruby/1.8/net/http.rb:542:in `start'
 from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
 from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
 from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
 from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
 from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
 from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
 from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
 from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'
 from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
 from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
 from /usr/lib/ruby/1.8/facter/ec2.rb:33
 from /usr/lib/ruby/1.8/facter/util/loader.rb:72:in `load'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:72:in `load_file'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:38:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `each'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `each'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/collection.rb:94:in `load_all'
 from /usr/lib/ruby/1.8/facter.rb:91:in `to_hash'
 from /usr/bin/facter:138

Stracing this process reveals that's it's attempting to connect to 169.254.169.254 and failing when it times out:

connect(5, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("169.254.169.254")}, 16) = -1 EINPROGRESS (Operation now in progress)
select(7, [], [5], [5], {1, 976432}) = 0 (Timeout)
select(7, [], [5], [5], {0, 0}) = 0 (Timeout)
mmap(NULL, 167936, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb087093000
rt_sigprocmask(SIG_BLOCK, NULL, [PIPE], 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
select(7, [], [5], [5], {0, 0}) = 0 (Timeout)
rt_sigprocmask(SIG_BLOCK, NULL, [PIPE], 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
close(5) = 0

We've seen this on three of our servers now. We expect it to happen on all of them if we allow the facter package to update.

Ben Davis (bendavis78) wrote :

I'm currently getting this error on a Rackspace Cloud instance, using facter 1.5.8-2ubuntu2.1

# facter -p
/usr/lib/ruby/1.8/timeout.rb:64:in `open': execution expired (Timeout::Error)
 from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 from /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
 from /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
 from /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 from /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
 from /usr/lib/ruby/1.8/net/http.rb:542:in `start'
 from /usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
 from /usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
 from /usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
 from /usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
 from /usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
 from /usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
 from /usr/lib/ruby/1.8/open-uri.rb:518:in `open'
 from /usr/lib/ruby/1.8/open-uri.rb:30:in `open'
 from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
 from /usr/lib/ruby/1.8/facter/ec2.rb:10:in `can_connect?'
 from /usr/lib/ruby/1.8/facter/ec2.rb:33
 from /usr/lib/ruby/1.8/facter/util/loader.rb:73:in `load'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:73:in `load_file'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:38:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `each'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:33:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `each'
 from /usr/lib/ruby/1.8/facter/util/loader.rb:30:in `load_all'
 from /usr/lib/ruby/1.8/facter/util/collection.rb:94:in `load_all'
 from /usr/lib/ruby/1.8/facter.rb:95:in `to_hash'
 from /usr/bin/facter:137

Craig Miskell (3-crjig-7) wrote :

Confirmed here; same symptoms, breaking the puppet client.

Adam Gandelman (gandelman-a) wrote :

Regression confirmed and currently being tracked in bug #885998.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers