Eucalyptus Loses Public IP Address

Bug #455625 reported by Joseph Salisbury on 2009-10-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Eucalyptus
Invalid
Undecided
Unassigned
eucalyptus (Ubuntu)
Medium
Unassigned

Bug Description

I started an instance, which was assigned an IP Address from the cluster DHCP. However, after a couple of days, the instance was no longer using the IP address from DHCP, and would only respond to the private IP Address.

This is what euca-describe-instance reported initially:

$ euca-describe-instances
RESERVATION r-32CF078D admin default
INSTANCE i-47C70906 emi-230B11AE 172.23.46.78 172.19.1.2 running mykey 0 c1.medium 1970-01-01T00:00:00.328Z ubuntucluster1 eki-405016BE eri-23A51639

After a few days the public IP Address changed to the private IP address:

RESERVATION r-32CF078D admin default
INSTANCE i-47C70906 emi-230B11AE 172.19.1.2 172.19.1.2 running mykey 0 c1.medium 1970-01-01T00:00:00.328Z ubuntucluster1 eki-405016BE eri-23A51639

The instance is still running, and I can log in to it via ssh using the private IP address.

I am running the following eucalyptus package version: 1.6~bzr931-0ubuntu3

Thierry Carrez (ttx) wrote :

Maybe some issue with the DHCP lease not being renewed. Takes some time to reproduce, unfortunately...
Joseph: would there be anything in the logs (host side or guest side) about this ?

Changed in eucalyptus (Ubuntu):
importance: Undecided → High
status: New → Incomplete
tags: added: eucalyptus
Joseph Salisbury (jsalisbury) wrote :

I'm reviewing the logs now to see if there are additional details.

I think the max-lease-time is running out which is defined in /etc/dhcp3/dhcpd.conf. The two configurable settings are:

default-lease-time 600;
max-lease-time 7200;

7200 is three days, which is the right about when my public IP address went away.

I restarted the instance and a new public IP address was assigned. So I expect it would expire after the max-lease-time again.

I could try a test to reproduce faster by lowering default-lease-time and max-lease-time.

The issue seems to be that the instance does not get a new IP address when the existing lease expires. I will try to see if the logs show that an attempt for a new IP address was made but failed. Or maybe an attempt for a new IP address was never made.

Thierry Carrez (ttx) wrote :

IIUC the public IP address is not handed by DHCP but by Eucalyptus itself, from a pool defined at VNET_PUBLICIPS. That doesn't mean there isn't a timeout involved in that mechanism as well.

Matt Zimmerman (mdz) wrote :

Assigning to nurmi per:

<nurmi> 16:42:24> ttx: We're going to be looking into 455625 asap

Changed in eucalyptus (Ubuntu):
assignee: nobody → Daniel Nurmi (nurmi)
Matt Zimmerman (mdz) wrote :

Possibly caused by bug 454405?

Dustin Kirkland  (kirkland) wrote :

Looks like I just saw this happen here...

Every 5.0s: euca-describe-instances Thu Oct 22 10:24:21 2009

RESERVATION r-46C30944 admin default
INSTANCE i-45EE07E9 emi-E60E17EC 0.0.0.0 0.0.0.0 running mykey 0 m1.small 1970-01-01T00:00:
00.762Z canyonedge eki-4E481CFC eri-2C271C72
RESERVATION r-37CB0607 admin default
INSTANCE i-28EB0628 emi-E60E17EC 192.168.1.30 172.19.1.2 running mykey 0 c1.medium 2
009-10-22T15:21:28.09Z canyonedge eki-4E481CFC eri-2C271C72
RESERVATION r-3EB8063A admin default
INSTANCE i-30D20671 emi-E60E17EC 0.0.0.0 0.0.0.0 running mykey 0 m1.small 1970-01-01T00:00:
00.762Z canyonedge eki-4E481CFC eri-2C271C72

Changed in eucalyptus (Ubuntu):
status: Incomplete → Confirmed
Thierry Carrez (ttx) wrote :

Dustin: sounds like yours is slightly different, you end up with 0.0.0.0 rather than the private IP, and you hit it after one hour, rather than 3 days. I'd suggest opening a separate bug, if you can reproduce it. We'll merge the bugs if they happen to have the same root cause.

Keeping this one incomplete until someone reproduces the same symptoms

Changed in eucalyptus (Ubuntu):
status: Confirmed → Incomplete
Joseph Salisbury (jsalisbury) wrote :

I restarted my instance on Monday after the public IP address changed. The initial issue happened after three days. However, it's now been four days, and I have not been able to reproduce it again. It now seems there is not a pre-determined amount of time before the IP address expires.

I'll investigate further and try to reproduce.

Thierry Carrez (ttx) wrote :

Downgrading to Medium since it cannot be easily reproduced

Changed in eucalyptus (Ubuntu):
importance: High → Medium
Joseph Salisbury (jsalisbury) wrote :

Looks like I just reproduced it after seven days:

$ euca-describe-instances
RESERVATION r-39C7074C admin default
INSTANCE i-46910829 emi-230B11AE 172.19.1.2 172.19.1.2 running mykey 0 c1.medium 1970-01-01T00:00:00.813Z ubuntucluster1 eki-405016BE eri-23A51639

This may or may not be related, but I did an apt-get dist-upgrade this morning. I'll restart my instance again, and re-run the apt-get dist-upgrade and see if this is what triggers it.

Joseph Salisbury (jsalisbury) wrote :

One other thing to note. The following is the output of euca-describe-instances from this morning. Before the issue was reproduced:

$ euca-describe-instances
RESERVATION r-39C7074C admin default
INSTANCE i-46910829 emi-230B11AE 172.23.46.78 172.19.1.2 running mykey 0 c1.medium 2009-10-19T20:23:53.744Z ubuntucluster1 eki-405016BE eri-23A51639

The 14th column(2009-10-19T20:23:53.744Z) seems like an instance start date of some sort. This date changes to a 1970 date after the issue is reproduced(1970-01-01T00:00:00.813Z):

$ euca-describe-instances
RESERVATION r-39C7074C admin default
INSTANCE i-46910829 emi-230B11AE 172.19.1.2 172.19.1.2 running mykey 0 c1.medium 1970-01-01T00:00:00.813Z ubuntucluster1 eki-405016BE eri-23A51639

Changed in eucalyptus:
status: New → Invalid
Thierry Carrez (ttx) wrote :

Joseph: how is the "cluster controller" main IP address assigned ? Is it DHCP ? If yes, could there have been an issue at the end of the DHCP lease (IP address change after a multiple of 3 days ??)

Joseph Salisbury (jsalisbury) wrote :

It looks like the problem has not happened again in a while. I've had an instance running for almost a month now, and the IP Address has not been lost:

$ euca-describe-instances
RESERVATION r-44510791 admin default
INSTANCE i-4A870825 emi-230B11AE 172.23.46.78 172.19.1.2 running mykey 0 c1.medium 2009-11-09T16:29:40.139Z ubuntucluster1 eki-405016BE eri-23A51639

The only way I can reproduce the issue is to run an apt-get dist-upgrade while an instance is up and running AND there are updates to install.

Thierry Carrez (ttx) wrote :

Hmm... It could be a duplicate of bug 460089 (solved in 7.3, in karmic-updates now). Some update would trigger a cluster controller restart (either eucalyptus-cc update or some reboot) and network status would be lost.

I'll mark as invalid, please reopen if you can reproduce with the latest (and the procedure followed to do so :)

Changed in eucalyptus (Ubuntu):
assignee: Daniel Nurmi (nurmi) → nobody
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers