Eucalyptus Loses Public IP Address

[Joseph Salisbury]

I started an instance, which was assigned an IP Address from the cluster DHCP. However, after a couple of days, the instance was no longer using the IP address from DHCP, and would only respond to the private IP Address.

This is what euca-describe-instance reported initially:

$ euca-describe-instances
RESERVATION r-32CF078D admin default
INSTANCE i-47C70906 emi-230B11AE 172.23.46.78 172.19.1.2 running mykey 0 c1.medium 1970-01-01T00:00:00.328Z ubuntucluster1 eki-405016BE eri-23A51639

After a few days the public IP Address changed to the private IP address:

RESERVATION r-32CF078D admin default
INSTANCE i-47C70906 emi-230B11AE 172.19.1.2 172.19.1.2 running mykey 0 c1.medium 1970-01-01T00:00:00.328Z ubuntucluster1 eki-405016BE eri-23A51639

The instance is still running, and I can log in to it via ssh using the private IP address.

I am running the following eucalyptus package version: 1.6~bzr931-0ubuntu3

[Thierry Carrez (ttx)]

Maybe some issue with the DHCP lease not being renewed. Takes some time to reproduce, unfortunately...
Joseph: would there be anything in the logs (host side or guest side) about this ?

[Joseph Salisbury (jsalisbury)]

I'm reviewing the logs now to see if there are additional details.

I think the max-lease-time is running out which is defined in /etc/dhcp3/dhcpd.conf. The two configurable settings are:

default-lease-time 600;
max-lease-time 7200;

7200 is three days, which is the right about when my public IP address went away.

I restarted the instance and a new public IP address was assigned. So I expect it would expire after the max-lease-time again.

I could try a test to reproduce faster by lowering default-lease-time and max-lease-time.

The issue seems to be that the instance does not get a new IP address when the existing lease expires. I will try to see if the logs show that an attempt for a new IP address was made but failed. Or maybe an attempt for a new IP address was never made.

[Thierry Carrez (ttx)]

IIUC the public IP address is not handed by DHCP but by Eucalyptus itself, from a pool defined at VNET_PUBLICIPS. That doesn't mean there isn't a timeout involved in that mechanism as well.

[Matt Zimmerman (mdz)]

Assigning to nurmi per:

<nurmi> 16:42:24> ttx: We're going to be looking into 455625 asap

[Matt Zimmerman (mdz)]

Possibly caused by bug 454405?

[Dustin Kirkland  (kirkland)]

Looks like I just saw this happen here...

Every 5.0s: euca-describe-instances Thu Oct 22 10:24:21 2009

RESERVATION r-46C30944 admin default
INSTANCE i-45EE07E9 emi-E60E17EC 0.0.0.0 0.0.0.0 running mykey 0 m1.small 1970-01-01T00:00:
00.762Z canyonedge eki-4E481CFC eri-2C271C72
RESERVATION r-37CB0607 admin default
INSTANCE i-28EB0628 emi-E60E17EC 192.168.1.30 172.19.1.2 running mykey 0 c1.medium 2
009-10-22T15:21:28.09Z canyonedge eki-4E481CFC eri-2C271C72
RESERVATION r-3EB8063A admin default
INSTANCE i-30D20671 emi-E60E17EC 0.0.0.0 0.0.0.0 running mykey 0 m1.small 1970-01-01T00:00:
00.762Z canyonedge eki-4E481CFC eri-2C271C72

[Thierry Carrez (ttx)]

Dustin: sounds like yours is slightly different, you end up with 0.0.0.0 rather than the private IP, and you hit it after one hour, rather than 3 days. I'd suggest opening a separate bug, if you can reproduce it. We'll merge the bugs if they happen to have the same root cause.

Keeping this one incomplete until someone reproduces the same symptoms

[Joseph Salisbury (jsalisbury)]

I restarted my instance on Monday after the public IP address changed. The initial issue happened after three days. However, it's now been four days, and I have not been able to reproduce it again. It now seems there is not a pre-determined amount of time before the IP address expires.

I'll investigate further and try to reproduce.

[Thierry Carrez (ttx)]

Downgrading to Medium since it cannot be easily reproduced

[Joseph Salisbury (jsalisbury)]

Looks like I just reproduced it after seven days:

$ euca-describe-instances
RESERVATION r-39C7074C admin default
INSTANCE i-46910829 emi-230B11AE 172.19.1.2 172.19.1.2 running mykey 0 c1.medium 1970-01-01T00:00:00.813Z ubuntucluster1 eki-405016BE eri-23A51639

This may or may not be related, but I did an apt-get dist-upgrade this morning. I'll restart my instance again, and re-run the apt-get dist-upgrade and see if this is what triggers it.

[Joseph Salisbury (jsalisbury)]

One other thing to note. The following is the output of euca-describe-instances from this morning. Before the issue was reproduced:

$ euca-describe-instances
RESERVATION r-39C7074C admin default
INSTANCE i-46910829 emi-230B11AE 172.23.46.78 172.19.1.2 running mykey 0 c1.medium 2009-10-19T20:23:53.744Z ubuntucluster1 eki-405016BE eri-23A51639

The 14th column(2009-10-19T20:23:53.744Z) seems like an instance start date of some sort. This date changes to a 1970 date after the issue is reproduced(1970-01-01T00:00:00.813Z):

$ euca-describe-instances
RESERVATION r-39C7074C admin default
INSTANCE i-46910829 emi-230B11AE 172.19.1.2 172.19.1.2 running mykey 0 c1.medium 1970-01-01T00:00:00.813Z ubuntucluster1 eki-405016BE eri-23A51639

[Thierry Carrez (ttx)]

Joseph: how is the "cluster controller" main IP address assigned ? Is it DHCP ? If yes, could there have been an issue at the end of the DHCP lease (IP address change after a multiple of 3 days ??)

[Joseph Salisbury (jsalisbury)]

It looks like the problem has not happened again in a while. I've had an instance running for almost a month now, and the IP Address has not been lost:

$ euca-describe-instances
RESERVATION r-44510791 admin default
INSTANCE i-4A870825 emi-230B11AE 172.23.46.78 172.19.1.2 running mykey 0 c1.medium 2009-11-09T16:29:40.139Z ubuntucluster1 eki-405016BE eri-23A51639

The only way I can reproduce the issue is to run an apt-get dist-upgrade while an instance is up and running AND there are updates to install.

[Thierry Carrez (ttx)]

Hmm... It could be a duplicate of bug 460089 (solved in 7.3, in karmic-updates now). Some update would trigger a cluster controller restart (either eucalyptus-cc update or some reboot) and network status would be lost.

I'll mark as invalid, please reopen if you can reproduce with the latest (and the procedure followed to do so :)