10GB root partition for EBS boot AMIs on EC2

Bug #670161 reported by James Clemence on 2010-11-02
66
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Ubuntu on EC2
Medium
Scott Moser

Bug Description

Amazon's latest trial offer of a free 10Gb EBS backed EC2 micro instance can't be used with the default canonical instances which use 15Gb root partitions. Is there any need for the extra 5Gb, and if not, would it make sense to decrease its size to enhance uptake within the wider user community?

J

Eric Hammond (esh) on 2010-11-02
tags: added: ec2-images
Eric Hammond (esh) wrote :

I initially only supported this for the not-yet-released Ubuntu 11.04 Natty, but the more I think about it, the more I believe it would be good to reduce the EBS boot disk sizes for existing Ubuntu releases including Ubuntu 10.04 Lucid and Ubuntu 10.10 Maverick.

Reasons for doing this include:

+ 10GB still leaves a lot of spare room for normal system operations (software installation, logs, kernel upgrades, etc).

+ The recommended practice on EC2 is to put data (e.g., mysql files) on a separate EBS volume.

+ If users are already running 10.04 or 10.10 AMIs, they will not be affected if a new AMI is published at a slightly smaller size. The instances they are running and any new instances they run of the current AMIs will continue to be 15GB.

+ The reduced size only affects people who choose to run the newly published AMIs.

+ Anybody who wants to run the new 10GB EBS boot AMI with a larger boot disk, can do so easily:
   http://alestic.com/2009/12/ec2-ebs-boot-resize

+ It's not easy to reduce the boot disk size from the one published in the AMI, so this cuts out a bit of the bloat for people who want to run smaller EBS boot disks anyway.

+ It's about time for new AMIs to be published for older versions of Ubuntu anyways to pick up the latest software updates.

+ This would let Ubuntu promote usage of Ubuntu with the AWS free tier using the word "free".

summary: - 10GB AMI root partition.
+ 10GB root partition for EBS boot AMIs on EC2
Eric Hammond (esh) wrote :

+ This would make Ubuntu EBS boot AMI sizes consistent with the new de-facto standard from Amazon. This may benefit Ubuntu and Ubuntu users on EC2 in the future as well.

James Clemence (jvc26) wrote :

Does this mean this could be implemented sometime in the near future? (I assume the AMI release isn't tied to a particular date if it is simply an update of the images?)

J

Scott Moser (smoser) wrote :

I've just added code to the publishing scripts to decide on a per-release basis what size the root ebs filesystem should be:
http://bazaar.launchpad.net/~ubuntu-on-ec2/ubuntu-on-ec2/ec2-publishing-scripts/revision/247

My argument against changing the size in stable releases is
a.) this probably doesn't normally qualify as a SRU able change. It doesn't clearly fall under one of the 'when' at https://wiki.ubuntu.com/StableReleaseUpdates .
b.) it is somewhat an arbitrary event to respond to.

This isn't just a macho response. I really would like to have ubuntu easily used as "Free" on ec2. But we can't arbitrarily change users expectation in a stable release in response to external events (other than security).

Changed in ubuntu-on-ec2:
importance: Undecided → Medium
Scott Moser (smoser) wrote :

I've written a blog entry on how to create your own 10GB EBS instance. I realise that the process is likely difficult for a "ec2 newbie".

http://ubuntu-smoser.blogspot.com/2010/11/using-ubunt-images-on-aws-free-tier.html

James Clemence (jvc26) wrote :

Thanks for the help, I think I can probably work through those instructions, with the exception of ec2, I'm very familiar with linux/cli.

J

Scott Moser (smoser) wrote :

FYI, I've sent a mail to the technical board [1] suggesting that we change the default size for future refreshes of 10.04 and 10.10 to 10G. There is some chance of choosing a 8G volume rather than 10G. 8G was a suggestion from Amazon, and I'm waiting for more information as to why.

--
[1] https://wiki.ubuntu.com/TechnicalBoard

Scott Moser (smoser) wrote :

technical board mailing list thread is at https://lists.ubuntu.com/archives/technical-board/2010-November/thread.html#526 ("resize of root filesystem from 15G to 10G for new ec2 ebs images ").

James Clemence (jvc26) wrote :

Thanks for the instructions on the blog post, was able to follow them and get a new 10Gb AMI functioning. Much appreciated. Also cheers for the update on the progress of moving the ec2 EBS images over to 10Gb from 15.

J

Scott Moser (smoser) wrote :

The tech board approved of this change. Due to recommendation from Amazon, I'm going to make our volume size 8G.

I've just put the change in place so that all future daily builds will create 8G ebs volumes.

When we release new refreshed images of lucid and maverick, they'll have 8G ebs root.

Note, that if you wanted a 15G (or larger) volume, you can simply launch the instance with a larger volume. This can be done via the RunInstances api [1] or via ec2-run-instances [2] with --block-device-mapping.

For example:
  ec2-run-instances --block-device-mapping /dev/sda1=:100 --key mykey ami-480df921
Then, inside the instance, simply:
  sudo resize2fs /dev/sda1

See [2] for more information.

Note, that in Natty, the plan is to invoke 'resize2fs' on the root volume on first boot. That means that if you launched an instance a larger-than-default volume for sda1, it would automatically resize the filesystem to fill it.

--
[1] http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-query-RunInstances.html
[2] http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-RunInstances.html
[3] http://alestic.com/2009/12/ec2-ebs-boot-resize

Changed in ubuntu-on-ec2:
assignee: nobody → Scott Moser (smoser)
status: New → Fix Committed
Nick Wiedenbrueck (niwi8877) wrote :

I'm a complete newbie to EC2. Thanks for creating these new AMIs. When will they be available an how can I use them? Will the currently available AMIs be updated (and when)? Or are daily builds available and published and how can I find the AMI ids?

Scott Moser (smoser) wrote :

The first of the daily builds with an 8GB root filesystem are available at http://uec-images.ubuntu.com/server/maverick/ . There is a 10GB root build of lucid at http://uec-images.ubuntu.com/server/lucid/20101105/ and subsequent dailies will have 8GB for lucid also.

The documents at https://wiki.ubuntu.com/UEC/Images/RefreshPolicy and https://wiki.ubuntu.com/UEC/Images/NamingConvention describe what the daily builds are in relation to "released" builds which can be found at http://uec-images.ubuntu.com/releases/ .

At some point this month, we will take one of these daily builds, test it and create new AMIs based on it (same content different ami-abcdefg numbers), and mark those as 'released'.

The short summary, is that you can use one of the maverick daily EBS root images and be completely happy with it. The ami will be removed at some point by Canonical (daily build amis are removed, 'released' builds never are). That will in no way affect your cost or usage of that running instance. However, future instances would have to be launched using a different AMI (which you can find in the 'releases' url above). If you launch a maverick instance and are generally happy with it, there is real reason not to do this. You will be able to receive software updates via 'apt-get' for any changes released, effectively upgrading you to any later version.

I would personally suggest using maverick, as at the moment the lucid AMIs do not use 'pv-grub' as the loader, and thus are not able to 'apt-get update && apt-get dist-upgrade && reboot' into a new kernel.

Scott Moser (smoser) wrote :

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=4349
will only ever list released builds. There are no officially available released builds with 8G root volume.

Any of the dailies 20101106 or later at http://uec-images.ubuntu.com/server/maverick/ will have 8G root filesystem.

BigBaaadBob (witr) wrote :

Total dumb-ass question: how do you use (boot) a daily UEC image (say the latest Maverick one) on EC2?

IRobot (kapokfly) wrote :

Thanks for the work, FYI using the new 8GB EBS root AMI got below update error.

ap-southeast-1 | 64-bit | ebs | ami-a0750bf2

ubuntu@:~$ sudo apt-get update && sudo apt-get upgrade -y
Hit http://ap-southeast-1.ec2.archive.ubuntu.com maverick Release.gpg
Ign http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/ maverick/main Translation-en
Ign http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/ maverick/main Translation-en_US
Ign http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/ maverick/universe Translation-en
Ign http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/ maverick/universe Translation-en_US
Hit http://ap-southeast-1.ec2.archive.ubuntu.com maverick-updates Release.gpg
Ign http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/ maverick-updates/main Translation-en
Ign http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/ maverick-updates/main Translation-en_US
Ign http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/ maverick-updates/universe Translation-en
Ign http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/ maverick-updates/universe Translation-en_US
Hit http://ap-southeast-1.ec2.archive.ubuntu.com maverick Release
Hit http://ap-southeast-1.ec2.archive.ubuntu.com maverick-updates Release
Hit http://ap-southeast-1.ec2.archive.ubuntu.com maverick/main Sources
Hit http://ap-southeast-1.ec2.archive.ubuntu.com maverick/universe Sources
Hit http://ap-southeast-1.ec2.archive.ubuntu.com maverick/main amd64 Packages
Hit http://ap-southeast-1.ec2.archive.ubuntu.com maverick/universe amd64 Packages
Get:1 http://ap-southeast-1.ec2.archive.ubuntu.com maverick-updates/main Sources [22.0kB]
Get:2 http://ap-southeast-1.ec2.archive.ubuntu.com maverick-updates/universe Sources [5,658B]
Hit http://ap-southeast-1.ec2.archive.ubuntu.com maverick-updates/main amd64 Packages
Hit http://ap-southeast-1.ec2.archive.ubuntu.com maverick-updates/universe amd64 Packages
Hit http://security.ubuntu.com maverick-security Release.gpg
Ign http://security.ubuntu.com/ubuntu/ maverick-security/main Translation-en
Ign http://security.ubuntu.com/ubuntu/ maverick-security/main Translation-en_US
Ign http://security.ubuntu.com/ubuntu/ maverick-security/universe Translation-en
Ign http://security.ubuntu.com/ubuntu/ maverick-security/universe Translation-en_US
Hit http://security.ubuntu.com maverick-security Release
Hit http://security.ubuntu.com maverick-security/main Sources
Hit http://security.ubuntu.com maverick-security/universe Sources
Hit http://security.ubuntu.com maverick-security/main amd64 Packages
Hit http://security.ubuntu.com maverick-security/universe amd64 Packages
Fetched 2B in 1s (2B/s)
W: Failed to fetch http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/dists/maverick-updates/main/source/Sources.bz2 Hash Sum mismatch

W: Failed to fetch http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/dists/maverick-updates/universe/source/Sources.bz2 Hash Sum mismatch

E: Some index files failed to download, they have been ignored, or old ones used instead.

Scott Moser (smoser) wrote :

IRobot,
  Your apt-get update error is not related to the image, but is a apt-server error. I've contacted the people in charge of those apt mirrors.

Bob,
  http://uec-images.ubuntu.com/server/maverick/ has ec2-run-instances commands (in the ec2-api-tools package). You may find information at https://help.ubuntu.com/community/EC2StartersGuide to be useful.

IRobot (kapokfly) wrote :

Thanks for contacting the person who is in charge :-)

Another small question:

$ sudo fdisk -l

Disk /dev/sda1: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sda1 doesn't contain a valid partition table

Do I need worry about this line 'Disk /dev/sda1 doesn't contain a valid partition table'?.

Thanks

Scott Moser (smoser) wrote :

Irobot. that is expected behavior on ec2. /dev/sda1 is a partition, and thus does not have a partition table.

BigBaaadBob (witr) wrote :

Any update about the problem in #16? Should this be a separate bug somewhere? I just started a new instance and have the same problem.

I was charged $0.01 for regional data transfer! :-) The only thing it could be for are the Ubuntu updates that were downloaded from http://us-east-1.ec2.archive.ubuntu.com (default settings). Regional data transfers are suprisingly not included in the free tier. It seems it's better to use non-amazon mirror here.

Should the default repository address be changed? It's better for free tier users, but worse for regular users.

Scott Moser (smoser) wrote :

@FooBar,
  Sorry, I don't think we're going to change the default mirrors. Using local mirrors is really better for everyone involved (Canonical, Amazon, and the end user) other than the minute charge associated with network transfer in.

  Also, according to http://aws.amazon.com/ec2/pricing/ "all data transfer" "in" is $0.10 per GB. In region cost is $0.01/GB. So, had you pulled the same GB of data to the instance from us.archive.ubuntu.com you would have paid 10 times that price. In the past amazon has had "move in specials" where all data transfer in was free, usually posted as until a given date.

bpursley (bernie-bpursley) wrote :

It has been nearly six weeks since the 8GB AMI's got the green light from the
technical board - is there a planned release date for the AMI's to Amazon EC2?

Scott Moser (smoser) wrote :

Well, the images are ready. they're building daily.
I was about to try to push a refresh this week, but I'd like to wait on the kernel that is sitting now in -proposed.

That is expected to make it into -updates end of next week .

Given holiday break, i really expect that it will be into the new year now.

However, you can use a daily, and they're "just as good" as "released" for the time being.

Eric Hammond (esh) wrote :

Scott: For many purposes, dailies are not "just as good" because they don't live forever. Some processes regularly run instances and need to have an AMI id which will not suddenly disappear. I would like to see Ubuntu release AMIs much more frequently, especially after bugs that cause problems for folks running the latest AMIs (like #683379).

On Fri, 17 Dec 2010, Eric Hammond wrote:

> Scott: For many purposes, dailies are not "just as good" because they
> don't live forever. Some processes regularly run instances and need to
> have an AMI id which will not suddenly disappear. I would like to see
> Ubuntu release AMIs much more frequently, especially after bugs that
> cause problems for folks running the latest AMIs (like #683379).

I realize that dailies aren't the same as releases. I also would like to
get to the point where we're making regular monthly releases in addition
to releases in response to security issues or other flaws that would need
addressing.

I normally wouldn't suggest just using dailies for anything more than
preview/verification/test. However, in the case where the user is simply
wanting to run a "Free Tier" image and expects to be using that image as a
long-term instance, it would be acceptable for my personal needs. I can't
really imagine much other use for free tier.

Regarding bug 683379, we'll get refreshed images out. I simply didn't
think it made sense to push a release out this week with a kernel sitting
in -proposed.
I realize its painful, but there are documented work arounds.

For my website I plan to start a backup server on EC2 to only use when my normal server stops functioning properly. So it will only run once or twice a year for a couple of days or so. As it won't be used much, I like to make use of the free tier level (don't use more than 10GB) and have it take up as few disk space as possible.

When I activate it again in the future, I just want to run the updates and get going!

Would it suffice if I take the latest maverick build from here http://uec-images.ubuntu.com/server/maverick/
Or is there already a better option at the moment?

Scott Moser (smoser) wrote :

This is now "Fix Released".
Updated released versions of 10.04 [1] and 10.10 [2] are available that have 8GB root disks.
The release announcements can be seen at [3] and [4] respectively.

Sorry for being delinquent in updating this bug.

--
[1] http://uec-images.ubuntu.com/releases/10.04/release/
[2] http://uec-images.ubuntu.com/releases/10.10/release/
[3] https://lists.ubuntu.com/archives/ubuntu-cloud/2010-December/000466.html
[4] https://lists.ubuntu.com/archives/ubuntu-cloud/2010-December/000467.html

Changed in ubuntu-on-ec2:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers