Karmic EC2 images don't use a karmic kernel

Bug #418130 reported by Chuck Short on 2009-08-24
32
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Ubuntu on EC2
High
John Johansen
Ubuntu
High
Scott Moser
Karmic
High
Scott Moser

Bug Description

Karmic alpha4 and daily AMIs currently use an intrepid kernel and not a karmic kernel. Ideally they should ship with a karmic kernel.

Chuck Short (zulcss) wrote :

WARNING: vmlinux.o (__xen_guest): unexpected non-allocatable section.
We have the Suse patches however we are still tweaking the config for i386:

Did you forget to use "ax"/"aw" in a .S file?
Note that for example <linux/init.h> contains
section definitions for use in .S files.

WARNING: modpost: Found 2 section mismatch(es).
To see full details build your kernel with:
'make CONFIG_DEBUG_SECTION_MISMATCH=y'
  GEN .version
  CHK include/linux/compile.h
  UPD include/linux/compile.h
  CC init/version.o
  LD init/built-in.o
  LD .tmp_vmlinux1
/usr/bin/ld.real: kernel image bigger than KERNEL_IMAGE_SIZE
/usr/bin/ld.real: kernel image bigger than KERNEL_IMAGE_SIZE

Thierry Carrez (ttx) on 2009-08-27
summary: - Karmic EC2 kernel tracking bug
+ Karmic EC2 images don't use a karmic kernel
description: updated
Changed in ubuntu-on-ec2:
assignee: nobody → John Johansen (jjohansen)
importance: Undecided → High
status: New → Triaged
Matt Zimmerman (mdz) wrote :

Bug 398568 needs to be fixed in order to use a Karmic kernel.

John Johansen (jjohansen) wrote :

There are 2 possible kernel paths
  A pv-ops based Karmic kernel which currently exhibits the failure in Bug 398568 except in all zones

And a xen patchset kernel which currently fails on 32 bit builds with the above message, and fails linking in 64 builds due to missing symbols.

The Karmic alpha4 and alpha5 test images have now been successfully tested with a new Jaunty + xen patchset kernel.

John Johansen (jjohansen) wrote :

One of the disadvantages of using an older kernel on Karmic is AppArmor incompatibilities.

The Karmic user space tools are not compatible with the intrepid kernel. To obtain a functioning AppArmor for the intrepid kernel the intrepid user space tools will need to be used, and the Karmic profiles will need to be modified to remove use of features not supported in Intrepid.

The Karmic user space tools are compatible with the Jaunty kernel, however the Karmic profiles will need to be modified to remove use of features not supported in Jaunty.

Matt Zimmerman (mdz) wrote :

Tim Gardner of the kernel team put together this wiki page:

https://wiki.ubuntu.com/KernelTeam/EC2

illustrating the different options being explored to correct this bug.

John Johansen (jjohansen) wrote :

We now have a vanilla 2.6.31 + xen patch kernel build but it fails in CentOS 5.0, with the following message from xmcreate

Error: (2 'Invalid Kernel', 'elf_xen_addr_calc_check: ERROR: ELF start or entries are out of bounds.\n')

John Johansen (jjohansen) wrote :

We have a vanilla 2.6.31 kernel running now and are working on a port to the karmic kernel.

> We have a vanilla 2.6.31 kernel running now and are working on a port to
> the karmic kernel.

Can you give the aki/ari here ?

John Johansen (jjohansen) wrote :

The current vanilla 2.6.31 kernel can be tested against the ami for karmic alpha5 (ami-3520c05c) by manually specifying a kernel image of aki-1b3ada72 and a ramdisk of ari-1d3ada74.

John Johansen (jjohansen) wrote :

The patch series and config used to generate the vanilla 2.6.31 kernel resides on chinstrap in /home/jj/ec2-kernel/2.6.31/

EC2 requires xen 3.0.2 compatibility in addition DMI must be turned off and the LOAD_OFFSET must be set to 0. The last patch in the series contains the magic required to get the kernel to run.

Scott Moser (smoser) on 2009-09-08
tags: added: ec2-images uec-images
Scott Moser (smoser) wrote :

creating ubuntu/linux task and retiring ubuntu-on-ec2 task

Changed in linux (Ubuntu):
status: New → Triaged
assignee: nobody → John Johansen (jjohansen)
importance: Undecided → High
Changed in ubuntu-on-ec2:
status: Triaged → Invalid
Tim Gardner (timg-tpi) wrote :

I created a topic branch in the Karmic git repository for EC2:

git://kernel.ubuntu.com/ubuntu/ubuntu-karmic.git ec2

First upload at https://edge.launchpad.net/~timg-tpi/+archive/ppa (if it builds successfully)

John Johansen (jjohansen) wrote :

There is now an i386 Karmic Kernel for testing
  aki-f700e09e
  ari-c900e0a0

John Johansen (jjohansen) wrote :

The i386 kernel aki-f700e09e has been booted and smoke tested using both ami-3520c05c (Jaunty test) and ami-3520c05c (Karmic alpha5).
  Both instances returned system logs and were successfully logged into with uname -a reporting the correct kernel version.

John Johansen (jjohansen) wrote :

A x86_64 Karmic Kernel from the PPA is available for testing
  aki-4501e12c
  ari-4701e12e

I have had success booting with it on all 4 us-east zones. With successful log in and console logs

Dylan Barrell (dylan-barrell) wrote :

Trying to launch an image with the i386 kernel and ramdisk

aki-f700e09e
ari-c900e0a0

results in the following error: Registered kernel image manifest for aki-f700e09e and manifest in S3 differ. Has the image been rebundled but not re-registered?

Eric Hammond (esh) wrote :

The 64-bit kernels overwrote the 32-bit kernels since both were named "test-kernel-01". The 32-bit kernels are unusable.

Please build the kernels and ramdisks using descriptive manifest prefixes including the kernel version, architecture, date, and tags like:

  ubuntu-kernel-2.6.31-1-xen-i386-20090909-test-01
  ubuntu-ramdisk-2.6.31-1-xen-i386-20090909-test-01

Hopefully this can be automated instead of manually entered each time a kernel is built.

John Johansen (jjohansen) wrote :

Right I was just overloading the name during testing and forgot to upload to a new name for wider testing.

I have uploaded a new x86_64 kernel and ramdisk with a manifest name following your suggested format
  aki-a004e4c9 ubuntu-kernel-2.6.31-300-ec2-xen-x86_64-20090910-test-02
  ari-a204e4cb ubuntu-ramdisk-2.6.31-300-ec2-xen-x86_64-20090910-test-02

John Johansen (jjohansen) wrote :

New i386 kernel and ramdisk for testing
  aki-9c04e4f5 ubuntu-kernel-2.6.31-300-ec2-xen-i386-20090910-test-02
  ari-9e04e4f7 ubuntu-ramdisk-2.6.31-300-ec2-xen-i386-20090910-test-02

NOTE: the current karmic alpha5 ami requires setting the user data field

Eric Hammond (esh) wrote :

Thanks!

Are there matching kernel modules available for download?

John Johansen (jjohansen) wrote :

The can be grabbed from the PPA .deb's or if you prefer I can tar them up and put them modules up some where.

John Johansen (jjohansen) wrote :

An alternate i386 test kernel is available,
  aki-dc06e6b5 ubuntu-kernel-2.6.31-300-ec2-xen-i386-20090910-test-03
  ari-de06e6b7 ubuntu-ramdisk-2.6.31-300-ec2-xen-i386-20090910-test-03

this kernel removes the following warning from boot
[ 1.588049] ***************************************************************
[ 1.588055] ***************************************************************
[ 1.588060] ** WARNING: Currently emulating unsupported memory accesses **
[ 1.588065] ** in /lib/tls glibc libraries. The emulation is **
[ 1.588070] ** slow. To ensure full performance you should **
[ 1.588075] ** install a 'xen-friendly' (nosegneg) version of **
[ 1.588079] ** the library, or disable tls support by executing **
[ 1.588084] ** the following as root: **
[ 1.588088] ** mv /lib/tls /lib/tls.disabled **
[ 1.588093] ** Offending process: init (pid=1007) **
[ 1.588098] ***************************************************************
[ 1.588102] ***************************************************************

Eric Hammond (esh) wrote :

PPA debs are great. I see some kernel modules here, but not for 2.6.31-300-ec2-xen:
  https://launchpad.net/~ubuntu-on-ec2/+archive/ppa

Dylan Barrell (dylan-barrell) wrote :

I started my Fedora 11 AMI instance using the new i386 kernel (aki-dc06e6b5) and it started fine. I could not find a way to specify it should use the new ramdisk through the AWS console.

However, when trying to make an AMI from this instance, I get the following error:

mount: could not find any free loop device
ERROR: execution failed: "mount -o loop /mnt/fed11-irc-kernel2.6.31 /mnt/img-mnt"

> mount: could not find any free loop device
> ERROR: execution failed: "mount -o loop /mnt/fed11-irc-kernel2.6.31 /mnt/img-mnt"

Dylan,
   Could you please try installing the appropriate (i386 or x86_64)
linux-image deb in the instance. At
https://edge.launchpad.net/~timg-tpi/+archive/ppa use:
  - linux-image-2.6.31-300-ec2_2.6.31-300.2_amd64.deb
  - linux-image-2.6.31-300-ec2_2.6.31-300.2_i386.deb

   Then
$ modprobe loop
   Then, re-try your re-bundle.

   The difference here is that previous ec2 kernels (and other karmic
kernels) kernels had
CONFIG_BLK_DEV_LOOP=y
this one has
CONFIG_BLK_DEV_LOOP=m

Chris Lukas (lukasware) wrote :

Dylan, you have to load the kernel modules, which is what Eric is looking for. The karmic OS AMI might be
"2.6.31" but the Amazon Kernel Image (AKI) is 2.6.27-13. This bug is the effort to make the AKI and the AMI
be the same version of the OS and have the correct modules so the loop driver (and others) can be added.

Chris Lukas (lukasware) wrote :

sorry dylan you are probably way ahead of me I can't even see the aki-dc06e6b5 / ari-de06e6b7 mentioned above, but did find the kernel modules (thanks Scott)

Dylan Barrell (dylan-barrell) wrote :

ok, that fixed the loop issue. I had to convert the .deb to a .rpm which was painful.

I used http://packages.debian.org/unstable/source/alien but modified the Alien/Packages/Deb.pm file to look for bz2 and use bzip2 instead of gzip and gz.

Then ran ./alien.pl on the .deb
Then ran depmod
Then ran modprobe loop

The AMI bundling worked but gave the following warning message:

NOTE: rsync with preservation of extended file attributes failed. Retrying rsync
without attempting to preserve extended file attributes...

John Johansen (jjohansen) wrote :

>PPA debs are great. I see some kernel modules here, but not for 2.6.31-300-ec2-xen:
> https://launchpad.net/~ubuntu-on-ec2/+archive/ppa

Both
  ubuntu-kernel-2.6.31-300-ec2-xen-i386-20090910-test-02
  ubuntu-kernel-2.6.31-300-ec2-xen-x86_64-20090910-test-02

are built from the PPA the extra -xen- comes from my script I used to create and upload the kernel images. I have fixed the bug so the problem won't appear in future uploads.

The alternate
  ubuntu-kernel-2.6.31-300-ec2-xen-i386-20090910-test-03

is built directly from source, but the patch that was applied shouldn't have changed the abi so the PPA modules should be compatible

Eric Hammond (esh) wrote :

I see from Scott's message that the kernel modules were uploaded to a different PPA from previous versions. Is there a reason these weren't placed in https://launchpad.net/~ubuntu-on-ec2/+archive/ppa ?

I would recommend that the kernel modules and other production required components for EC2 be placed in an official team PPA rather than an individual developer's PPA.

Tim Gardner (timg-tpi) wrote :

@Eric - I'm writing the package MIR (main inclusion request) this evening, so I should be able to get the EC2 kernel uploaded to the official Ubuntu archive pretty soon. The PPA versions were merely for smoke testing.

Eric Hammond (esh) wrote :

Tim: I understand. I love the policy of getting test images out early and often, and can live with temporary locations and inconveniences. As long as it's clear that things are moving towards standardized and reproducible processes with long term vision, I'm happy.

Eric Hammond (esh) wrote :

<offtopic>Dylan: Rather than convert deb to rpm, you could simply fire up an EC2 Ubuntu instance, install the deb, and tar up /lib/modules/... for use in Fedora 11</offtopic>

Tim Gardner (timg-tpi) wrote :

Here is the Main Inclusion Request: https://bugs.edge.launchpad.net/ubuntu/+bug/427658

Scott Moser (smoser) wrote :

just for the record, John opened bug 427288 to address the boot warning in comment 22.

Matt Zimmerman (mdz) on 2009-09-12
affects: linux (Ubuntu) → ubuntu
Changed in ubuntu:
assignee: John Johansen (jjohansen) → Scott Moser (smoser)
Eric Hammond (esh) on 2009-09-12
Changed in ubuntu:
status: Triaged → In Progress
Chris Lukas (lukasware) wrote :

this might be superflous at this point,
but I wanted to see what the smallest number
of changes necessary to the alpha5 i386
ami-3520c05c / aki-dc06e6b5 / ari-de06e6b7
2.6.31 kernel were needed in order to re-bundle at will.

echo "EC2_HOME=/usr/lib/ec2-ami-tools
EC2_PRIVATE_KEY=/appl/pk-V6JAMNSK2VIDB4GUBXK7BFMRQOX2Z43B.pem
EC2_CERT=/appl/cert-V6JAMNSK2VIDB4GUBXK7BFMRQOX2Z43B.pem
JAVA_HOME=/usr/lib/jvm/java-6-sun" >> /etc/environment
. /etc/environment # gets executed on sudo
echo "deb http://us.archive.ubuntu.com/ubuntu/ karmic-updates multiverse" >> /etc/apt/sources.list
apt-get update
apt-get install -y unzip sun-java6-jre
cd $EC2_HOME
wget https://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
unzip ec2-api-tools.zip
rm ec2-api-tools-*/bin/*.cmd
mv ec2-api-tools-*/bin .
cp bin/* /usr/bin
mv ec2-api-tools-*/lib/* lib
rm -r ec2-api-tools.zip ec2-api-tools-*

# link works until the MIR is approved?
wget http://ppa.launchpad.net/timg-tpi/ubuntu/pool/main/l/linux-ec2/linux-image-2.6.31-300-ec2_2.6.31-300.2_i386.deb
apt-get install wireless-crda # unusual dependency
dpkg -i /appl/linux-image-2.6.31-300-ec2_2.6.31-300.2_i386.deb

ec2-bundle-vol -d /mnt -p reb-ka5 --cert /appl/cert-1234567890.pem --privatekey pk-1234567890.pem -u 400008040000 -r i386 --kernel aki-dc06e6b5 --ramdisk ari-de06e6b7 --ec2cert /etc/ec2/amitools/cert-ec2.pem

ec2-upload-bundle -b ka5 -m /mnt/reb-ka5.manifest.xml --access-key 1234567890A6M0G2 --secret-key t123b+qvlG

ec2reg ka5/reb-ka5.manifest.xml

Chris Lukas (lukasware) wrote :

modprobe loop
before the bundle command might help

Matt Zimmerman (mdz) wrote :

On Sat, Sep 12, 2009 at 04:29:12PM -0000, Chris Lukas wrote:
> this might be superflous at this point,
> but I wanted to see what the smallest number
> of changes necessary to the alpha5 i386
> ami-3520c05c / aki-dc06e6b5 / ari-de06e6b7
> 2.6.31 kernel were needed in order to re-bundle at will.

This is interesting information, but I don't think it's relevant to this
bug.

> echo "EC2_HOME=/usr/lib/ec2-ami-tools
> EC2_PRIVATE_KEY=/appl/pk-V6JAMNSK2VIDB4GUBXK7BFMRQOX2Z43B.pem
> EC2_CERT=/appl/cert-V6JAMNSK2VIDB4GUBXK7BFMRQOX2Z43B.pem
> JAVA_HOME=/usr/lib/jvm/java-6-sun" >> /etc/environment
> . /etc/environment # gets executed on sudo
> echo "deb http://us.archive.ubuntu.com/ubuntu/ karmic-updates multiverse" >> /etc/apt/sources.list
> apt-get update
> apt-get install -y unzip sun-java6-jre
> cd $EC2_HOME
> wget https://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
> unzip ec2-api-tools.zip
> rm ec2-api-tools-*/bin/*.cmd
> mv ec2-api-tools-*/bin .
> cp bin/* /usr/bin
> mv ec2-api-tools-*/lib/* lib
> rm -r ec2-api-tools.zip ec2-api-tools-*

What's wrong with "apt-get install ec2-api-tools" instead of these 13 commands?

--
 - mdz

Chris Lukas (lukasware) wrote :

the relevance is that John's ari and aki work fine on alpha 5
so there is a path to a matching ari/aki/ami/userland/modules

an that seems to be a rare commodity on EC2 images,
and even more rare if you want a current kernel.

and "apt-get install api-tools" is the easy way...which
I mistakenly overlooked...

Chris Lukas (lukasware) wrote :

The "alpha5.1" ami-a40fefcd (aki-9c04e4f5 / ari-9e04e4f7)
still needs
linux-image-2.6.31-300-ec2_2.6.31-300.2_i386.deb
for the modules...(with the wireless-crda dependency)
but works better than ami-3520c05c / aki-dc06e6b5 / ari-de06e6b7

apt-get-install ec2-api-tools
gets the older version for some reason.

Matt Zimmerman (mdz) wrote :

On Sat, Sep 12, 2009 at 05:30:12PM -0000, Chris Lukas wrote:
> the relevance is that John's ari and aki work fine on alpha 5
> so there is a path to a matching ari/aki/ami/userland/modules
>
> an that seems to be a rare commodity on EC2 images,
> and even more rare if you want a current kernel.

Our intent with Ubuntu 9.10 is to make a complete, matching stack easily
available on EC2. The next development milestone will be alpha 6 next week,
and will offer exactly that, ready to run or rebundle directly from EC2.

I don't think we yet have official documentation on the simplest way to
rebundle, but perhaps your instructions would make a good starting point for
a wiki document?

--
 - mdz

Chris Lukas (lukasware) wrote :

That I'll gladly do even though there are people on this
thread that have forgotten more about this process than I'll ever know.

The instructions would be very short if the ami + api tools were included
in alpha 6, just as the ami tools were in a5 (why stop at ami only?)

Eric Hammond (esh) wrote :

- Here is Amazon's description of how to rebundle: http://ec2gsg-creating.notlong.com
- Here is an article I wrote on the topic: http://alestic.com/2009/06/ec2-ami-bundle
- For security reasons, I would not model putting the certificate and private key into the rebundled image as described above. Users will blindly copy this and not understand the ramifications.
- With the commands above, the secret access key is stored in .bash_history on the rebundled image which can also be a security issue.
- You can now use the openjdk-6 packages now instead of Sun's Java, but...
- I wouldn't include Java or the API tools on the image because it adds a lot to the image size and instance startup time.
- You can ec2-register from a different computer or call the API directly from a lighter program like Tim Kay's "aws" script.
- Yes, the AMI tools should be included in Canonical's EC2 images if they aren't.
- Yes, the kernel modules should be included in Canonical's EC2 images if they aren't.
- Yes, the ec2-api-tools and ec2-ami-tools packages need to be kept up to date more than they have been in the Ubuntu archives for all Ubuntu releases.
- How do you move a discussion away from a launchpad bug to which it does not relate?

Matt Zimmerman (mdz) wrote :

On Sat, Sep 12, 2009 at 11:39:06PM -0000, Eric Hammond wrote:
> - How do you move a discussion away from a launchpad bug to which it does not relate?

Set Reply-to: to an appropriate mailing list (probably ubuntu-ec2 in this
case)

--
 - mdz

Scott Moser (smoser) on 2009-09-13
tags: removed: uec-images
Scott Moser (smoser) wrote :

The kernels listed above are going to be used for alpha 6. I've copied the images to canonical-cloud-us bucket and published them in eu region under canonical-cloud-eu.

   current/us-east-1/i386/karmic aki-841efeed ari-9a1efef3
   current/us-east-1/x86_64/karmic aki-9c1efef5 ari-901efef9
   current/eu-west-1/i386/karmic aki-4879523c ari-4a79523e
   current/eu-west-1/x86_64/karmic aki-34795240 ari-36795242

aki-841efeed canonical-cloud-us/ubuntu-kernel-2.6.31-300-ec2-xen-i386-20090910-test-02.manifest.xml
ari-9a1efef3 canonical-cloud-us/ubuntu-ramdisk-2.6.31-300-ec2-xen-i386-20090910-test-02.manifest.xml
aki-9c1efef5 canonical-cloud-us/ubuntu-kernel-2.6.31-300-ec2-xen-x86_64-20090910-test-02.manifest.xml
ari-901efef9 canonical-cloud-us/ubuntu-ramdisk-2.6.31-300-ec2-xen-x86_64-20090910-test-02.manifest.xml
aki-4879523c canonical-cloud-eu/ubuntu-kernel-2.6.31-300-ec2-xen-i386-20090910-test-02.manifest.xml
ari-4a79523e canonical-cloud-eu/ubuntu-ramdisk-2.6.31-300-ec2-xen-i386-20090910-test-02.manifest.xml
aki-34795240 canonical-cloud-eu/ubuntu-kernel-2.6.31-300-ec2-xen-x86_64-20090910-test-02.manifest.xml
ari-36795242 canonical-cloud-eu/ubuntu-ramdisk-2.6.31-300-ec2-xen-x86_64-20090910-test-02.manifest.xml

Scott Moser (smoser) wrote :

I think i've had a hang using this kernel.

$ uname -a
Linux domU-12-31-39-00-86-11 2.6.31-300-ec2 #2-Ubuntu SMP Wed Sep 9 14:36:27 UTC 2009 i686 GNU/Linux

The ami was the alpha5.1, using this kernel with all updates applied to maybe monday. The system had been up for ~ 2 days, then it failed to respond with my ssh connection. Ping and new ssh also failed. I rebooted (ec2-reboot-instances) it and it came back up.

I'm attaching the console output, although its not much use. The reboot occurred around line 665.

Scott Moser (smoser) wrote :

Eric pointed out that there had been a ec2 networking issue this morning (http://ec2status.notlong.com).

The system described above has uptime of '1:04' as of '16:07:13 UTC' (09:07:13 PDT), meaning that the reboot was right in the middle of the following two messages from amazon:

07:41:11 AM PDT We are investigating connectivity issues in US-EAST-1.
08:18:25 AM PDT We have identified a problematic transit route leading into the US-EAST-1 region. We have modified our Internet routing topology to avoid the problematic path.

Thierry Carrez (ttx) wrote :

Alpha6 release uses a karmic kernel.

This may be a bit off topic for this bug, but I can’t figure out anywhere else to get help with this (The #Ubuntu IRC channel on Freenode was completely useless, heh).

I’m trying to use the Karmic alpha6 kernel for some AMIs of my own (because every other Linux kernel in the EC2 system is *absolutely ancient*). Unfortunately, booting the alpha6 AMI, there are no kernel modules for said kernel built (/lib/modules is completely empty). How can I get ahold of the Karmic alpha6 kernel modules? I was intending to rip them out of the official AMI, but… like I said, empty.

Thierry Carrez (ttx) wrote :

@elliottcable: please see bug 429169

Chris Lukas (lukasware) wrote :

@elliotcable

launch an instance using aki-841efeed (or alpha6) then
 apt-get update
 apt-get install linux-image-`uname -r`

or use the beta, it has the modules installed as part of the ami

Ross Golder (ross-golder) wrote :

I've created an instance of alpha6, but it's missing the 'ext4' module. Any ideas why?

Chris Lukas (lukasware) wrote :

fixed in the karmic EC2 beta AMI
ami-52be5d3b
64 bit:
ami-5ebe5d37

Scott Moser (smoser) wrote :

Ross, bug 428692 is there to address differences between other karmic ubuntu kernels and that of ec2.
There are aki/ari available there that we would *love* to have your feedback on.

Chris Lukas (lukasware) wrote :

let us know what they are!
  x86 beta ami-52be5d3b does not support ext4
contrary to what I just said

Chris Lukas (lukasware) wrote :

ami-52be5d3b with ari-e4896a8d and aki-e2896a8b

umount /mnt
mkfs.ext4 /dev/sda2
mount /dev/sda2 /mnt

root of course is still ext3, but that ari/aki has ext4 support...

Ross Golder (ross-golder) wrote :

I went to set up an instance with that ari/aki, but I don't see them in the selection on the Amazon Management Console when launching an instance.

Scott Moser (smoser) wrote :

Ross, please move this discussion to bug 428692.
See the aki/ari pairs mentioned in bug 428692 comment 5. Note, though, those are us-east-1 IDs there are none loaded for eu-west-1.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers