ec2 stream data images missing at least in us-east-2

[Scott Moser]

Many images present on EC2 are not published in stream data.
Examples of missing information:
 * us-east-2 daily builds of precise, trusty, xenial
 * us-east-2 release builds of precise, trusty, xenial

Some quick searching shows that possibly these builds are being
published incorrectly with a 'ubuntu/images-testing-dev' prefix rather
than 'ubuntu/images'. That might be wrong, but something is wrong.

$ aws --output=text ec2 --region=us-east-2 describe-images --owners=099720109477 | grep ^IMAGES > out
$ awk '-F\t' '$11 ~ /ubuntu\/images\/.*xenial/ {print $7, $11}' out | pastebinit
http://paste.ubuntu.com/24140539/

## image-status is
## https://github.com/smoser/talk-simplestreams/blob/master/bin/image-status

$ regions=$(image-status ec2-release | awk '{print $3}' | sort -u)
$ echo $regions
ap-northeast-1 ap-northeast-2 ap-south-1 ap-southeast-1 ap-southeast-2 ca-central-1 eu-central-1 eu-west-1 eu-west-2 sa-east-1 us-east-1 us-east-2 us-west-1 us-west-2

$ releases="precise trusty xenial yakkety zesty"
$ streams="ec2-daily ec2-release"

## For brevity, I'm setting only looking at us-east-1 and us-east-2
## there are other issues

$ regions="us-east-1 us-east-2"
$ for stream in $streams; do
   echo "==== $stream ===="
   for region in $regions; do
     echo "=== $region ==="
     for release in $releases; do
        image-status $stream region=$region release=$release
     done; done; done
==== ec2-daily ====
=== us-east-1 ===
precise 20170307 us-east-1 ebs hvm ami-6c31977a
precise 20170307 us-east-1 ebs pv ami-77298f61
precise 20170307 us-east-1 instance hvm ami-2d2b8d3b
precise 20170307 us-east-1 instance pv ami-07288e11
precise 20170307 us-east-1 io1 hvm ami-bf3096a9
precise 20170307 us-east-1 io1 pv ami-bd3492ab
precise 20170307 us-east-1 ssd hvm ami-783c9a6e
precise 20170307 us-east-1 ssd pv ami-4e319758
trusty 20170307 us-east-1 ebs hvm ami-f73f99e1
trusty 20170307 us-east-1 ebs pv ami-dc3593ca
trusty 20170307 us-east-1 instance hvm ami-9d288e8b
trusty 20170307 us-east-1 instance pv ami-3c2a8c2a
trusty 20170307 us-east-1 io1 hvm ami-403d9b56
trusty 20170307 us-east-1 io1 pv ami-51319747
trusty 20170307 us-east-1 ssd hvm ami-e93e98ff
trusty 20170307 us-east-1 ssd pv ami-3d31972b
xenial 20170303.1 us-east-1 instance hvm ami-a3d309b5
xenial 20170303.1 us-east-1 instance pv ami-86dc0690
xenial 20170303.1 us-east-1 ssd hvm ami-c7d00ad1
xenial 20170303.1 us-east-1 ssd pv ami-70d10b66
yakkety 20170307 us-east-1 instance hvm ami-ed8425fb
yakkety 20170307 us-east-1 instance pv ami-14892802
yakkety 20170307 us-east-1 ssd hvm ami-78b7166e
yakkety 20170307 us-east-1 ssd pv ami-0dbd1c1b
zesty 20170306.1 us-east-1 instance hvm ami-e508acf3
zesty 20170306.1 us-east-1 instance pv ami-9c09ad8a
zesty 20170306.1 us-east-1 ssd hvm ami-263f9b30
zesty 20170306.1 us-east-1 ssd pv ami-2107a337
=== us-east-2 ===
yakkety 20170307 us-east-2 instance hvm ami-f06a4e95
yakkety 20170307 us-east-2 instance pv ami-ba684cdf
yakkety 20170307 us-east-2 ssd hvm ami-6a6f4b0f
yakkety 20170307 us-east-2 ssd pv ami-4c684c29
zesty 20170306.1 us-east-2 instance hvm ami-019abf64
zesty 20170306.1 us-east-2 instance pv ami-7864401d
zesty 20170306.1 us-east-2 ssd hvm ami-7e64401b
zesty 20170306.1 us-east-2 ssd pv ami-36654153
==== ec2-release ====
=== us-east-1 ===
precise 20170307 us-east-1 ebs hvm ami-f39f3ee5
precise 20170307 us-east-1 ebs pv ami-f59e3fe3
precise 20170307 us-east-1 instance hvm ami-da9637cc
precise 20170307 us-east-1 instance pv ami-21882937
precise 20170307 us-east-1 io1 hvm ami-5662c340
precise 20170307 us-east-1 io1 pv ami-f99e3fef
precise 20170307 us-east-1 ssd hvm ami-7e9e3f68
precise 20170307 us-east-1 ssd pv ami-6a63c27c
trusty 20170307 us-east-1 ebs hvm ami-9dde7f8b
trusty 20170307 us-east-1 ebs pv ami-9fde7f89
trusty 20170307 us-east-1 instance hvm ami-a8d170be
trusty 20170307 us-east-1 instance pv ami-06c36210
trusty 20170307 us-east-1 io1 hvm ami-5ea10048
trusty 20170307 us-east-1 io1 pv ami-79a0016f
trusty 20170307 us-east-1 ssd hvm ami-7fa00169
trusty 20170307 us-east-1 ssd pv ami-59a1004f
xenial 20170303 us-east-1 instance hvm ami-28bb613e
xenial 20170303 us-east-1 instance pv ami-8a8b519c
xenial 20170303 us-east-1 ssd hvm ami-09b3691f
xenial 20170303 us-east-1 ssd pv ami-78bd676e
yakkety 20170222 us-east-1 instance hvm ami-900ddc86
yakkety 20170222 us-east-1 instance pv ami-f70adbe1
yakkety 20170222 us-east-1 ssd hvm ami-e600d1f0
yakkety 20170222 us-east-1 ssd pv ami-da04d5cc
=== us-east-2 ===
yakkety 20170222 us-east-2 instance hvm ami-fba0859e
yakkety 20170222 us-east-2 instance pv ami-23a38646
yakkety 20170222 us-east-2 ssd hvm ami-4fa6832a
yakkety 20170222 us-east-2 ssd pv ami-4ba6832e

[Scott Moser (smoser)]

it seems the quickstart panel is *also* out of date. Just walking through the EC2 web UI to launch an instance, I get suggested for ubuntu
 us-east-1 ami-f4cc1de2 ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20170221
 us-east-2 ami-fcc19b99 ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20161221

So the us-east-2 image is 2 months older than the us-east-1.

[Scott Moser (smoser)]

So it seems this wasn't as bad as I originally thought.
Juju can find an image in that region, as there *is* an image.

The issue is that the latest version (20170307) of the product
  com.ubuntu.cloud:server:16.04:amd64
does not have an image in us-east-2. The 'image-status' expects
that all images are present.

The issue, then is just that us-east-2 has no 20170307 version.

[Joshua Powers (powersj)]

Scott, I am going to go ahead and close this as it was about an image many years ago. If it is still an issue feel free to re-open. Thanks!