some xenial amis are missing snapshots in aws

[kesten broughton]

This may relate to
#1671194 ec2 stream data images missing at least in us-east-2

For me, it manifests as ami's that exist in ec2, but their snapshots do not.
This is a problem as we are trying to automate ami building based on snapshots.

Eg.

The latest hvm:evs-ssd of xenial released 2017 03 30 for us=west-2 is ami-fa27b39a
This image exists in aws, and you can launch an instance from it.

Its metadata shows a snapshot-id of 'snap-06083cf1c6817cc4f'
This snap doesn't appear to exist in AWS EC2 snapshots.

16:10 $ aws ec2 describe-images --image-id ami-fa27b39a
{
    "Images": [
        {
            "VirtualizationType": "hvm",
            "Name": "ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20170330",
            "Hypervisor": "xen",
            "EnaSupport": true,
            "SriovNetSupport": "simple",
            "ImageId": "ami-fa27b39a",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/sda1",
                    "Ebs": {
                        "DeleteOnTermination": true,
                        "SnapshotId": "snap-06083cf1c6817cc4f",
                        "VolumeSize": 8,
                        "VolumeType": "gp2",
                        "Encrypted": false
                    }
                },
                {
                    "DeviceName": "/dev/sdb",
                    "VirtualName": "ephemeral0"
                },
                {
                    "DeviceName": "/dev/sdc",
                    "VirtualName": "ephemeral1"
                }
            ],
            "Architecture": "x86_64",
            "ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20170330",
            "RootDeviceType": "ebs",
            "OwnerId": "099720109477",
            "RootDeviceName": "/dev/sda1",
            "CreationDate": "2017-03-31T12:47:02.000Z",
            "Public": true,
            "ImageType": "machine",
            "Description": "Canonical, Ubuntu, 16.04 LTS, amd64 xenial image build on 2017-03-30"
        }
    ]
}

HOWEVER, the snapshot cannot be found.

16:47 $ aws ec2 describe-snapshots --snapshot-id snap-06083cf1c6817cc4f

An error occurred (InvalidSnapshot.NotFound) when calling the DescribeSnapshots operation: The snapshot 'snap-06083cf1c6817cc4f' does not exist.

Searching only owner=099720109477 (ubuntu) and xenial string and sort descending on date there are large gaps in snapshot releases
march 17, 16, 11, 8, 3...

Searching the same but for amis, there are a few gaps but it is almost daily.

[kesten broughton (dathomir)]

Here is a script to detect the amis with missing snapshots
---------------

from __future__ import print_function
import boto3
import logging

# See https://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Client.describe_snapshots
# And https://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.Client.describe_images
# Limitations: Does not handle paging, so if NextToken is not null, warn and fail

ARGPARSE_DESCRIPTION = '''
Script to find latest complete ami and report missing snapshots.

Primarily designed to fetch latest ubuntu cloud image, but flexible options for other targets.
It turns out that sometimes ubuntu pushes ami's which have snapshots that can't be found.
https://bugs.launchpad.net/cloud-images/+bug/1679855
We check to make sure that the snapshots used by the ami are present.
'''

DEFAULT_OWNER = "099720109477" # ubuntu

try:
    import configargparse as argparse
    import dotmap
    parser = argparse.ArgumentParser(
        description=ARGPARSE_DESCRIPTION,
        config_file_parser_class=argparse.YAMLConfigFileParser)
    parser.add_argument('-c', '--config-file', is_config_file=True, dest='config_file', required=False,
                        help='provide a yaml config file (using dest names) instead of commandline args')
    HAS_CONFIG_ARGPARSE = True
except:
    import argparse
    parser = argparse.ArgumentParser(
        description=ARGPARSE_DESCRIPTION)
    HAS_CONFIG_ARGPARSE = False

def get_amis(ec2, args):
    amis = ec2.describe_images(Owners=args.owners,
                               Filters=args.ami_filters)
    return sorted(amis["Images"], key=lambda k: k['CreationDate'], reverse=True)

def main():

    parser.add_argument('-r', '--region', dest='region', default='us-west-2')
    parser.add_argument('-o', '--owner-ids', dest="owners", default=[DEFAULT_OWNER],
                        help="""AWS org owner, default is id for ubuntu org, ['string1',]
                        Use ['self'] for caller's aws org""")
    parser.add_argument('-f', '--ami-filters', dest="ami_filters",
                        default=[{'Name': 'root-device-type',
                                     'Values': ['ebs'] },
                                    {'Name': 'architecture',
                                     'Values': ['x86_64'] },
                                    {'Name': 'virtualization-type',
                                     'Values': ['hvm'] },
                                    {'Name': 'name',
                                     'Values': ['*ubuntu-xenial-16.04-amd64-server*'] }]
                                )
    parser.add_argument('-n', '--num-amis', dest='max_amis', type=int, default=4)

    if HAS_CONFIG_ARGPARSE:
        args = parser.parse_known_args()
        args = dotmap.DotMap(vars(args[0]))
    else:
        args = parser.parse_args()

    ec2 = boto3.client('ec2')

    good_ami_snap_map = []
    bad_ami_snap_map = []
    amis = get_amis(ec2, args)
    if not amis:
        raise Exception("No amis found. Check your filters {}".format(args.ami_filters))
    for ami in amis[:args.max_amis]:
        snaps = []
        for i, device in enumerate(ami["BlockDeviceMappings"...

[Dan Watkins (oddbloke)]

This should now be fixed for all new images, and for images published after the start of this month.