No images error when deploying services

[John George]

The structured-metadata feature branch has many tests failing (across substrates) with error ‘no "trusty" images in <region name>’ (same for “precise”). Bootstrap completes with a trusty image but deployed charms hit this error.

Failing tests can be seen here:
http://reports.vapour.ws/releases/issue/567af103749a565a333a8d09

[Ian Booth (wallyworld)]

Not having much luck so far reproducing. After much hacking, got the reproduction script in #1 to run but it gives an error:

+ /home/ian/projects/juju-ci-tools/jujuci.py -v setup-workspace --clean-env aws-deploy-trusty-amd64-ian /tmp/workspace/aws-deploy-trusty-amd64
Removing artifacts
Creating artifacts dir.
Environment "aws-deploy-trusty-amd64-ian" does not exist.
Done.
++ /home/ian/projects/juju-ci-tools/jujuci.py get-build-vars --version 3464

I also can't reproduce by hand. On both AWS and GCE I can bootstrap and deploy charms just fine. After bootstrap, juju metadata list-images shows image metadata correctly stored in state. Deploying charms correctly selects the relevant image id from that metadata.

[Ian Booth (wallyworld)]

Damn, truncated the last error message

+ /home/ian/projects/juju-ci-tools/jujuci.py -v setup-workspace --clean-env aws-deploy-trusty-amd64-ian /tmp/workspace/aws-deploy-trusty-amd64
Removing artifacts
Creating artifacts dir.
Environment "aws-deploy-trusty-amd64-ian" does not exist.
Done.
++ /home/ian/projects/juju-ci-tools/jujuci.py get-build-vars --version 3464
+ VERSION='HTTP Error 404: Not Found'

[John George (jog)]

I just reproduced with the following environments.yaml and the juju binary from build-revision 3464 (http://data.vapour.ws/juju-ci/products/version-3464/build-binary-trusty-amd64/build-713/juju-core_1.26-alpha4-0ubuntu1~14.04.1~juju1_amd64.deb).
    juju-core-1.26/usr/lib/juju-1.26-alpha4/bin/juju bootstrap
    juju-core-1.26/usr/lib/juju-1.26-alpha4/bin/juju deploy ubuntu
    juju-core-1.26/usr/lib/juju-1.26-alpha4/bin/juju status

machines:
  "0":
    agent-state: started
    agent-version: 1.26-alpha4
    dns-name: 52.25.165.59
    instance-id: i-ccfc1f0b
    instance-state: running
    series: trusty
    hardware: arch=amd64 cpu-cores=1 cpu-power=300 mem=3840M root-disk=8192M availability-zone=us-west-2a
    state-server-member-status: has-vote
  "1":
    agent-state: error
    agent-state-info: no "trusty" images in us-west-2 with arches [amd64 i386 ppc64el]
    instance-id: pending
    series: trusty

environments.yaml (credentials removed, let me know if you don't have access to them in cloud-city):

default: parallel-aws-jog
environments:
  parallel-aws-jog:
    type: ec2
    test-mode: true
    region: us-west-2
    access-key:
    secret-key:
    authorized-keys: |
    enable-os-upgrade: false
    default-series: trusty
    tools-metadata-url: http://juju-dist.s3.amazonaws.com/parallel-testing/agents
    agent-stream: revision-build-3464
    logging-config: <root>=DEBUG;unit=DEBUG

[Ian Booth (wallyworld)]

There are 2 issues.

1. The environment datasource is not set up properly so that reading image metadata from state fails.
2. Fallback to simplestreams image metadata fails due to a badly formed product id.

As seen in these logs:

2016-01-05 23:25:57 ERROR juju.apiserver.provisioner provisioner.go:1723 encountered invalid URL "" not found while getting published images metadata from environment storage
2016-01-05 23:25:57 DEBUG juju.apiserver.provisioner provisioner.go:1719 looking in data source default cloud images
2016-01-05 23:25:58 ERROR juju.apiserver.provisioner provisioner.go:1723 encountered invalid URL "https://streams.canonical.com/juju/images/releases/streams/v1/index.sjson" not found while getting published images metadata from default cloud images
2016-01-05 23:25:58 DEBUG juju.apiserver.provisioner provisioner.go:1719 looking in data source default ubuntu cloud images
2016-01-05 23:25:59 DEBUG juju.apiserver apiserver.go:275 <- [6] machine-0 {"RequestId":134,"Type":"InstancePoller","Version":1,"Request":"InstanceId","Params":"'params redacted'"}
2016-01-05 23:25:59 DEBUG juju.apiserver apiserver.go:289 -> [6] machine-0 776.851us {"RequestId":134,"Response":"'body redacted'"} InstancePoller[""].InstanceId
2016-01-05 23:25:59 DEBUG juju.environs.simplestreams simplestreams.go:430 read metadata index at "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson"
2016-01-05 23:25:59 DEBUG juju.environs.simplestreams simplestreams.go:438 index file has no data for product name(s) ["com.ubuntu.cloud.revision-build-3473:server:14.04:amd64" "com.ubuntu.cloud.revision-build-3473:server:14.04:i386" "com.ubuntu.cloud.revision-build-3473:server:14.04:armhf" "com.ubuntu.cloud.revision-build-3473:server:14.04:arm64" "com.ubuntu.cloud.revision-build-3473:server:14.04:ppc64el"]
2016-01-05 23:25:59 DEBUG juju.apiserver.provisioner provisioner.go:1608 got from data sources 0 metadata
2016-01-05 23:25:59 DEBUG juju.apiserver.provisioner provisioner.go:1548 available image metadata for provisioning: []

[Andrew Wilkins (axwalk)]

There's another issue.

- The "FindMetadata" method of the cloud image metadata store will return a NotFound error if there are no data matching the search criteria
- Image metadata are saved one at a time
- The ec2 provider (possibly others?) issues a partial query to image metadata, and then performs further filtering (on storage types)

So I think what's happening is there is *some* image metadata for precise/amd64, but not the ones we want at the time the provisioner kicked off. I will look at changing the cloud image metadata code to store metadata in bulk.

[Andrew Wilkins (axwalk)]

P.S. I was able to reliably reproduce the error by adding a precise machine as soon as bootstrap finishes (using a script to eliminate fingers as a source of slowness). With my changes (https://github.com/juju/juju/pull/4072), error went away.