cannot save cloud image metadata

[David Ames]

Running the snap edge version of juju:
Name Version Rev Developer Notes
juju 2.2-rc1+develop-139a6ae 1876 canonical classic

Against an openstack cloud. After we updated our images with simplestreams we started to see some odd behavior. Juju either tries to use a non-existent image id or it just states no matching image can be found.

 no "xenial" images in serverstack with arches [amd64]

Yet the image-stream json has a matching image.

Juju controller machine log shows the following:

2017-05-18 23:25:15 WARNING juju.apiserver.provisioner provisioninginfo.go:534 encountered index file has no data for cloud {serverstack http://10.245.161.156:5000/v3} not found while getting published images metadata from default ubuntu cloud images
2017-05-18 23:25:15 WARNING juju.apiserver.provisioner provisioninginfo.go:549 failed to save published image metadata: cannot save cloud image metadata: state changing too quickly; try again soon
2017-05-18 23:25:19 WARNING juju.provisioner provisioner_task.go:739 failed to start instance (no "xenial" images in serverstack with arches [amd64]), retrying in 10s (10 more attempts)
2017-05-18 23:25:29 WARNING juju.provisioner provisioner_task.go:739 failed to start instance (no "xenial" images in serverstack with arches [amd64]), retrying in 10s (9 more attempts)
2017-05-18 23:25:40 WARNING juju.provisioner provisioner_task.go:739 failed to start instance (no "xenial" images in serverstack with arches [amd64]), retrying in 10s (8 more attempts)
2017-05-18 23:25:51 WARNING juju.provisioner provisioner_task.go:739 failed to start instance (no "xenial" images in serverstack with arches [amd64]), retrying in 10s (7 more attempts)
2017-05-18 23:26:02 WARNING juju.provisioner provisioner_task.go:739 failed to start instance (no "xenial" images in serverstack with arches [amd64]), retrying in 10s (6 more attempts)
2017-05-18 23:26:12 WARNING juju.provisioner provisioner_task.go:739 failed to start instance (no "xenial" images in serverstack with arches [amd64]), retrying in 10s (5 more attempts)

[Ian Booth (wallyworld)]

Would it be possible to attach a dump of the cloudimagemetadata collection, as well as the simplestreams json files which were updated, so we can see why the save error occurs?

[James Page (james-page)]

daily image stream from the impacted cloud:

http://paste.ubuntu.com/24603822/

[James Page (james-page)]

not quite sure how to 'dump of the cloudimagemetadata'

[James Page (james-page)]

Bumped debug log up a bit:

>> This part is normal <<
2017-05-19 10:31:35 DEBUG juju.environs.simplestreams simplestreams.go:454 skipping index "http://cloud-images.ubuntu.com/daily/streams/v1/index.sjson" because of missing information: index file has no data for cloud {serverstack http://10.245.161.156:5000/v3} not found
2017-05-19 10:31:35 WARNING juju.apiserver.provisioner provisioninginfo.go:534 encountered index file has no data for cloud {serverstack http://10.245.161.156:5000/v3} not found while getting published images metadata from default ubuntu cloud images

>> Pickup of stream data from cloud <<

2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:amd64:lxd::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:i386:lxd::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:amd64:kvm::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:i386:lxd::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:i386:kvm::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:amd64:lxd::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:amd64:lxd::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:i386:lxd::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:amd64:kvm::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:i386:lxd::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:i386:kvm::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:amd64:lxd::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:amd64:lxd::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:i386:lxd::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:amd64:kvm::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metadata for daily:serverstack:xenial:i386:lxd::keystone catalog
2017-05-19 10:31:35 DEBUG juju.state.cloudimagemetadata image.go:66 inserting cloud image metad...

[James Page (james-page)]

OK scratched away at this a bit further - turns out this was due to the fact that we where syncing both squashfs and root.tar.gz format images for some ubuntu series into the cloud; this resulted in multiple entry types for the lxd virt type for a particular image, which appears to have been the caused of the error in juju being unable to save the cloud image metadata.

[Ian Booth (wallyworld)]

@James, thanks for the extra info - Juju clearly needs to handle this better and surface a clear explanation of what went wrong.

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/7372

[Ian Booth (wallyworld)]

Improved the error message when attempting to save duplicate records.