glance mirror sync should not delete images not matched by item_filters

[Alexandre Gomes]

Currently glance mirror image sync is deleting images that aren't in the item_filters. From the log below we can see that disco-arm64 has been filtered out of creation but not from deletion:

DEBUG * 06-27 21:35:28 [PID:1872394] * sstreams * Filtered out item: squashfs.manifest/{u'size': 13623, u'path': u'server/releases/disco/release-20190627/ubuntu-19.04-server-cloudimg-arm64.squashfs.manifest', u'ftype': u'squashfs.manifest'}
DEBUG * 06-27 21:35:28 [PID:1872394] * keystoneauth.session * REQ: curl -g -i --cacert "/home/ubuntu/ca.pem" -X DELETE https://...:9292/v2/images/a1f83d58-66fb-4830-a6bc-db3ae7dfbbc9 -H "User-Agent: python-glanceclient" -H "Content-Type: application/octet-stream"
DEBUG * 06-27 21:35:29 [PID:1872394] * urllib3.connectionpool * https://10.48.7.15:9292 "DELETE /v2/images/a1f83d58-66fb-4830-a6bc-db3ae7dfbbc9 HTTP/1.1" 204 0

I believe item_filters should be used on deletion too.

[Dan Watkins (oddbloke)]

Hi Alexandre,

Thanks for the bug report! It's not entirely clear to me what I'm seeing here, or how I could reproduce it. Could you provide more details of how I could go about reproducing what you're seeing? Also, could you confirm that this isn't a bug in the glance-simplestreams-sync charm, rather than the simplestreams Python library that it utilises? Once you've provided this information, please do set the bug status back to New.

Thanks!

Dan

[Alexandre Gomes (alejdg)]

Hi Dan,

I don't think the bug is in g-s-s charm as it just calls for a sync using simplestreams and passing item_filters used in https://git.launchpad.net/simplestreams/tree/simplestreams/mirrors/glance.py#n131

I didn't went too deep in the code but I think that the param `src` is different when calling `remove_item`[1] and _insert_item[2], maybe due to insert_version[3]?

Explaining better the logs, the image a1f83d58-66fb-4830-a6bc-db3ae7dfbbc9 in the DELETE line is the same that was filtered out in the first line of the logs "server/releases/disco/release-20190627/ubuntu-19.04-server-cloudimg-arm64.squashfs.manifest". I don't think simplestreams should remove an image that it's not "responsible for".

[1]https://git.launchpad.net/simplestreams/tree/simplestreams/mirrors/glance.py#n581,
[2]https://git.launchpad.net/simplestreams/tree/simplestreams/mirrors/glance.py#n426
[3]https://git.launchpad.net/simplestreams/tree/simplestreams/mirrors/glance.py#n556

Please let me know if you need any extra info.

Thanks,
Alexandre

[Dan Watkins (oddbloke)]

Hi Alexandre,

I'm still not 100% sure I follow along here. "server/releases/disco/release-20190627/ubuntu-19.04-server-cloudimg-arm64.squashfs.manifest" is not an image, it's a manifest file (as identified by the ftype "squashfs.manifest"). I think we would expect it to always be filtered out when considering an image sync, because it isn't an image.

So it's still not entirely clear to me what the issue here is, I'm afraid. Am I missing something?

Dan