after some 1-2 weeks of use glance starts to send garbage to the db and finally glance won't work anymore until a reinitialization takes place (drop db, sync_db, re upload imgs). on my production db it started to happen today again
i'm using postgresql 9.1, python-psycopg2 2.4.5-1, havana release of glance
postgres log
Feb 4 08:39:02 infra01 postgres[30429]: [4-1] 2014-02-04 08:39:02 ART ERROR: invalid input syntax for type timestamp: "" at character 4367
Feb 4 08:39:02 infra01 postgres[30429]: [4-2] 2014-02-04 08:39:02 ART STATEMENT: SELECT anon_1.anon_2_images_created_at AS anon_1_anon_2_images_created_at, anon_1.anon_2_images_updated_at AS anon_1_anon_2_images_updated_at, anon_1.anon_2_images_deleted_at AS anon_1_anon_2_images_deleted_at, anon_1.anon_2_images_deleted AS anon_1_anon_2_images_deleted, anon_1.anon_2_images_id AS anon_1_anon_2_images_id, anon_1.anon_2_images_name AS anon_1_anon_2_images_name, anon_1.anon_2_images_disk_format AS anon_1_anon_2_images_disk_format, anon_1.anon_2_images_container_format AS anon_1_anon_2_images_container_format, anon_1.anon_2_images_size AS anon_1_anon_2_images_size, anon_1.anon_2_images_status AS anon_1_anon_2_images_status, anon_1.anon_2_images_is_public AS anon_1_anon_2_images_is_public, anon_1.anon_2_images_checksum AS anon_1_anon_2_images_checksum, anon_1.anon_2_images_min_disk AS anon_1_anon_2_images_min_disk,
Feb 4 08:39:02 infra01 postgres[30429]: [4-3] anon_1.anon_2_images_min_ram AS anon_1_anon_2_images_min_ram, anon_1.anon_2_images_owner AS anon_1_anon_2_images_owner, anon_1.anon_2_images_protected AS anon_1_anon_2_images_protected, image_properties_1.created_at AS image_properties_1_created_at, image_properties_1.updated_at AS image_properties_1_updated_at, image_properties_1.deleted_at AS image_properties_1_deleted_at, image_properties_1.deleted AS image_properties_1_deleted, image_properties_1.id AS image_properties_1_id, image_properties_1.image_id AS image_properties_1_image_id, image_properties_1.name AS image_properties_1_name, image_properties_1.value AS image_properties_1_value, image_locations_1.created_at AS image_locations_1_created_at, image_locations_1.updated_at AS image_locations_1_updated_at, image_locations_1.deleted_at AS image_locations_1_deleted_at, image_locations_1.deleted AS image_locations_1_deleted,
Feb 4 08:39:02 infra01 postgres[30429]: [4-4] image_locations_1.id AS image_locations_1_id, image_locations_1.image_id AS image_locations_1_image_id, image_locations_1.value AS image_locations_1_value, image_locations_1.meta_data AS image_locations_1_meta_data
Feb 4 08:39:02 infra01 postgres[30429]: [4-5] #011FROM (SELECT anon_2.images_created_at AS anon_2_images_created_at, anon_2.images_updated_at AS anon_2_images_updated_at, anon_2.images_deleted_at AS anon_2_images_deleted_at, anon_2.images_deleted AS anon_2_images_deleted, anon_2.images_id AS anon_2_images_id, anon_2.images_name AS anon_2_images_name, anon_2.images_disk_format AS anon_2_images_disk_format, anon_2.images_container_format AS anon_2_images_container_format, anon_2.images_size AS anon_2_images_size, anon_2.images_status AS anon_2_images_status, anon_2.images_is_public AS anon_2_images_is_public, anon_2.images_checksum AS anon_2_images_checksum, anon_2.images_min_disk AS anon_2_images_min_disk, anon_2.images_min_ram AS anon_2_images_min_ram, anon_2.images_owner AS anon_2_images_owner, anon_2.images_protected AS anon_2_images_protected
Feb 4 08:39:02 infra01 postgres[30429]: [4-6] #011FROM (SELECT images.created_at AS images_created_at, images.updated_at AS images_updated_at, images.deleted_at AS images_deleted_at, images.deleted AS images_deleted, images.id AS images_id, images.name AS images_name, images.disk_format AS images_disk_format, images.container_format AS images_container_format, images.size AS images_size, images.status AS images_status, images.is_public AS images_is_public, images.checksum AS images_checksum, images.min_disk AS images_min_disk, images.min_ram AS images_min_ram, images.owner AS images_owner, images.protected AS images_protected
Feb 4 08:39:02 infra01 postgres[30429]: [4-7] #011FROM images
Feb 4 08:39:02 infra01 postgres[30429]: [4-8] #011WHERE images.is_public = true OR images.owner = 'af41371701004068b2532695ed57f16f' UNION SELECT images.created_at AS images_created_at, images.updated_at AS images_updated_at, images.deleted_at AS images_deleted_at, images.deleted AS images_deleted, images.id AS images_id, images.name AS images_name, images.disk_format AS images_disk_format, images.container_format AS images_container_format, images.size AS images_size, images.status AS images_status, images.is_public AS images_is_public, images.checksum AS images_checksum, images.min_disk AS images_min_disk, images.min_ram AS images_min_ram, images.owner AS images_owner, images.protected AS images_protected
Feb 4 08:39:02 infra01 postgres[30429]: [4-9] #011FROM images JOIN image_members ON images.id = image_members.image_id
Feb 4 08:39:02 infra01 postgres[30429]: [4-10] #011WHERE image_members.deleted = false AND image_members.member = 'af41371701004068b2532695ed57f16f') AS anon_2
Feb 4 08:39:02 infra01 postgres[30429]: [4-11] #011WHERE anon_2.images_deleted = false AND anon_2.images_status != 'killed' AND (CASE WHEN (anon_2.images_created_at IS NOT NULL) THEN anon_2.images_created_at ELSE '' END < '2014-01-28T16:08:16.845874'::timestamp OR CASE WHEN (anon_2.images_created_at IS NOT NULL) THEN anon_2.images_created_at ELSE '' END = '2014-01-28T16:08:16.845874'::timestamp AND CASE WHEN (anon_2.images_id IS NOT NULL) THEN anon_2.images_id ELSE '' END < '38fe03a3-8ef2-4445-8754-585d3e84b2ac') ORDER BY anon_2.images_created_at DESC, anon_2.images_id DESC
Feb 4 08:39:02 infra01 postgres[30429]: [4-12] #011 LIMIT 10) AS anon_1 LEFT OUTER JOIN image_properties AS image_properties_1 ON anon_1.anon_2_images_id = image_properties_1.image_id LEFT OUTER JOIN image_locations AS image_locations_1 ON anon_1.anon_2_images_id = image_locations_1.image_id ORDER BY anon_1.anon_2_images_created_at DESC, anon_1.anon_2_images_id DESC
$ glance index
ID Name Disk Format Container Format Size
------------------------------------ ------------------------------ -------------------- -------------------- --------------
86d2fa74-aa86-435c-bce8-84a3acf12000 web2py-testing-environment qcow2 ovf 1401683968
ba234ba1-6ab6-45b1-a820-cfb0811a85c1 Win2008r2 qcow2 ovf 3280732160
af19bcb6-2621-46b4-bb4e-bb33a52dd8f8 Win7x86 qcow2 ovf 3374317568
b3ca4b0d-df9d-4f0c-8518-7bbe9e173a2d Win2012 qcow2 ovf 4284743680
11bf41ef-2478-49d9-9262-f751c1f37e13 Win7x64 qcow2 ovf 4608753664
cc75052c-3d5f-4774-ab50-abcf82554e60 Win2012-sql qcow2 ovf 13241942016
d89e01c0-b862-418f-a19d-66b504a6ff04 mojo-testing-environment ami ami 3125346304
bee5e021-3786-441b-9133-e590a78b21de wheezy-14-01-17 raw bare 996999168
b7d0db05-0b86-4707-8511-7f99a2fc32ea initrd-mockup ari ari 17541543
38fe03a3-8ef2-4445-8754-585d3e84b2ac kernel-mockup aki aki 4100880
Request returned failure status.
HTTPInternalServerError (HTTP 500)
a diff of the db's dump btw the time of the error and the previous day shows no differences