Activity log for bug #1869746

Date Who What changed Old value New value Message
2020-03-30 15:19:24 Rodrigo Barbieri bug added bug
2020-03-30 15:20:09 Rodrigo Barbieri description Environment: Ubuntu Bionic, OpenStack Train, Volume and Backup Driver Ceph Steps to reproduce: 1) Create a volume v1 as user demo 2) Create a full backup of v1 as user admin 3) Create an incremental backup of v1 as user demo 4) Result: Backup is created successfully, but error in logs. Logs do not show INFO message and parent_backup.num_dependent_backups is not incremented [0]. Expectation: Either no error successfully finishing method create_backup, or not creating backup at all, with backup entity set as Error (see below). Using POSIX driver, the backup is not created successfully and backup entity remains in error state. The POSIX driver tries to read backup.parent_id before creating the backup therefore raises the error earlier. Trace: 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server [req-19fccda2-b765-4b68-9663-aeae5cf3002e 3bee8c02ecce4744af287b3b8dd40ca5 9370e1e898214d88bd9eda4f88eccba9 - f883c6315a004471b2ab1b04d8a111d9 f883c6315a004471b2ab1b04d8a111d9] Exception during message handling: cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a could not be found. 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 447, in create_backup 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server backup.parent_id) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/objects/base.py", line 352, in get_by_id 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/api.py", line 1801, in get_by_id 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return IMPL.get_by_id(context, model, id, *args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 7125, in get_by_id 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return _GET_METHODS[model](context, id, *args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5154, in backup_get 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server project_only=project_only) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5165, in _backup_get 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server raise exception.BackupNotFound(backup_id=backup_id) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a could not be found. 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server [0] https://github.com/openstack/cinder/blob/e79b98367bd72f0323258dc0506de28b31c4aa48/cinder/backup/manager.py#L448 Environment: Ubuntu Bionic, OpenStack Train, Volume and Backup Driver Ceph This bug was also reproduced on Queens. Steps to reproduce: 1) Create a volume v1 as user demo 2) Create a full backup of v1 as user admin 3) Create an incremental backup of v1 as user demo 4) Result: Backup is created successfully, but error in logs. Logs do not show INFO message and parent_backup.num_dependent_backups is not incremented [0]. Expectation: Either no error successfully finishing method create_backup, or not creating backup at all, with backup entity set as Error (see below). Using POSIX driver, the backup is not created successfully and backup entity remains in error state. The POSIX driver tries to read backup.parent_id before creating the backup therefore raises the error earlier. Trace: 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server [req-19fccda2-b765-4b68-9663-aeae5cf3002e 3bee8c02ecce4744af287b3b8dd40ca5 9370e1e898214d88bd9eda4f88eccba9 - f883c6315a004471b2ab1b04d8a111d9 f883c6315a004471b2ab1b04d8a111d9] Exception during message handling: cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a could not be found. 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 447, in create_backup 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server backup.parent_id) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/objects/base.py", line 352, in get_by_id 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/api.py", line 1801, in get_by_id 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return IMPL.get_by_id(context, model, id, *args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 7125, in get_by_id 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return _GET_METHODS[model](context, id, *args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5154, in backup_get 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server project_only=project_only) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5165, in _backup_get 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server raise exception.BackupNotFound(backup_id=backup_id) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a could not be found. 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server [0] https://github.com/openstack/cinder/blob/e79b98367bd72f0323258dc0506de28b31c4aa48/cinder/backup/manager.py#L448
2020-04-02 20:25:11 Sofia Enriquez cinder: status New Confirmed
2020-04-15 10:50:49 Rodrigo Barbieri tags sts
2020-04-18 13:54:51 Rodrigo Barbieri cinder: status Confirmed In Progress
2020-04-18 13:54:55 Rodrigo Barbieri cinder: assignee Rodrigo Barbieri (rodrigo-barbieri2010)
2020-05-20 14:27:23 Brian Rosmaita cinder: milestone victoria-1
2020-07-01 19:05:34 OpenStack Infra cinder: status In Progress Fix Released
2020-07-02 20:40:01 OpenStack Infra tags sts in-stable-ussuri sts
2020-07-06 20:50:33 OpenStack Infra tags in-stable-ussuri sts in-stable-train in-stable-ussuri sts
2020-07-20 21:39:55 OpenStack Infra tags in-stable-train in-stable-ussuri sts in-stable-stein in-stable-train in-stable-ussuri sts
2020-07-24 11:58:08 OpenStack Infra tags in-stable-stein in-stable-train in-stable-ussuri sts in-stable-rocky in-stable-stein in-stable-train in-stable-ussuri sts
2020-07-24 16:08:42 Rodrigo Barbieri description Environment: Ubuntu Bionic, OpenStack Train, Volume and Backup Driver Ceph This bug was also reproduced on Queens. Steps to reproduce: 1) Create a volume v1 as user demo 2) Create a full backup of v1 as user admin 3) Create an incremental backup of v1 as user demo 4) Result: Backup is created successfully, but error in logs. Logs do not show INFO message and parent_backup.num_dependent_backups is not incremented [0]. Expectation: Either no error successfully finishing method create_backup, or not creating backup at all, with backup entity set as Error (see below). Using POSIX driver, the backup is not created successfully and backup entity remains in error state. The POSIX driver tries to read backup.parent_id before creating the backup therefore raises the error earlier. Trace: 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server [req-19fccda2-b765-4b68-9663-aeae5cf3002e 3bee8c02ecce4744af287b3b8dd40ca5 9370e1e898214d88bd9eda4f88eccba9 - f883c6315a004471b2ab1b04d8a111d9 f883c6315a004471b2ab1b04d8a111d9] Exception during message handling: cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a could not be found. 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 447, in create_backup 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server backup.parent_id) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/objects/base.py", line 352, in get_by_id 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/api.py", line 1801, in get_by_id 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return IMPL.get_by_id(context, model, id, *args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 7125, in get_by_id 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return _GET_METHODS[model](context, id, *args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5154, in backup_get 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server project_only=project_only) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5165, in _backup_get 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server raise exception.BackupNotFound(backup_id=backup_id) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a could not be found. 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server [0] https://github.com/openstack/cinder/blob/e79b98367bd72f0323258dc0506de28b31c4aa48/cinder/backup/manager.py#L448 Environment: Ubuntu Bionic, OpenStack Train, Volume and Backup Driver Ceph This bug was also reproduced on Queens. Steps to reproduce: 1) Create a volume v1 as user demo 2) Create a full backup of v1 as user admin 3) Create an incremental backup of v1 as user demo 4) Result: Backup is created successfully, but error in logs. Logs do not show INFO message and parent_backup.num_dependent_backups is not incremented [0]. Expectation: Either no error successfully finishing method create_backup, or not creating backup at all, with backup entity set as Error (see below). Using POSIX driver, the backup is not created successfully and backup entity remains in error state. The POSIX driver tries to read backup.parent_id before creating the backup therefore raises the error earlier. Trace: 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server [req-19fccda2-b765-4b68-9663-aeae5cf3002e 3bee8c02ecce4744af287b3b8dd40ca5 9370e1e898214d88bd9eda4f88eccba9 - f883c6315a004471b2ab1b04d8a111d9 f883c6315a004471b2ab1b04d8a111d9] Exception during message handling: cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a could not be found. 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 447, in create_backup 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server backup.parent_id) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/objects/base.py", line 352, in get_by_id 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/api.py", line 1801, in get_by_id 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return IMPL.get_by_id(context, model, id, *args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 7125, in get_by_id 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return _GET_METHODS[model](context, id, *args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server return f(*args, **kwargs) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5154, in backup_get 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server project_only=project_only) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/db/sqlalchemy/api.py", line 5165, in _backup_get 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server raise exception.BackupNotFound(backup_id=backup_id) 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server cinder.exception.BackupNotFound: Backup 7a15b9ec-7648-483f-ad32-6074567eca2a could not be found. 2020-03-30 14:53:21.341 25018 ERROR oslo_messaging.rpc.server [0] https://github.com/openstack/cinder/blob/e79b98367bd72f0323258dc0506de28b31c4aa48/cinder/backup/manager.py#L448 ==================================================================== [Impact] Whenever an admin creates a backup for a tenant volume, all subsequent incremental backups will have a silent error during creation when linking the dependent backup to its parent because the code detects it has a parent (by listing backups from all projects) but is unable to find the DB entity while scoped to only its project. Upon deleting the child backup, it will hit another silent error (because of lack of link) and it will skip cleaning up the quota for the backup. The quota will then misrepresent the number and size of backups, and the discrepancy will increase as this error is hit for several backups. Ultimately, the user will hit a quota error with a lower number/size of backups than the actual quota limit. The fix addressed the problem by changing the code that searches for the parent backup. The code will now only search for backups under the same project, so it will prevent the error from happening. [Test case] 1) Reproducing the bug: 1a) Create a volume v1 as user demo 1b) Create a full backup b1 of v1 as user admin 1c) Create an incremental backup b2 of v1 as user demo 1d) Confirm backup b2 is created successfully 1e) Check log file "/var/log/cinder/cinder-backup.log" for error message below with b2's ID. cinder.exception.BackupNotFound: Backup <b2_id> could not be found 2) Cleanup not necessary 3) Install package that contains fixed code 4) Confirm bug is fixed 4a) Repeat steps 1a to 1d, make sure to create a new volume and new backups from it. 4b) Check log file "/var/log/cinder/cinder-backup.log" for error messages related to new b2 ID. There shouldn't be any. [Regression Potential] The fix has already been validated in upstream CI with new functional tests that perform that specific workflow (however, checks API backup field instead of logs), as far back as Rocky release. The code changes affect the creation step of backups, and both the new and previously existing functional tests thoroughly test the creation step in different ways, therefore regressions with this patch are not expected. [Other Info] For releases older than Train, the fix for bug 1809323 needs to be included as a dependency.
2020-07-24 16:09:47 Rodrigo Barbieri bug task added cinder (Ubuntu)
2020-07-24 16:10:16 Rodrigo Barbieri bug task added cloud-archive
2020-07-24 16:43:04 Edward Hope-Morley nominated for series cloud-archive/stein
2020-07-24 16:43:04 Edward Hope-Morley bug task added cloud-archive/stein
2020-07-24 16:43:04 Edward Hope-Morley nominated for series cloud-archive/victoria
2020-07-24 16:43:04 Edward Hope-Morley bug task added cloud-archive/victoria
2020-07-24 16:43:04 Edward Hope-Morley nominated for series cloud-archive/train
2020-07-24 16:43:04 Edward Hope-Morley bug task added cloud-archive/train
2020-07-24 16:43:04 Edward Hope-Morley nominated for series cloud-archive/ussuri
2020-07-24 16:43:04 Edward Hope-Morley bug task added cloud-archive/ussuri
2020-07-28 07:11:13 Chris MacNaughton cloud-archive/stein: status New Fix Committed
2020-07-28 07:11:17 Chris MacNaughton cloud-archive/ussuri: status New Fix Committed
2020-07-28 07:11:20 Chris MacNaughton cloud-archive/victoria: status New Fix Committed
2020-07-28 07:11:25 Chris MacNaughton cloud-archive/train: status New Fix Committed