IndexError when creating Backups

[Juan Pablo Suazo]

When creating buckups, the following error is raised:

2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 366, in create_backup
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err))
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 355, in create_backup
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume)
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/manager.py", line 429, in _run_backup
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server backup, tpool.Proxy(device_file))
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/backup/chunkeddriver.py", line 599, in backup
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server if sha != parent_backup_shalist[shaindex]:
2019-08-22 13:42:08.870 8 ERROR oslo_messaging.rpc.server IndexError: list index out of range

The error stems from the following block of code inside cinder/backup/chunkeddriver.py:

      for idx, sha in enumerate(shalist):
          if sha != parent_backup_shalist[shaindex]:
              if extent_off == -1:
                  # Start of new extent.
                  extent_off = idx * self.sha_block_size_bytes
          else:
              if extent_off != -1:
                  # We've reached the end of extent.
                  extent_end = idx * self.sha_block_size_bytes
                  segment = data[extent_off:extent_end]
                  self._backup_chunk(backup, container, segment,
                                     data_offset + extent_off,
                                     object_meta,
                                     extra_metadata)
                  extent_off = -1
          shaindex += 1

The error is raised as it may happen that len(parent_backup_shalist) < len(shalist). In this case, when checking the conditional sha != parent_backup_shalist[shaindex], shaindex will be grater than the length of parent_backup_shalist at some point, raising an IndexError.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/831439

[Sofia Enriquez (lsofia-enriquez)]

Greetings,
For the sake of completeness, do you mind sharing the backup driver and the volume driver you are using as well as the steps to reproduce this?
Best Regards,
Sofia

[Sofia Enriquez (lsofia-enriquez)]

The reporter encountered the IndexError consistently while running tests for incremental backups.[1]

[1] https://review.opendev.org/c/openstack/cinder/+/831439/4/cinder/backup/chunkeddriver.py#605

[Juan Pablo Suazo (jsuazo)]

Greetings,

Volume driver used was GlusterFS and the backup driver was NFS backup driver.

Best Wishes,
Juan Pablo.