Cinder

backup point in the same container was overwrite if we set the back ends to NFS

Bug #1628768 reported by mylq on 2016-09-29

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	Fix Released	High	Gorka Eguileor

Bug Description

backup point in the same container was overwrite if we set the back ends to NFS
Here is part of my cinder.conf:

[DEFAULT]
backup_driver = cinder.backup.drivers.nfs
backup_share = 192.168.2.79:/backup
backup_mount_point_base = /rhome
graceful_shutdown_timeout = 5
os_privileged_user_tenant = service
os_privileged_user_password = nomoresecret
os_privileged_user_name = nova
glance_api_servers = http://192.168.2.206:9292
osapi_volume_workers = 2
logging_exception_prefix = %(color)s%(asctime)s.%(msecs)03d TRACE %(name)s ^[[01;35m%(instance)s^[[00m
logging_debug_format_suffix = ^[[00;33mfrom (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d^[[00m
logging_default_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [^[[00;36m-%(color)s] ^[[01;35m%(instance)s%(color)s%(message)s^[[00m
logging_context_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [^[[01;36m%(request_id)s ^[[00;36m%(user_id)s %(project_id)s%(color)s] ^[[01;35m%(instance)s%(color)s%(message)s^[[00m
volume_clear = zero
backup_swift_url = http://192.168.2.206:8080/v1/AUTH_

The command I use to create a backup point is below:

curl -s -H "X-Auth-Token: $OS_TOKEN" -H "Content-Type: application/json" -X POST http://127.0.0.1:8776/v3/$OS_TENANT_ID/backups -d '{ "backup":{"container":"lq8","descript2ion":"backuptest","name":"testone","volume_id":"d17dd373-246d-4b19-bddf-0879a2bd5114","force":true,"incremental":false} }' | python -m json.tool

and we could see that in 192.168.2.79:/backup there is a backuppoint created,like this:

root@ubuntu:/backup/lq8# ll
total 7116
drwxrwx--- 2 1001 1001 4096 Sep 29 12:42 ./
drwxrwxrwx 3 root root 4096 Sep 29 12:41 ../
-rw-rw---- 1 1001 1001 4876309 Sep 29 12:42 backup-00001
-rw-rw---- 1 1001 1001 2285 Sep 29 12:42 backup_metadata
-rw-rw---- 1 1001 1001 2392348 Sep 29 12:42 backup_sha256file

if I perform the command again,and rename the backup's name with "testtwo",the command is below:

curl -s -H "X-Auth-Token: $OS_TOKEN" -H "Content-Type: application/json" -X POST http://127.0.0.1:8776/v3/$OS_TENANT_ID/backups -d '{ "backup":{"container":"lq8","descript2ion":"backuptest","name":"testtwo","volume_id":"d17dd373-246d-4b19-bddf-0879a2bd5114","force":true,"incremental":false} }' | python -m json.tool

In 192.168.2.79:/backup, a new backup point was created,but the file name of the new backup point is still "backup-00001",so the older backup point was overwrite.

root@ubuntu:/backup/lq8# ll
total 7116
drwxrwx--- 2 1001 1001 4096 Sep 29 12:42 ./
drwxrwxrwx 3 root root 4096 Sep 29 12:41 ../
-rw-rw---- 1 1001 1001 4874356 Sep 29 12:44 backup-00001
-rw-rw---- 1 1001 1001 2290 Sep 29 12:44 backup_metadata
-rw-rw---- 1 1001 1001 2392353 Sep 29 12:44 backup_sha256file

In some circumstances,for example ,if we delete the new backup point,it would be wrong to restore the older backup point.
There is the log:

2016-09-29 12:51:14.167 ERROR oslo_messaging.rpc.server [req-2a24710d-b8f0-4434-8088-818f8785f323 c460943be79446ddbc0d236b2066f05e b3962222af2a459394908673591631c7] Exception during message handling
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/backup/manager.py", line 523, in restore_backup
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server backup.save()
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/backup/manager.py", line 517, in restore_backup
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server self._run_restore(context, backup, volume)
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/backup/manager.py", line 551, in _run_restore
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server device_file)
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/backup/chunkeddriver.py", line 665, in restore
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server metadata = self._read_metadata(backup)
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/backup/chunkeddriver.py", line 253, in _read_metadata
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server with self.get_object_reader(container, filename) as reader:
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/backup/drivers/posix.py", line 124, in get_object_reader
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server return open(path, 'rb')
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server IOError: [Errno 2] No such file or directory: u'/rhome/c1ddc05eb45c4fa54c90bd043b191808/lq8/backup_metadata'
2016-09-29 12:51:14.167 TRACE oslo_messaging.rpc.server
2016-09-29 12:51:30.130 DEBUG oslo_service.periodic_task [req-2a24710d-b8f0-4434-8088-818f8785f323 c460943be79446ddbc0d236b2066f05e b3962222af2a459394908673591631c7] Running periodic task BackupManager._publish_service_capabilities from (pid=32339) run_periodic_tasks /usr/local/lib/python2.7/dist-packages/oslo_service/periodic_task.py:215

See original description

Tags:

Revision history for this message

mylq (leibnizlq) wrote on 2016-09-29:

The version of my devstack is newton,my OS is Ubuntu14.04

mylq (leibnizlq) on 2016-09-29

description:	updated
tags:	added: backup cinder

mylq (leibnizlq) on 2016-09-29

no longer affects:

swift

Eric Harney (eharney) on 2016-09-29

Changed in cinder:
assignee:	nobody → Eric Harney (eharney)
status:	New → Confirmed

Revision history for this message

Eric Harney (eharney) wrote on 2016-12-21:

Marking as "High" importance since this can trigger data loss.

Changed in cinder:
importance:	Undecided → High

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-12-21: Related fix proposed to cinder (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/413752

Changed in cinder:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-12-21: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/413753

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-11: Related fix merged to cinder (master)

Reviewed: https://review.openstack.org/413752
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=dde4f79114255ee1bcd974506b12fa6efde5c935
Submitter: Jenkins
Branch: master

commit dde4f79114255ee1bcd974506b12fa6efde5c935
Author: Eric Harney <email address hidden>
Date: Wed Dec 21 14:03:55 2016 -0500

Tests: Split up NFS backup notification test

    This test writes backup data to disk multiple times,
    which means it no longer works when bug 1628768 is
    fixed.

For now, split it into three separate tests which don't
write into the same path.

A suggested later step would be to mock out all writes
to disk.

Related-Bug: #1628768

Change-Id: I8c569d87f5a3e8e3eaf23b676ec73a412cb51d43

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-03-09: Change abandoned on cinder (master)

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/413753
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-05-18: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/465869

Changed in cinder:
assignee:	Eric Harney (eharney) → Gorka Eguileor (gorka)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-05-22: Fix merged to cinder (master)

Reviewed: https://review.openstack.org/465869
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=535e71797031c3d3e3a5e2023c5ede470b02e3a7
Submitter: Jenkins
Branch: master

commit 535e71797031c3d3e3a5e2023c5ede470b02e3a7
Author: Gorka Eguileor <email address hidden>
Date: Thu May 18 10:28:43 2017 +0200

NFS Backup: Fix overwritting backups

When using the NFS backup driver if we do multiple backups using the
same container we end up overwriting older backups.

    The issue comes from a misunderstanding in the Posix backup driver of
    the purpose of the "prefix" metadata used in the ChunkedBackupDriver
    base class.

    This prefix is for the name of the backup objects to store, but unlike
    the prefix for the volumes, here it must be unique as the base driver
    will only add numbers to identify the chunk (for the volume we add the
    volume id). Unfortunately the Posix driver just assumed that the prefix
    had the same meaning as the prefix for volumes thus making one backups
    override one another.

    This patch changes the prefix generated by the Posix driver so we have
    the following format: "volume_$VOL_ID_$TIMESTAMP_backup_$BACK_ID", thus
    allowing multiple backups in the same container.

    The new name is backward compatible with existing backups because the
    new prefix will only be used on new backups as the prefix for already
    existing backups is stored in the DB.

Change-Id: I2903c27633facde6370d95ba0b9e06025ccaef26
Closes-Bug: #1628768

Changed in cinder:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-05-23: Fix proposed to cinder (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/467262

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-05-23: Fix proposed to cinder (driverfixes/newton)

#10

Fix proposed to branch: driverfixes/newton
Review: https://review.openstack.org/467267

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-05-23: Fix proposed to cinder (driverfixes/mitaka)

#11

Fix proposed to branch: driverfixes/mitaka
Review: https://review.openstack.org/467268

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-05-23: Fix merged to cinder (driverfixes/newton)

#12

Reviewed: https://review.openstack.org/467267
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=bc1e8d6782106ec1f0fb4ebb1afa3f7269a4f063
Submitter: Jenkins
Branch: driverfixes/newton

commit bc1e8d6782106ec1f0fb4ebb1afa3f7269a4f063
Author: Gorka Eguileor <email address hidden>
Date: Thu May 18 10:28:43 2017 +0200

NFS Backup: Fix overwritting backups

When using the NFS backup driver if we do multiple backups using the
same container we end up overwriting older backups.

    The issue comes from a misunderstanding in the Posix backup driver of
    the purpose of the "prefix" metadata used in the ChunkedBackupDriver
    base class.

    This patch changes the prefix generated by the Posix driver so we have
    the following format: "volume_$VOL_ID_$TIMESTAMP_backup_$BACK_ID", thus
    allowing multiple backups in the same container.

    The new name is backward compatible with existing backups because the
    new prefix will only be used on new backups as the prefix for already
    existing backups is stored in the DB.

    Change-Id: I2903c27633facde6370d95ba0b9e06025ccaef26
    Closes-Bug: #1628768
    (cherry picked from commit 535e71797031c3d3e3a5e2023c5ede470b02e3a7)
    (cherry picked from commit 640b9dc2b739b04f1f7d70c2172f5b5fbc3b9b28)

tags:

added: in-driverfixes-newton

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-05-23: Fix merged to cinder (driverfixes/mitaka)

#13

Reviewed: https://review.openstack.org/467268
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=186a0a8a050af379cfc4c96c9baa55605c714eeb
Submitter: Jenkins
Branch: driverfixes/mitaka

commit 186a0a8a050af379cfc4c96c9baa55605c714eeb
Author: Gorka Eguileor <email address hidden>
Date: Thu May 18 10:28:43 2017 +0200

NFS Backup: Fix overwritting backups

When using the NFS backup driver if we do multiple backups using the
same container we end up overwriting older backups.

    The issue comes from a misunderstanding in the Posix backup driver of
    the purpose of the "prefix" metadata used in the ChunkedBackupDriver
    base class.

    This patch changes the prefix generated by the Posix driver so we have
    the following format: "volume_$VOL_ID_$TIMESTAMP_backup_$BACK_ID", thus
    allowing multiple backups in the same container.

    The new name is backward compatible with existing backups because the
    new prefix will only be used on new backups as the prefix for already
    existing backups is stored in the DB.

tags:

added: in-driverfixes-mitaka

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-05-24: Fix merged to cinder (stable/ocata)

#14

Reviewed: https://review.openstack.org/467262
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=640b9dc2b739b04f1f7d70c2172f5b5fbc3b9b28
Submitter: Jenkins
Branch: stable/ocata

commit 640b9dc2b739b04f1f7d70c2172f5b5fbc3b9b28
Author: Gorka Eguileor <email address hidden>
Date: Thu May 18 10:28:43 2017 +0200

NFS Backup: Fix overwritting backups

When using the NFS backup driver if we do multiple backups using the
same container we end up overwriting older backups.

    The issue comes from a misunderstanding in the Posix backup driver of
    the purpose of the "prefix" metadata used in the ChunkedBackupDriver
    base class.

    This patch changes the prefix generated by the Posix driver so we have
    the following format: "volume_$VOL_ID_$TIMESTAMP_backup_$BACK_ID", thus
    allowing multiple backups in the same container.

    The new name is backward compatible with existing backups because the
    new prefix will only be used on new backups as the prefix for already
    existing backups is stored in the DB.

    Change-Id: I2903c27633facde6370d95ba0b9e06025ccaef26
    Closes-Bug: #1628768
    (cherry picked from commit 535e71797031c3d3e3a5e2023c5ede470b02e3a7)

tags:

added: in-stable-ocata

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-06-08: Fix included in openstack/cinder 11.0.0.0b2

#15

This issue was fixed in the openstack/cinder 11.0.0.0b2 development milestone.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-06-21: Fix included in openstack/cinder 10.0.3

#16

This issue was fixed in the openstack/cinder 10.0.3 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-25: Change abandoned on cinder (master)

#17

Change abandoned by Eric Harney (<email address hidden>) on branch: master
Review: https://review.openstack.org/413753

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.