ubuntu@galaxy2:/mnt$
ubuntu@galaxy2:/mnt$ sudo dd if=/dev/zero of=/dev/mapper/vg0-lv0_snapshot_0 bs=1M count=1024 conv=fdatasync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 124.915 s, 8.6 MB/s
ubuntu@galaxy2:/mnt$
ubuntu@galaxy2:/mnt$
ubuntu@galaxy2:/mnt$ sudo lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
lv0 vg0 owi-a- 1.00g
lv0_snapshot_0 vg0 swi-a- 2.00g lv0 50.20
lv0_snapshot_1 vg0 swi-a- 1.00g lv0 0.00
ubuntu@galaxy2:/mnt$
ubuntu@galaxy2:/mnt$
ubuntu@galaxy2:/mnt$ sudo lvdisplay vg0/lv0_snapshot_0
--- Logical volume ---
LV Name /dev/vg0/lv0_snapshot_0
VG Name vg0
LV UUID KE7i1E-ZNCR-mc2Z-XNtQ-wiO3-tNjE-a8p2pT
LV Write Access read/write
LV snapshot status active destination for /dev/vg0/lv0
LV Status available
# open 0
LV Size 1.00 GiB
Current LE 256
COW-table size 2.00 GiB
COW-table LE 512
Allocated to snapshot 50.20%
Snapshot chunk size 4.00 KiB
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:1
ubuntu@galaxy2:/mnt$
### 4. zero the second snapshot
ubuntu@galaxy2:/mnt$
ubuntu@galaxy2:/mnt$
ubuntu@galaxy2:/mnt$ sudo dd if=/dev/zero of=/dev/mapper/vg0-lv0_snapshot_1 bs=1M count=1024 conv=fdatasync
dd: fdatasync failed for `/dev/mapper/vg0-lv0_snapshot_1': Input/output error
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 48.987 s, 21.9 MB/s
ubuntu@galaxy2:/mnt$
ubuntu@galaxy2:/mnt$
ubuntu@galaxy2:/mnt$
ubuntu@galaxy2:/mnt$ sudo lvs
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
lv0 vg0 owi-a- 1.00g
lv0_snapshot_0 vg0 swi-a- 2.00g lv0 50.20
lv0_snapshot_1 vg0 Swi-I- 1.00g lv0 100.00
ubuntu@galaxy2:/mnt$
ubuntu@galaxy2:/mnt$ sudo lvdisplay vg0/lv0_snapshot_1
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
--- Logical volume ---
LV Name /dev/vg0/lv0_snapshot_1
VG Name vg0
LV UUID fO6gFN-c2b5-8bQO-TsKU-or9Q-SYEy-zlxY38
LV Write Access read/write
LV snapshot status INACTIVE destination for /dev/vg0/lv0
LV Status available
# open 0
LV Size 1.00 GiB
Current LE 256
COW-table size 1.00 GiB
COW-table LE 256
Snapshot chunk size 4.00 KiB
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:4
ubuntu@galaxy2:/mnt$
ubuntu@galaxy2:/mnt$ dmesg
[2069983.216951] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
[2070028.707217] quiet_error: 136 callbacks suppressed
[2070028.707223] Buffer I/O error on device dm-4, logical block 261122
[2070028.707236] Buffer I/O error on device dm-4, logical block 261123
[2070028.707240] lost page write due to I/O error on dm-4
......
###########
we notice that it is 50.20% in Snap% after we wrote 1024MB data to the first snapshot.
2048 * 50.2% = 1028.096
1028.096 - 1024 = 4.096
I think 4.096MB is additional metadata of snapshot that need to be written to snapshot space along with 1024 MB data.
So, I think when we wrote 1024MB data to the second snapshot , data and additional data is more than snapshot size, an error occurred in device-mapper.
After we created the clone volume, we need to zero the temp_snapshot, cinder will write snapshot full and an error occurred in device-mapper.
We can reproduce it .
OS: Ubuntu 12.04.2
Kernel: 3.2.0-40-virtual
LVM:
LVM version: 2.02.66(2) (2010-05-20)
Library version: 1.02.48 (2010-05-20)
Driver version: 4.22.0
### 1. create vg
ubuntu@ galaxy2: /mnt$ galaxy2: /mnt$ sudo truncate -s 10G stack-volumes galaxy2: /mnt$ ll galaxy2: /mnt$ galaxy2: /mnt$ sudo losetup -f --show stack-volumes galaxy2: /mnt$ galaxy2: /mnt$ sudo pvcreate /dev/loop0 galaxy2: /mnt$ sudo vgcreate vg0 /dev/loop0
ubuntu@
ubuntu@
total 24
drwxr-xr-x 3 root root 4096 Jul 5 07:09 ./
drwxr-xr-x 23 root root 4096 Jul 5 04:36 ../
drwx------ 2 root root 16384 Jun 11 08:29 lost+found/
-rw-r--r-- 1 root root 10737418240 Jul 5 07:09 stack-volumes
ubuntu@
ubuntu@
/dev/loop0
ubuntu@
ubuntu@
Physical volume "/dev/loop0" successfully created
ubuntu@
Volume group "vg0" successfully created
### 2. create the origin volume and snapshots of the origin volume
ubuntu@ galaxy2: /mnt$ sudo lvcreate -L 1G -n lv0 vg0 galaxy2: /mnt$ galaxy2: /mnt$ sudo lvs galaxy2: /mnt$ galaxy2: /mnt$ sudo lvcreate -L 2G -n lv0_snapshot_0 --snapshot vg0/lv0 galaxy2: /mnt$ galaxy2: /mnt$ sudo lvcreate -L 1G -n lv0_snapshot_1 --snapshot vg0/lv0 galaxy2: /mnt$ galaxy2: /mnt$ sudo lvs galaxy2: /mnt$
Logical volume "lv0" created
ubuntu@
ubuntu@
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
lv0 vg0 -wi-a- 1.00g
ubuntu@
ubuntu@
Logical volume "lv0_snapshot_0" created
ubuntu@
ubuntu@
Logical volume "lv0_snapshot_1" created
ubuntu@
ubuntu@
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
lv0 vg0 owi-a- 1.00g
lv0_snapshot_0 vg0 swi-a- 2.00g lv0 0.00
lv0_snapshot_1 vg0 swi-a- 1.00g lv0 0.00
ubuntu@
### 3. zero the first snapshot
ubuntu@ galaxy2: /mnt$ galaxy2: /mnt$ sudo dd if=/dev/zero of=/dev/ mapper/ vg0-lv0_ snapshot_ 0 bs=1M count=1024 conv=fdatasync galaxy2: /mnt$ galaxy2: /mnt$ galaxy2: /mnt$ sudo lvs galaxy2: /mnt$ galaxy2: /mnt$ galaxy2: /mnt$ sudo lvdisplay vg0/lv0_snapshot_0 lv0_snapshot_ 0 ZNCR-mc2Z- XNtQ-wiO3- tNjE-a8p2pT
ubuntu@
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 124.915 s, 8.6 MB/s
ubuntu@
ubuntu@
ubuntu@
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
lv0 vg0 owi-a- 1.00g
lv0_snapshot_0 vg0 swi-a- 2.00g lv0 50.20
lv0_snapshot_1 vg0 swi-a- 1.00g lv0 0.00
ubuntu@
ubuntu@
ubuntu@
--- Logical volume ---
LV Name /dev/vg0/
VG Name vg0
LV UUID KE7i1E-
LV Write Access read/write
LV snapshot status active destination for /dev/vg0/lv0
LV Status available
# open 0
LV Size 1.00 GiB
Current LE 256
COW-table size 2.00 GiB
COW-table LE 512
Allocated to snapshot 50.20%
Snapshot chunk size 4.00 KiB
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:1
ubuntu@ galaxy2: /mnt$
### 4. zero the second snapshot
ubuntu@ galaxy2: /mnt$ galaxy2: /mnt$ galaxy2: /mnt$ sudo dd if=/dev/zero of=/dev/ mapper/ vg0-lv0_ snapshot_ 1 bs=1M count=1024 conv=fdatasync vg0-lv0_ snapshot_ 1': Input/output error galaxy2: /mnt$ galaxy2: /mnt$ galaxy2: /mnt$ galaxy2: /mnt$ sudo lvs galaxy2: /mnt$ galaxy2: /mnt$ sudo lvdisplay vg0/lv0_snapshot_1 lv0_snapshot_ 1 c2b5-8bQO- TsKU-or9Q- SYEy-zlxY38
ubuntu@
ubuntu@
dd: fdatasync failed for `/dev/mapper/
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 48.987 s, 21.9 MB/s
ubuntu@
ubuntu@
ubuntu@
ubuntu@
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
lv0 vg0 owi-a- 1.00g
lv0_snapshot_0 vg0 swi-a- 2.00g lv0 50.20
lv0_snapshot_1 vg0 Swi-I- 1.00g lv0 100.00
ubuntu@
ubuntu@
/dev/dm-4: read failed after 0 of 4096 at 0: Input/output error
--- Logical volume ---
LV Name /dev/vg0/
VG Name vg0
LV UUID fO6gFN-
LV Write Access read/write
LV snapshot status INACTIVE destination for /dev/vg0/lv0
LV Status available
# open 0
LV Size 1.00 GiB
Current LE 256
COW-table size 1.00 GiB
COW-table LE 256
Snapshot chunk size 4.00 KiB
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:4
ubuntu@ galaxy2: /mnt$ galaxy2: /mnt$ dmesg
ubuntu@
[2069983.216951] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
[2070028.707217] quiet_error: 136 callbacks suppressed
[2070028.707223] Buffer I/O error on device dm-4, logical block 261122
[2070028.707236] Buffer I/O error on device dm-4, logical block 261123
[2070028.707240] lost page write due to I/O error on dm-4
......
###########
we notice that it is 50.20% in Snap% after we wrote 1024MB data to the first snapshot.
2048 * 50.2% = 1028.096
1028.096 - 1024 = 4.096
I think 4.096MB is additional metadata of snapshot that need to be written to snapshot space along with 1024 MB data.
So, I think when we wrote 1024MB data to the second snapshot , data and additional data is more than snapshot size, an error occurred in device-mapper.