File system corruption in containers on master node
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
High
|
Matthew Mosesohn | ||
5.1.x |
Fix Released
|
High
|
Matthew Mosesohn | ||
6.0.x |
Fix Released
|
High
|
Matthew Mosesohn | ||
6.1.x |
Fix Released
|
High
|
Matthew Mosesohn |
Bug Description
Several customers reported file system corruption in docker containers.
It could be related https:/
I have managed to reproduce it by filling free space on master node
"fallocate -l <big number> /var/BIG_FILE"
and then
"docketctl backup".
log:
EXT4-fs error (device dm-10): ext4_put_super: Couldn't clean up the journal
...
Buffer I/O error on device dm-10, logical block 1678800
lost page write due to I/O error on dm-10
Buffer I/O error on device dm-10, logical block 1678801
lost page write due to I/O error on dm-10
Buffer I/O error on device dm-10, logical block 1678802
lost page write due to I/O error on dm-10
Buffer I/O error on device dm-10, logical block 1678803
lost page write due to I/O error on dm-10
Buffer I/O error on device dm-10, logical block 1647360
lost page write due to I/O error on dm-10
Buffer I/O error on device dm-10, logical block 1647361
lost page write due to I/O error on dm-10
Buffer I/O error on device dm-10, logical block 1647362
lost page write due to I/O error on dm-10
Buffer I/O error on device dm-10, logical block 1647363
lost page write due to I/O error on dm-10
Buffer I/O error on device dm-10, logical block 1647364
lost page write due to I/O error on dm-10
Buffer I/O error on device dm-10, logical block 1647365
lost page write due to I/O error on dm-10
JBD2: Detected IO errors while flushing file data on dm-10-8
__ratelimit: 512 callbacks suppressed
The other report is here:
http://
Changed in fuel: | |
assignee: | nobody → Matthew Mosesohn (raytrac3r) |
importance: | Undecided → Medium |
status: | New → Confirmed |
milestone: | none → 6.0 |
Changed in fuel: | |
importance: | Medium → High |
tags: | added: customer-found |
Changed in fuel: | |
assignee: | Matthew Mosesohn (raytrac3r) → Meg McRoberts (dreidellhasa) |
Changed in fuel: | |
assignee: | Meg McRoberts (dreidellhasa) → Matthew Mosesohn (raytrac3r) |
Changed in fuel: | |
milestone: | 6.0 → 6.1 |
status: | In Progress → Triaged |
tags: | added: release-notes |
no longer affects: | fuel/6.0.x |
Changed in fuel: | |
status: | Triaged → In Progress |
Changed in fuel: | |
milestone: | 6.0 → 6.1 |
no longer affects: | fuel/6.1.x |
tags: | added: docs |
no longer affects: | fuel/6.1.x |
Changed in fuel: | |
status: | Confirmed → Won't Fix |
tags: | added: release-notes-done |
tags: | added: support |
Evgeniy, did you try these steps:
1 - clear space on the disk
2 - reboot
3 - dockerctl check all
I can't seem to get it to fail after that. I was hoping you had a scenario where a container FS was broken or a single container broke so much it couldn't be found.