Comment 60 for bug 1470250

Revision history for this message
Frederik Bosch (f-bosch) wrote :

@decui @jsalisbury That is indeed good news. Since we did not make any changes to our platform yet, we can help to test possible fixes. There were also some questions.

1. Occurrence of the issue: with the fixes of bug #1445195 the amount of read-only errors reduced drastically. But it is still there. We are making backups two times a week on three Ubuntu machines, and read-only occurs I believe once a month. And when a machine goes read-only, it is always just one machine, not all three. This situation requires us to create backups during working days because then we are able to restart machines immediately, hence causing little downtime.

2. Specificness of the issue: it is indeed a Ubuntu only problem, at least for us. In our Hyper-V cluster we have four virtual machines, three Ubuntu, one CentOS. The latter never suffered from the read-only problem.

3. IO-spike issue: you are correct that we have a I/O performance downgrade. And I now for sure it is related to backups. During a week we never have any I/O problems, only those exactly two times a week when we create backups. This downgrade came up recently, at least we are feeling the downgrade now. The cause might be that more and more disk space is getting occupied. Or that was a (kernel) update that is causing this issue. But my guess (and it is really a guess) is that is caused by the disk space and therefore the amount of data to backup.

Regarding our system. Since the fixes for bug #1445195 were released, we switched back to the latest 3.13 kernel in our production environments. But if I need to switch kernel to help fixing the bug, I think that would be no problem. Luckily we can now, with the work of @jsalisbury, do this without problems on a testing machine. Thank you for the work so far already!