/dev/.initramfs/progress_state is corrupted at shutdown

[Martin Schwenke]

Sometimes shutdown fails with the message:

/dev/.initramfs/progress_state: 1: Syntax error: Unterminated quoted string error: '/etc/init.d/rc' exited outside the expected flow control.

The only thing that writes to this file appears to be the update_progress() function in /usr/share/initramfs-tools/scripts/functions. I've even unpacked the initramfs for the current kernel to make sure that file isn't somehow corrupted. It looks fine.

So /dev/.initramfs/progress_state ends up in an unusual state. Could this be a race condition? Is anything happening in parallel at shutdown? Could the file be partially written when it is sourced by /etc/init.d/rc?

For now I've added something that prints the contents of /dev/.initramfs/progress_state before it is sourced by /etc/init.d/rc. However, if this is a race condition, I don't hold out much hope of actually seeing anything useful any time soon...

So, any ideas? :-)

[Martin Schwenke (martin-meltin)]

This happened again. Here are the contents of /dev/.initramfs/progress_state:

"0(p'-KA

Meaningless nonsense! Actually, the last 2 characters were some sort of scripted K (kappa?) and a scripted A (?), probably form some distant part of the console font.

Whatever the case, the file contains rubbish. However, the rubbish is somewhat consistent because the "Unterminated quoted string error" is common.

So, I'm wondering what this is? Filesystem corruption? The result of a bizarre race condition? Memory corruption?

I'm betting against a hardware problem. I'm not seeing problems at other times, it seems relatively consistent and this only started occurring when I upgraded to Feisty. However, I've seen less likely coincidences.... :-)

Ideas?

Thanks...

[Juho L (leppis)]

I'm having the same problem in one of my two Feisty computers. Progress_state gets corrupted sometimes, about on every fifth shutdown. The problem appeared a few weeks ago.

The problem appears in amd64 Feisty.

[Martin Schwenke (martin-meltin)]

I wonder if this is semi-random corruption related to Bug #96715? I've uninstalled evms and the swap space problem has gone away. Juho, when you do:

  cat /proc/swaps

do you see the same partition used twice (once via /dev/mapper)?

This bug is probably going to be very hard to debug... so this is a wild guess... but repeatable corruption has to be coming from somewhere. So I'm wondering if it happens when swapoff interacts with something else?

Let's trawl through swap space for the above junk (which I have now seen several times):

  $ sudo dd if=/dev/sda2 | hexdump -C | grep '0(p'
  01f35000 00 22 00 30 28 70 00 00 01 27 2d 4b 00 00 05 03 |.".0(p...'-K....|

Apart from the non-printables, that looks familiar!

Now, change the grep to less and browsing backwards, the previous non-NUL item in swap is:

  01f34000 50 52 4f 47 52 45 53 53 5f 53 54 41 54 45 3d 33 |PROGRESS_STATE=3|

That is *way* too much of a coincidence. There's obviously something weird happening with swap...

I wonder if you have the double-swap issue and if the file corruption goes away completely if you resolve the double-swap issue?

[Juho L (leppis)]

Yes, my both swap partitions are used twice in my amd64, once directly and once via /dev/mapper. This is also the case for my i386. I'll have to try uninstalling evms to see if it solves the case...

[Martin Schwenke (martin-meltin)]

Hi Juho. Just wondering if you've fixed the swap problem and whether this has caused the corruption on shutdown to go away. Unfortunately I've had to retired the machine where I was seeing the problem, so I can no longer test for this problem.

It would be nice to if we were able to add some information to Bug #96715 to convince someone that the importance should change from "undecide" to "real corruption has been observed so this should be fixed right now". :-)

[xteejx (xteejx)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. You reported this bug a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue for you. Can you try with the latest Ubuntu release? Thanks in advance.

[Martin Schwenke (martin-meltin)]

I don't think I've seen this problem since Gutsy.

I think it was a race connected to the evms-induced double swap bug. evms seems to be gone and I don't think feisty is supported anymore, so I think this should be closed.

[xteejx (xteejx)]

Closing due to previous comment about this being fixed