The generated binaries always crash

[Marco Maggesi]

Binary package hint: cryopid

All checkpointed binaries generated with Cryopid crash immediately with "Segmentation fault".

My setup:
Ubuntu 8.04
Cryopid 0.5.9.1-4build1
Kernel: installed from pkg linux-image-2.6.24-16-server
CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz

Here is an example session (the sources of the example program "ciao" used for this test are attached).

maggesi@flipper:~$ ./ciao
PID = 14098
1
2
...

<<switch terminal>>

maggesi@flipper:~$ freeze ciao.dump 14098
Map: 08048000-08049000 r-xp 00000000 08:01 427799 /home/maggesi/ciao
Map: 08049000-0804a000 rw-p 00000000 08:01 427799 /home/maggesi/ciao
Map: b7e78000-b7e79000 rw-p b7e78000 00:00 0 (null)
[+] Found scribble zone: 0xb7e78000
Map: b7e79000-b7fc2000 r-xp 00000000 08:01 2665633 /lib/tls/i686/cmov/libc-2.7.so
[+] Found a syscall location at 0xb7ea4125
Map: b7fc2000-b7fc3000 r--p 00149000 08:01 2665633 /lib/tls/i686/cmov/libc-2.7.so
Map: b7fc3000-b7fc5000 rw-p 0014a000 08:01 2665633 /lib/tls/i686/cmov/libc-2.7.so
Map: b7fc5000-b7fc8000 rw-p b7fc5000 00:00 0 (null)
Map: b7fe0000-b7fe3000 rw-p b7fe0000 00:00 0 (null)
Map: b7fe3000-b7fe4000 r-xp b7fe3000 00:00 0 [vdso]
Map: b7fe4000-b7ffe000 r-xp 00000000 08:01 2632683 /lib/ld-2.7.so
Map: b7ffe000-b8000000 rw-p 00019000 08:01 2632683 /lib/ld-2.7.so
Map: bff1b000-bff30000 rw-p bffeb000 00:00 0 [stack]
[+] Terminal device appears to be 136:1
Saved console chunk (0).
Saved console chunk (1).
Saved console chunk (2).
Heap was at 0x12900. Want to be at 0xb7e78000. offset = 0xb7e65700
Compressed 147059 bytes into 12246 bytes (92% compression)
Written image.
maggesi@flipper:~$ ./ciao.dump
Segmentation fault
maggesi@flipper:~$

[Ralph Corderoy (ralph-inputplus)]

I've independently discovered this and then found this bug. Confirming. I'm using exactly the same version of cryopid as Marco on x86. Makes no difference whether the -l option is given.

[Ralph Corderoy (ralph-inputplus)]

I've found `strace /tmp/foo.frozen' is a good means of debugging. Here, I find one of the first things foo.frozen attempts, regardless of what program was frozen, is

    mmap2(0x1000, 57344, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, 0, 0) = -1 EACCES (Permission denied)

and we suffer a SIGSEGV. Running it as `sudo strace /tmp/foo.frozen' means it gets past this and can either successfully run, or crash for program-specific reasons.

[Gabor Szabo (gabor-szabo)]

I've experienced this. This happens only in Ubuntu 8.04 though, apparently due to newer kernels. The binary generated by freeze works OK with older kernels (definitely with 2.6.20-15-386), even if it was saved with a new kernel (2.6.24-16-generic).

It may be that (1) the kernel config was changed between the two kernel versions, or that (2) freeze just doesn't work with the new kernels. What a pity it would be....

[Gabor Szabo (gabor-szabo)]

I figured out that the binaries crash only with certain kernel configs. In particular, I changed a few things in the Processor type and features section of the kernel config and now the generated binaries restart all right. I've changed three or four things at the same time and am not sure which one is responsible for the behavior, but attached both a working and a not working configuration. The kernel I'm using now is 2.6.25.7 but it's the same with some older kernels as well (2.6.20.15 certainly).

I've found out about this by comparing the kernel configs for the kernels in 7.04 (binaries working) and 8.04 (not working). If anybody is willing to narrow down the kernel option causing this, they're more than welcome.

[Gabor Szabo (gabor-szabo)]

I attached a wrong .config file above, this is the one that freeze binaries truly work with. This one turns the Tickless system & High resolution timer support options off. It seems this is really necessary for the binaries to work.

[BernardB (b-launchpad)]

Some mmap restrictions recently went into the kernel to help mitigate null-pointer deref bugs and other classes of exploits.

Running sudo sh -c 'echo 4096 > /proc/sys/vm/mmap_min_addr' should allow cryopid to get past this stumbling block.

Alternately, the resumer code should be put at a higher address. Ubuntu ships with a default mmap_min_addr of 64K (0x10000). It shouldn't hurt to put the resumer there instead.

[Luca Falavigna (dktrkranz)]

Could you please check if recent kernels work with cryopid?

[Luca Falavigna (dktrkranz)]

cryopid has been recently removed from Ubuntu and Debian, so I'm closing ths bug report for now. Please reopen if you can provide further informations. Thank you in advance!

[BernardB (b-launchpad)]

In addition to the mmap_min_addr change above, you also need to set /proc/sys/vm/vdso_enabled to either 0 (no vDSO) or 2 (compatibility vDSO mode - mapping to 0xffffxxxx). I'd recommend setting it to 2 for performance.

The default is 1 which maps the vDSO page somewhere in the user process's address space, and that's not being restored correctly. I'm not entirely sure what it's unhappy about it still, but at least this suffice as a workaround for now. (Additionally, you can actually save a image with the default of vdso_enabled=1, so long as you restore it with vdso_enabled=2).