ghost file engine gone haywire

[Mark Gavalda]

Trying to do a live migration with the latest LXD/LXC package on Ubuntu 16.04, CRIU version 2.0-2ubuntu3:

mg@instance-3:~$ sudo lxc copy eXc-testsite instance-4:eXc-testsite
error: Error transferring container data: migration restore failed
(01.036974) 690: Error (files-reg.c:1196): Can't link var/lib/logstash-forwarder/.logstash-forwarder.cr.8.ghost -> var/lib/logstash-forwarder/.logstash-forwarder: File exists
(01.127565) 547: Error (files.c:594): 547: Can't setup 14 file owner pid: No such process
(01.248507) Error (cr-restore.c:1352): 6160 killed by signal 9
(01.270902) Error (files-reg.c:516): `- XFail [/tmp/.ZendSem.bQSNBE.cr.1.ghost] ghost: No such file or directory
(01.270911) Error (files-reg.c:516): `- XFail [/tmp/.ZendSem.kTyvBE.cr.2.ghost] ghost: No such file or directory
(01.270961) Error (files-reg.c:516): `- XFail [.criu.mntns.1WVK2W/15run/mysqld/ibfeQIB0.cr.3.ghost] ghost: No such file or directory
(01.270970) Error (files-reg.c:516): `- XFail [.criu.mntns.1WVK2W/15run/mysqld/ibwOJFNR.cr.4.ghost] ghost: No such file or directory
(01.270975) Error (files-reg.c:516): `- XFail [.criu.mntns.1WVK2W/15run/mysqld/ibn9HCZI.cr.5.ghost] ghost: No such file or directory
(01.270981) Error (files-reg.c:516): `- XFail [.criu.mntns.1WVK2W/15run/mysqld/ibgtcvWr.cr.6.ghost] ghost: No such file or directory
(01.270986) Error (files-reg.c:516): `- XFail [.criu.mntns.1WVK2W/15run/mysqld/ibaiH6Ok.cr.7.ghost] ghost: No such file or directory
(01.270994) Error (files-reg.c:516): `- XFail [/var/lib/logstash-forwarder/.logstash-forwarder.cr.8.ghost] ghost: No such file or directory
(01.288279) Error (cr-restore.c:2182): Restoring FAILED.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Tycho Andersen (tycho-s)]

Hi, sorry for the delay. It looks like something in the ghost-remap engine went haywire. Do you by chance have the full log from this still? Or a way to reproduce?

[Mark Gavalda (markgavalda)]

Unfortunately don't have the full logs anymore. I will wait for the next release that contains the fix for the other bug and I will test that thoroughly though! Cheers.

[Michael Sparmann (theseven)]

I just had something very similar after statefully stopping a container (in order to do a kernel update from 4.4.0-59 to 4.4.0-62) and attempting to start it back up.

I have saved the state files and logs in case they can be of any help, please let me know how I can send those to you: ~7MB of log files and ~1GB of state data (100MB zipped), which I'd rather not post publicly, but could share with criu developers if needed.

[Tycho Andersen (tycho-s)]

They would be useful, although unfortunately our management has forbidden me to work on CRIU until we have paying customers for it, so I can't help debug it right now :(