finalrd seems to hang, and generate incomplete shutdown-rd on some shutdowns

[Dimitri John Ledkov]

As seen in

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic/cosmic/amd64/o/open-iscsi/20180906_220028_32af0@/log.gz

         Stopping Create final runtime dir for shutdown pivot root...
[ *** ] (2 of 2) A stop job is running for …utdown pivot root (30s / 1min 30s)

and similar

[ OK ] Reached target Final Step.
         Starting Power-Off...
/shutdown: error while loading shared libraries: libdl.so.2: cannot open shared object file: No such file or directory

.....

I think things need to change.

Add dpkg triggers to be sensitive to changes in apps, and get a consistent finalrd snapshot maintained in like /var/lib/finalrd or some such.
Copy the /var/lib/finalrd into runtime dir on shutdown.

[Dimitri John Ledkov (xnox)]

Logging out of session [sid: 1, target: tgt-boot-test-34LfbG, portal: 10.1.1.2,3260]
[ 591.782918] sd-execu[1804]: /usr/lib/systemd/system-shutdown/open-iscsi.finalrd failed with exit status 141.
[ 591.828276] reboot: Power down

hmmmm

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Michael Hudson-Doyle (mwhudson)]

So I think what's happening here is that finalrd.sh is getting killed because it's running for more than 90s and so the hooks don't get to copy all needed libraries to /run/initramfs (maybe this was obvious, but it wasn't to me). The open-iscsi tests boot an image in system emulation without any kvm acceleration so it's not surprising that things are extremely slow.

I'm not sure maintaining the finalrd somewhere else via dpkg triggers and *copying* it into place would be much faster than what finalrd.sh + the two hoooks in the archive actually do (I guess you get to skip the ldconfig?). Bind-mounting it into place would be better, if that would survive the games switch_root_initramfs does.

We could probably make the tests pass reliably by adding "TimeoutStopSec=infinity" to finalrd.service and maybe we should do that anyway as it's hard to imagine any way killing finalrd.sh part way through could ever lead to happiness -- it hanging indefinitely is not any worse for the user than final shutdown panicking and also hanging indefinitely.

[Dimitri John Ledkov (xnox)]

Well. it looks like the autopkgtests have become stable since cosmic.

We did have fixes and improvements to the open-iscsi units. And i wonder, if now open-iscsi simply shuts down correctly byitself and either finalrd runs and quits quickly and everything is fine. Or maybe finalrd is not used at all. Hard to tell.

Previously, i thought that rootfs layers are getting torn down underneath finalrd hence it copied wrong deps.... but it seems fine now.

I'm just going to close this as invalid, until it becomes a problem again.