'vif vif-0: 16 Freezing timed out; the device may become inconsistent state' during hibernation

[Francis Ginther]

I've seen the following in syslog during some hibernation failures on xen systems on both the 4.15 xenial kernel and 5.3 bionic kernel:

Aug 26 05:44:27 ip-172-31-2-199 systemd-sleep[4223]: Suspending system...
Aug 26 05:44:27 ip-172-31-2-199 kernel: [ 269.303014] PM: hibernation entry
Aug 26 05:44:27 ip-172-31-2-199 kernel: [ 269.310721] PM: Syncing filesystems ...
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 269.334130] PM: done.
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 269.334132] Freezing user space processes ... (elapsed 0.010 seconds) done.
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 269.344657] OOM killer disabled.
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 269.344722] PM: Marking nosave pages: [mem 0x00000000-0x00000fff]
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 269.344723] PM: Marking nosave pages: [mem 0x0009e000-0x000fffff]
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 269.344725] PM: Basic memory bitmaps created
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 269.344736] PM: Preallocating image memory... done (allocated 388121 pages)
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 306.024356] PM: Allocated 1552484 kbytes in 36.67 seconds (42.33 MB/s)
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 306.024357] Freezing remaining freezable tasks ... (elapsed 2.023 seconds) done.
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 308.047913] Suspending console(s) (use no_console_suspend to debug)
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 318.236116] vif vif-0: 16 Freezing timed out; the device may become inconsistent state
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 318.238859] xenbus: freeze vif-0 failed: -16
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 318.238867] dpm_run_callback(): xenbus_dev_suspend+0x0/0xf0 returns -16
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 318.238870] PM: Device vif-0 failed to freeze: error -16
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 318.322352] PM: Basic memory bitmaps freed
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 318.322354] OOM killer enabled.
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 318.322354] Restarting tasks ... done.
Aug 26 05:45:16 ip-172-31-2-199 kernel: [ 318.329429] PM: hibernation exit
Aug 26 05:45:17 ip-172-31-2-199 systemd[1]: systemd-hibernate.service: Main process exited, code=exited, status=1/FAILURE
Aug 26 05:45:17 ip-172-31-2-199 systemd[1]: Failed to start Hibernate.
Aug 26 05:45:17 ip-172-31-2-199 systemd[1]: Dependency failed for Hibernate.
Aug 26 05:45:17 ip-172-31-2-199 systemd[1]: hibernate.target: Job hibernate.target/start failed with result 'dependency'.
Aug 26 05:45:17 ip-172-31-2-199 systemd[1]: sleep.target: Unit not needed anymore. Stopping.
Aug 26 05:45:17 ip-172-31-2-199 systemd[1]: systemd-hibernate.service: Unit entered failed state.
Aug 26 05:45:17 ip-172-31-2-199 systemd[1]: systemd-hibernate.service: Failed with result 'exit-code'.

The end result is hibernation never takes place and the instance is forced to reset after 20 minutes.