Newton -> Queens FFU upgrade: fast_forward_upgrade_playbook fails during Cinder db sync task with "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated."
TASK [Cinder db sync] *************************************************************************************************************************************************************************************************************************
fatal: [192.168.24.9]: FAILED! => {"changed": true, "cmd": ["cinder-manage", "db", "sync"], "delta": "0:00:02.538621", "end": "2018-01-18 12:26:04.920501", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-01-18 12:26:
02.381880", "stderr": "Option \"logdir\" from group \"DEFAULT\" is deprecated. Use option \"log-dir\" from group \"DEFAULT\".", "stderr_lines": ["Option \"logdir\" from group \"DEFAULT\" is deprecated. Use option \"log-dir\" from group \"D
EFAULT\"."], "stdout": "Error during database migration: There are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated.", "stdout_lines": ["Error during database migration: There
are still 1 unmigrated records in the messages table. Migration cannot continue until all records have been migrated."]}
Note: one cinder volume was created before starting the ffu upgrade procedure
The error is an internal check that we have in Cinder Pike to verify that online migrations from Ocata have been completed.
Just like with Nova you need to run online migration using cinder-manage since you won't be running the service during a FF upgrade.
We don't seem to have any online migrations in Pike, but calling the command will be a NOOP, so it's better to just run this command in Ocata and later releases after the DB sync, except on the final destination version, where we should leave them to be done online.