OpenStack Compute (nova)

postgresql online_data_migrations fail for m->n

Bug #1630446 reported by Matthew Thode on 2016-10-05

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Invalid	Undecided	Matthew Thode

Bug Description

Info is in the log file attached. Will be opening another bug for possibly related failures.

I should mention that db sync went fine

See original description

Tags:

Revision history for this message

Matthew Thode (prometheanfire) wrote on 2016-10-05:

nova pg live migration log Edit (5.8 KiB, text/plain)

description:

updated

Revision history for this message

Matthew Thode (prometheanfire) wrote on 2016-10-05:

nova-pg-live-migrate-fail2.log Edit (15.5 KiB, text/plain)

Revision history for this message

Matthew Thode (prometheanfire) wrote on 2016-10-05:

Download full text (3.6 KiB)

Here's the db sync itself

WARNING: cell0 mapping not found - not syncing cell0.
2016-10-05 00:01:37.682 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 319 -> 320...
2016-10-05 00:01:37.705 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.706 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 320 -> 321...
2016-10-05 00:01:37.717 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.718 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 321 -> 322...
2016-10-05 00:01:37.728 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.729 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 322 -> 323...
2016-10-05 00:01:37.740 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.741 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 323 -> 324...
2016-10-05 00:01:37.751 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.752 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 324 -> 325...
2016-10-05 00:01:37.762 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.763 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 325 -> 326...
2016-10-05 00:01:37.773 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.774 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 326 -> 327...
2016-10-05 00:01:37.785 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.785 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 327 -> 328...
2016-10-05 00:01:37.796 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.797 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 328 -> 329...
2016-10-05 00:01:37.807 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.808 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 329 -> 330...
2016-10-05 00:01:37.905 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.906 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 330 -> 331...
2016-10-05 00:01:38.322 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:38.323 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 331 -> 332...
2016-10-05 00:01:38.428 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2...

Here's the db sync itself

WARNING: cell0 mapping not found - not syncing cell0.
2016-10-05 00:01:37.682 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 319 -> 320... 
2016-10-05 00:01:37.705 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.706 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 320 -> 321... 
2016-10-05 00:01:37.717 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.718 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 321 -> 322... 
2016-10-05 00:01:37.728 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.729 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 322 -> 323... 
2016-10-05 00:01:37.740 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.741 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 323 -> 324... 
2016-10-05 00:01:37.751 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.752 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 324 -> 325... 
2016-10-05 00:01:37.762 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.763 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 325 -> 326... 
2016-10-05 00:01:37.773 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.774 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 326 -> 327... 
2016-10-05 00:01:37.785 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.785 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 327 -> 328... 
2016-10-05 00:01:37.796 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.797 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 328 -> 329... 
2016-10-05 00:01:37.807 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.808 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 329 -> 330... 
2016-10-05 00:01:37.905 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:37.906 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 330 -> 331... 
2016-10-05 00:01:38.322 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:38.323 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 331 -> 332... 
2016-10-05 00:01:38.428 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:38.429 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 332 -> 333... 
2016-10-05 00:01:38.529 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done
2016-10-05 00:01:38.530 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] 333 -> 334... 
2016-10-05 00:01:38.635 14347 INFO migrate.versioning.api [req-3a09ef14-9430-4a4a-8cac-e2f402998a5e - - - - -] done

Revision history for this message

Roman Podoliaka (rpodolyaka) wrote on 2016-10-05:

Matthew, are you sure you also executed:

nova-manage api_db sync

These failures look like api db was not populated properly.

tags:	added: db
Changed in nova:
status:	New → Incomplete
assignee:	nobody → Matthew Thode (prometheanfire)

Revision history for this message

Sylvain Bauza (sylvain-bauza) wrote on 2016-10-05:

Now that Nova uses 2 separate databases (since Mitaka, see relnotes http://docs.openstack.org/releasenotes/nova/mitaka.html), you need to populate both the "legacy" DB (ie. the existing one) *and* the API DB (a new model) in order to get all the incremental updates.

#1 nova-manage db sync && nova-manage api_db sync
#2 nova-manage db online_data_migrations

Changed in nova:
status:	Incomplete → Invalid

Revision history for this message

Matthew Thode (prometheanfire) wrote on 2016-10-05:

yep, forgot to run the api_db upgrade, thanks for pointing it out

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-10-05:

So is this invalid then?

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-10-05:

Ah I see you marked it already, thanks.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.