"nova db sync" is racy

[Soren Hansen]

If I run "nova db sync" simultaneously on multiple hosts, it very often fails. Migrate apparently doesn't take care to avoid races like this.

I realise this is far from enough information to start fixing this bug. I'm simply reporting it so that I don't forget it.

[Armando Migliaccio (armando-migliaccio)]

I remember we had this problem too. Don't know if you already have ideas on why this happens and how to solve this, however we ended up ensuring we ran db sync on a single host and that each nova upstart script would check that the db was at the right version before starting the service (and failing do to so otherwise).

We did this to get rid of the contention altogether; unless you ensure that nova-manage db sync locks out the entire DB (or tables affected) during the migration process, I cannot see any other way to solve the issue.

Hope this help.

[Soren Hansen (soren)]

Whatever the solution, this bug seems to belong in migrate itself rather than Nova.

For now, I've also resorted to a setup where I avoid running nova db sync on all but a single host. It's not very elegant, but it does the trick.

[Dan Prince (dan-prince)]

My thought is that users should not try to 'db sync' simultaneously. I feel like we could come up with *clever* solutions to this issue but having a bunch of services all trying to update the DB at the same time seems like a really bad idea?

If you want to have a really robust/secure Nova setup you'd actually prevent most of the Nova service nodes from making DDL database requests (CREATE, ALTER, etc.). You'd then run nova-manage db sync with a DB account that has those privs from a separate node. Doing this is arguably better and would avoid the problem described in this ticket altogether.

Perhaps a good *solution* to this issue is that we update the Nova documentation with a recommendation that users (including distro packages) not run 'db sync' simultaneously?

[Yaguang Tang (heut2008)]

Totally agree with what Dan said, db sync is not expected to be executed on more than one server ,it is an admin action. As nova is in development. so it is only necessory when db schema changes . whether or not upgrade the db schema is controlled by OpenStack Cloud system admin. also, the blueprint no-db-compute is trying to removing direct accessing db for nova-compute .

the problem above is a system management issue more than a project code bug.