nova-cloud-controller db sync sometimes fails in HA mode

[Alexander List]

We are deploying openstack infrastructure components (trusty/icehouse) to LXC containers on 3 physical machines provided by MAAS (full smoosh).

When deploying nova-cloud-controller, I got an error from hook shared-db-relation-changed that is caused by nova-manage db sync which tries to create a table that already exists.

My suspicion is that another instance of nova-cloud-controller already created table "instances", so a subsequent CREATE TABLE fails.

This will have to be fixed upstream by changing this to CREATE TABLE IF NOT EXISTS or even more HA awareness, to skip DB generation entirely if it exists already.

[Andreas Hasenack (ahasenack)]

I think are hitting this bug too. Attached is my log.

[Andreas Hasenack (ahasenack)]

I filed https://bugs.launchpad.net/charms/+source/nova-cloud-controller/+bug/1347245 with another backtrace I had, this might be a different bug.

[James Page (james-page)]

The db-changed hook is gated to ensure that only one unit runs the db-sync task:

@hooks.hook('shared-db-relation-changed')
@restart_on_change(restart_map())
def db_changed():
    if 'shared-db' not in CONFIGS.complete_contexts():
        log('shared-db relation incomplete. Peer not ready?')
        return
    CONFIGS.write_all()

    if eligible_leader(CLUSTER_RES):
        migrate_database()
        log('Triggering remote cloud-compute restarts.')
        [compute_joined(rid=rid, remote_restart=True)
         for rid in relation_ids('cloud-compute')]

The 'eligible_leader' function does a few checks

1) if fully clustered, then the owner of the VIP is the leader
2) if not fully clustered, the oldest service unit in the 'cluster' relation is declared the leader

If in the event of 2) the cluster relation is not fully formed with all service units prior to the shared-db relation being made, I could see how 2 nova-cc units both think they are the leader and you hit a race; do you see the db sync being run on multiple service units?

[Adam Collard (adam-collard)]

On 23 July 2014 14:01, James Page <email address hidden> wrote:

> The 'eligible_leader' function does a few checks
>
> 1) if fully clustered, then the owner of the VIP is the leader
> 2) if not fully clustered, the oldest service unit in the 'cluster'
> relation is declared the leader
>
> If in the event of 2) the cluster relation is not fully formed with all
> service units prior to the shared-db relation being made, I could see
> how 2 nova-cc units both think they are the leader and you hit a race;
> do you see the db sync being run on multiple service units?
>

Yes.

Given it's not possible for a user of the Juju API to know when a relation
is "fully formed" I'm not sure if/how we can work around this.

[James Page (james-page)]

We definitely hit this problem during original HA implementation and testing; two things where done to work-around this (pending any sort of leader election function from Juju itself):

1) juju deployer deploys service units and waits for them all to start before adding relations

2) peer relations should always fire before other relations when you introduce a new unit into an existing service

Are you waiting for all service units to fully start prior to adding relations between services?

[James Page (james-page)]

I raised bug 1228243 a while back to request this feature.

[Edward Hope-Morley (hopem)]

Just hit this with current /next charms (rev 154) - http://paste.ubuntu.com/10827860/

[Chad Smith (chad.smith)]

Just hit this one more time at Cisco. Could attach a ton of logs if necessary, but it's the same failure mode.

2015-08-28 20:22:38 INFO shared-db-relation-changed 2015-08-28 20:22:38.017 56639 CRITICAL nova [-] OperationalError: (OperationalError) (1050, "Table 'instances' already exists") "\nCREATE TABLE instances (\n\tcrea
....

2015-08-28 20:22:38 INFO shared-db-relation-changed subprocess.check_output(cmd)
2015-08-28 20:22:38 INFO shared-db-relation-changed File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
2015-08-28 20:22:38 INFO shared-db-relation-changed raise CalledProcessError(retcode, cmd, output=output)
2015-08-28 20:22:38 INFO shared-db-relation-changed subprocess.CalledProcessError: Command '['nova-manage', 'db', 'sync']' returned non-zero exit status 1

[Chad Smith (chad.smith)]

        u"cs:trusty/nova-cloud-controller-60",
        config={"openstack-origin": "cloud:trusty-icehouse",

Our juju charm settings from the above failed deployment

[Corey Bryant (corey.bryant)]

I'm going to mark this as fix released since leader elections landed in nova-cloud-controller on June 9 2015 and that should fix the issue reported in this bug. If that's not the case please feel free to open the bug back up.

commit eaaaec38ddd6b49ce8be530529b6ffaf165ba6e1
Merge: b651ebf d4b768f
Author: James Page <email address hidden>
Date: Tue Jun 9 10:59:06 2015 +0100

    Add support for leader-election