Savanna sometimes reports wrong cluster status in logs

[Andrew Lazarev]

If error occurs Savanna reports "Cluster status has been changed: id=%s, New status=%s" where new status is sometimes the status Savanna had before. This happens because "conductor.cluster_update" doesn't change cluster object and it is developer responsibility to update local cluster object.

[Dmitry Mescheryakov (dmitrymex)]

I knew of the problem before and I have an idea on how to fix it. Throughout the code we have the following pattern everywhere:

  cluster = conductor.cluster_update(ctx, cluster, {"status": "Adding Instances"})
  LOG.info(g.format_cluster_status(cluster))

If we just extract it to a single method of the cluster object:

  class Cluster(object)

    ...

    def update_status(self, status)
      updated_cluster = conductor.cluster_update(ctx, self, {"status": "Adding Instances"})
      LOG.info(g.format_cluster_status(updated_cluster))
      return updated_cluster

we can later reuse it everywhere and thus avoid the issue. Pretty obvious refactoring, I guess, but there is one thing: when I discussed that with Sergey Lukjanov that last time (couple month ago, I believe), he was concerned with placing code changing the object inside the object itself (or it was something else, I don't remember exactly).

So it might worth checking with Sergey before trying to implement my proposal. Anyway, if you feel you have enough time to fix it soon, feel free to assign bug to yourself.

[Andrew Lazarev (alazarev)]

I like the idea to place status change into utility method that also prints log message (instead of manual conductor call)
But I don't like idea of placing it into Cluster class.
I also don't like idea of changing Cluster instance internal state on status update.

[Dmitry Mescheryakov (dmitrymex)]

Note that with my proposal cluster object will _not_ change after update. That is one of the reasons the update_cluster method returns updated_cluster - because the 'self' is no longer a fresh representation of DB state.

[Sergey Lukjanov (slukjanov)]

Fixed in https://review.openstack.org/#/c/62971/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/62971
Committed: https://git.openstack.org/cgit/openstack/savanna/commit/?id=1c7cec7ac1b687abe2d5b9190c29982ad73af5e9
Submitter: Jenkins
Branch: master

commit 1c7cec7ac1b687abe2d5b9190c29982ad73af5e9
Author: Andrew Lazarev <email address hidden>
Date: Wed Dec 18 11:57:57 2013 -0800

    Fixed reporting about new cluster state

    Fixes bug #1262347

    Change-Id: I69f076be296070f81952517255ca8e138e417617