Handle flavorsync failure during nova-powervc service startup

[Arun Mani]

At times, the call to sync flavor as part of nova-powervc service seems to be blowing up causing the service to get restarted abruptly. Here's the stactrace from failure logs,

Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/opt/ibm/openstack/powervc-driver/powervc/nova/cmd/compute.py", line 20, in main
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: binary='nova-powervc')
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/usr/lib/python2.7/site-packages/nova/service.py", line 277, in create
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: db_allowed=db_allowed)
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/usr/lib/python2.7/site-packages/nova/service.py", line 148, in __init__
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: self.manager = manager_class(host=self.host, *args, **kwargs)
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/opt/ibm/openstack/powervc-driver/powervc/nova/driver/compute/manager.py", line 133, in __init__
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: self.scg_id_list).synchronize_flavors(ctx)
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/opt/ibm/openstack/powervc-driver/powervc/nova/driver/virt/powervc/sync/flavorsync.py", line 75, in synchronize_flavors
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: self._sync_flavor(ctx, flavor, response[1])
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/opt/ibm/openstack/powervc-driver/powervc/nova/driver/virt/powervc/sync/flavorsync.py", line 135, in _sync_flavor
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: self._insert_pvc_flavor_extraspecs(ctx, flavor, extra_specs)
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/opt/ibm/openstack/powervc-driver/powervc/nova/driver/virt/powervc/sync/flavorsync.py", line 170, in _insert_pvc_flavor_extraspecs
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: flavor_created = self._create_flavor(context, flavor)
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/opt/ibm/openstack/powervc-driver/powervc/nova/driver/virt/powervc/sync/flavorsync.py", line 204, in _create_flavor
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: is_public=is_public)
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/usr/lib/python2.7/site-packages/nova/compute/flavors.py", line 176, in create
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: flavor.create()
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/usr/lib/python2.7/site-packages/nova/objects/base.py", line 208, in wrapper
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: return fn(self, *args, **kwargs)
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/usr/lib/python2.7/site-packages/nova/objects/flavor.py", line 176, in create
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: db_flavor = db.flavor_create(self._context, updates, projects=projects)
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1466, in flavor_create
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: return IMPL.flavor_create(context, values, projects=projects)
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 4606, in flavor_create
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: raise exception.FlavorExists(name=values['name'])
Mar 10 22:15:09 inmd3gabzppwrctlr1 nova-powervc: nova.exception.FlavorExists: NV-40C1F9A Flavor with name PVC-VCI3-3C-4M-100D already exists.
Mar 10 22:15:10 inmd3gabzppwrctlr1 systemd: openstack-nova-powervc.service: main process exited, code=exited, status=1/FAILURE
Mar 10 22:15:10 inmd3gabzppwrctlr1 systemd: Unit openstack-nova-powervc.service entered failed state.
Mar 10 22:15:10 inmd3gabzppwrctlr1 systemd: openstack-nova-powervc.service failed.
Mar 10 22:15:10 inmd3gabzppwrctlr1 systemd: openstack-nova-powervc.service holdoff time over, scheduling restart.
Mar 10 22:15:10 inmd3gabzppwrctlr1 systemd: Stopped OpenStack PowerVC Nova Driver.
Mar 10 22:15:10 inmd3gabzppwrctlr1 systemd: Started OpenStack PowerVC Nova Driver.

This behavior cause the eventing mechanism to break leading to inconsistent failures.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/642995

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/642995
Committed: https://git.openstack.org/cgit/openstack/powervc-driver/commit/?id=6c2a535a22f2d3894bac99259dd1b355dbe3da88
Submitter: Zuul
Branch: master

commit 6c2a535a22f2d3894bac99259dd1b355dbe3da88
Author: Arun Mani <email address hidden>
Date: Wed Mar 13 05:29:04 2019 -0500

    Fix flavorsync issue during nova-powervc startup

    During nova-powervc service startup, if the flavor sync
    fails for some reasons, the code is blowing up causing
    the service to get restarted abruptly. This fix is
    to handle the same.

    Change-Id: I3ce98153d0d3c0e184fd5ae44d7f0e54c19cebe2
    Closes-Bug: #1819873
    Signed-off-by: Arun Mani <email address hidden>