Missing fields in compute.end and compute.update notifications

[Steve McLellan]

This is a follow up from some summit discussions between the searchlight and nova teams to note discrepancies between the nova API and notification payloads. The reason this causes us some issues is that at scale it becomes far simpler to treat notifications as a complete record of a servers's state rather than updating individual fields. At the present time we contact nova's API for some notification events to get that complete record.

An example API response to retrieve a server's information is at http://paste.openstack.org/show/xbv2CwtHnhhl1nLLiJeN/

An example compute.create.end notification is at http://paste.openstack.org/show/zG5aJeUpC3LAGr0J0P2T/

An example compute.update.end notification is at http://paste.openstack.org/show/uwh1izVsaW5eg7zDrgFm/

I'll break these into multiple bug reports since some of the fields pose more difficulties. The fields missing from both create and update notifications that I think will be easier to add are:

 * tags
 * description
 * key_name
 * locked
 * OS-DCF:diskConfig (some extensions are mapped to alternate field names, but this one and possibly some others are not)

I'm running devstack with it and all services from master as of Oct 28 2016, API microversion 2.35.

[Balazs Gibizer (balazs-gibizer)]

I understand that searchlight wants to get as much information in the notification as possible to avoid polling the API. However the goal to have exactly the same information in the notification than in the API answer is not easily doable right now as the notifications are coming from different nova services while the API answers always come from the nova-api service.

Anyhow adding some extra bits and pieces to the notifications should not be too hard.

There is an ongoing effort to transform the nova notifications to a well defined format [1] and I think the extension of the notification shall be done on the transformed and versioned payload.

The instance.create notification transformation is currently proposed in [2]. The additional fields can be added on top of that patch.
The instance.update notification is already transformed in [3]. An example of the versioned payload is visible in [4] so additional fields can already be proposed.

* tag: seems easy to add based on the tags field of the Instance object.
* description: I guess description is the same as Instance.display_description, so it is super easy to add
* key_name: It seems it is the name of the KeyPair attached to the instance. Do searchlight only needs the name of the KeyPair which is not unique or would it be better adding the whole KeyPair object to the notification? (user_id + key_name is unique together, and the id field of the KeyPair alone is unique)
* locked: It is Instance.locked so it is super easy
* OS-DCF:diskConfig: The related data is available in Instance.auto_disk_config but that is a boolean field and not a AUTO / MANUAL enum. Does searchlight needs the exact values in the notification? Here I feel that duplicating the enum values from the api to the notification shall be avoided and the nova object model needs to be evolved in a way that the Instance.auto_disk_config is already an enum instead of a boolean field. This is of course more work.

Bottom line, I think we need some answers from searchlight team about the exact needs but I feel that the needed fields can be added to the versioned notifications.

[1] https://blueprints.launchpad.net/nova/+spec/versioned-notification-transformation-ocata
[2] https://review.openstack.org/#/c/384621/
[3] https://review.openstack.org/#/c/325923/
[4] https://review.openstack.org/#/c/325923/31/doc/notification_samples/instance-update.json,unified
[5] https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/helpers.py#L32-L39

[Steve McLellan (sjmc7)]

Of these, i think tag, description, and key_name (we don't need the id particularly since I don't think the API returns that either) are the most important. We (the SL team) need to make decisions about how much special casing to do with notifications when they're not complete (whether we drop fields or try to merge the new data with existing, which is more expensive and complex), so it's not a case of "we must have this or nothing will work", more just identifying gaps.

[Balazs Gibizer (balazs-gibizer)]

I proposed the following bp to cover this changes https://blueprints.launchpad.net/nova/+spec/additional-notification-fields-for-searchlight

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]