trusted_filter cache does not work

[Bob Ball]

The cache in trusted_filter does not work because a new instance of TrustedFilter is made each time, which creates it's own cache.

This means that each time a new instance is scheduled, the TrustedFilter will query the attestation service for all hosts - which is a very slow operation.

[Guangya Liu (Jay Lau) (jay-lau-513)]

https://review.openstack.org/#/c/54073/6

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/64498

[Bob Ball (bob-ball)]

Why has this been changed to Invalid?

I believe this is still valid unless the logic in how the scheduler creates filters has changed?

[Bob Ball (bob-ball)]

Sorry - I've now seen the comments on https://review.openstack.org/#/c/64498/ - however, I think the point stands; the cache does not work currently.

[Bob Ball (bob-ball)]

Set back to confirmed pending the output of the comments on the review; I just don't want to forget this bug at the moment and I think there is more discussion to be had.

[Dave McCowan (dave-mccowan)]

I believe the comments indicate that the cache is working as designed. The cache saves the trust state of all of the hosts for the duration of scheduling a single VM. However, as Bob reports, the attestation server can take a long time to respond. I see delays around 10 seconds and I only have 2 hosts in my configuration. If I start 10 VMs at the same time, this adds at least 100 seconds to scheduling time. It would be a great performance boost if all VMs starting around the same time could share a cache.

[jiang, yunhong (yunhong-jiang)]

Dave, thanks for your input and your data.

Some thought:

a) I'm surprised to see 10 seconds for only 2 hosts. Which attestation server are you using?
b) I think with the local cache, it will not take 100 seconds to start VM at the same time if they are started in batch.
c) I agree that a global cache should help, I will comments on the patch.

Thanks
--jyh

[Bob Ball (bob-ball)]

I suggest we keep the discussions on here rather than the patch since the patch is abandoned and thereofre email notifcations etc seem to be disabled.

a) There was a bug with the TrustedFilter implementation (I believe now fixed) that would add both the compute and domU on XenServer hosts to the list of hosts to ask Mt Wilson about; the response from Mt Wilson took a long time although I haven't been playing with the environment recently enough to comment more on the length of time.

b) Since each scheduled VM start has it's own TrustedFilter the cache is completely useless for multi-VM starts. The filter works on class names until it gets through to the 'get_filtered_objects' method which is called once for each VM to be scheduled since we have filter_properties there; as such each class is instantiated for each VM schedule. We _always_ get a new cache because the cache is on ComputeAttestation which is initialised when TrustedFilter is initialised and there are no global singletons.

Surely since the cache has a timeout it can very safely be global since we will never trust the cache contents once the timeout has been reached.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/132229

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/132229
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c126d36640e0398e76ba01783b7f21f01f53a5f5
Submitter: Jenkins
Branch: master

commit c126d36640e0398e76ba01783b7f21f01f53a5f5
Author: Hans Lindgren <email address hidden>
Date: Thu Oct 23 19:42:35 2014 +0200

    Make scheduler filters/weighers only load once

    Right now, filters/weighers are instantiated on every invocation of the
    scheduler. This is both time consuming and unnecessary. In cases where
    a filter/weigher tries to be smart and store/cache something in between
    invocations this actually prohibits that.

    This change make base filter/weigher functions take objects instead of
    classes and then let schedulers create objects only once and then reuse
    them.

    This fixes a known bug in trusted_filter that tries to cache things.

    Related to blueprint scheduler-optimization

    Change-Id: I3174ab7968b51c43c0711033bac5d4bc30938b95
    Closes-Bug: #1223450