Metadata caching is fundamentally broken

Bug #1549814 reported by Sven Anderson
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Opinion
High
Unassigned

Bug Description

The caching backend for memcached/dogpile stores whole Python objects by creating a deep copy. Metadata is cached by storing the metadata Python object, and therefore the whole object tree underneath, including the "instance" attribute. Whenever the metadata is fetched, it also fetches the old version of instance with it, and works on it. This has several implications, like for example lazy fetched fields of the "instance" object are never cached, if they are not pre-fetched before caching. Also it might implicate security issues.

It would be probably a better approach to actually cache the DB query results only, and not the whole populated "living" object.

Sven Anderson (ansiwen)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/285530

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/285562

Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

@Sven Anderson:

It seems you are actively working on that [1], thanks for your efforts.
I'm switching the status to "In Progress" and set you as assignee to
signal that.
I don't yet now which prio this should have (I tend to "high"), but
let me check that with other folks in #openstack-nova.

References:
[1] https://review.openstack.org/#/q/topic:bug/1549814

Changed in nova:
status: New → In Progress
assignee: nobody → Sven Anderson (ansiwen)
importance: Undecided → High
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

This bug is suspicious, so I add the "mitaka-rc-potential" tag for potential RC blockers.

tags: added: mitaka-rc-potential
Changed in nova:
assignee: Sven Anderson (ansiwen) → Dan Smith (danms)
tags: removed: mitaka-rc-potential
Changed in nova:
assignee: Dan Smith (danms) → Diana Clarke (diana-clarke)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/285562
Reason: This code hasn't been updated in a long time, and is in merge conflict. I am going to abandon this review, but feel free to restore it if you're still working on this.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/285530
Reason: This code hasn't been updated in a long time, and is in merge conflict. I am going to abandon this review, but feel free to restore it if you're still working on this.

Changed in nova:
assignee: Diana Clarke (diana-clarke) → nobody
Robin Naundorf (senk)
Changed in nova:
status: In Progress → New
Revision history for this message
Sean Dague (sdague) wrote :

Caching models like this probably really should see a spec just to think through the access patterns, especially as people keep proposing to make MD writable from the guest, which impacts this.

Changed in nova:
status: New → Opinion
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.