[RFE] Expose trunk details over metadata API

[Bence Romsics]

Enable bringup of subports via exposing trunk/subport details over
the metadata API

With the completion of the trunk port feature in Newton (Neutron
bp/vlan-aware-vms [1]), trunk and subports are now available. But the
bringup of the subports' VLAN interfaces inside an instance is not
automatic. In Newton there's no easy way to pass information about
the subports to the guest operating system. But using the metadata
API we can change this.

Problem Description
-------------------

To bring up (and/or tear down) a subport the guest OS

(a) must know the segmentation-type and segmentation-id of a subport
    as set in 'openstack network trunk create/set --subport'

(b) must know the MAC address of a subport
    as set in 'openstack port create'

(c) must know which vNIC the subport belongs to

(d) may need to know when were subports added or removed
    (if they are added or removed during the lifetime of an instance)

Since subports do not have a corresponding vNIC, the approach used
for regular ports (with a vNIC) cannot work.

This write-up addresses problems (a), (b) and (c), but not (d).

Proposed Change
---------------

Here we propose a change involving both Nova and Neutron to expose
the information needed via the metadata API.

Information covering (a) and (b) is already available (read-only)
in the 'trunk_details' attribute of the trunk parent port (ie. the
port which the instance was booted with). [2]

We propose to use the MAC address of the trunk parent port to cover
(c). We recognize this may occasionally be problematic, because MAC
addresses (of ports belonging to different neutron networks) are not
guaranteed to be unique, therefore collision may happen. But this seems
to be a small price for avoiding the complexity of other solutions.

The mechanism would be the following. Let's suppose we have port0
which is a trunk parent port and instance0 was booted with '--nic
port-id=port0'. On every update of port0's trunk_details Neutron
constructs the following JSON structure:

PORT0-DETAILS = {
    "mac_address": PORT0-MAC-ADDRESS, "trunk_details":
    PORT0-TRUNK-DETAILS
}

Then Neutron sets a metadata key-value pair of instance0, equivalent
to the following nova command:

nova meta set instance0 trunk_details::PORT0-MAC-ADDRESS=PORT0-DETAILS

Nova in Newton limits meta values to <= 255 characters, this limit
must be raised. Assuming the current format of trunk_details roughly
150 characters/subport are needed. Alternatively meta values could
have unlimited length - at least for the service tenant used by
Neutron. (Though tenant-specific API validators may not be a good
idea.) The 'values' column of the the 'instance_metadata' table should
be altered from VARCHAR(255) to TEXT() in a Nova DB migration.
(A slightly related bug report: [3])

A program could read
http://169.254.169.254/openstack/2016-06-30/meta_data.json and
bring up the subport VLAN interfaces accordingly. This program is
not covered here, however it is worth pointing out that it could be
called by cloud-init.

Alternatives
------------

(1) The MAC address of a parent port can be reused for all its child
ports (when creating the child ports). Then VLAN subinterfaces
of a network interface will have the correct MAC address by
default. Segmentation type and ID can be shared in other ways, for
example as a VLAN plan embedded into a golden image. This approach
could even partially solve problem (d), however it cannot solve problem
(a) in the dynamic case. Use of this approach is currently blocked
by an openvswitch firewall driver bug. [4][5]

(2) Generate and inject a subport bringup script into the instance
as user data. Cannot handle subports added or removed after VM boot.

(3) An alternative solution to problem (c) could rely on the
preservation of ordering between NICs passed to nova boot and NICs
inside an instance. However this would turn the update of trunk_details
into an instance-level operation instead of the port-level operation
proposed here. Plus it would fail if this ordering is ever lost.

References
----------

[1] https://blueprints.launchpad.net/neutron/+spec/vlan-aware-vms
[2] https://review.openstack.org/#q,Id23ce8fc16c6ea6a405cb8febf8470a5bf3bcb43,n,z
[3] https://bugs.launchpad.net/nova/+bug/1117923
[4] https://bugs.launchpad.net/neutron/+bug/1626010
[5] https://bugs.launchpad.net/neutron/+bug/1593760

[Armando Migliaccio (armando-migliaccio)]

I either don't understand your proposal or I think you already have all you need in the trunk_details extension. Neutron is not involved in how metadata is exposed to the guest, and it should not be. I suspect there is nothing to do on the Neutron side, but please elaborate in case I am missing something.

When you fetch a parent port [1], what information is missing that you need to get hold of?

[1] https://review.openstack.org/#/c/353131/10/api-ref/source/v2/samples/trunks/trunk-details-show-response.json

[Armando Migliaccio (armando-migliaccio)]

A user can choose the MACs for all the ports in the trunk the way he/she sees fit. There is no good reason to move PORT0-MAC-ADDRESS into the trunk-details attribute when it's available as resource's attribute on the parent port. It creates needless duplication.

[Armando Migliaccio (armando-migliaccio)]

One more thing:

You say, and I quote:

To bring up (and/or tear down) a subport the guest OS "(c) must know which vNIC the subport belongs to".

Assumed a VM is booted with multiple virtual NICs (vNIC_1, ..., vNIC_N), each connected to a trunk (Trunk_1, ..., Trunk_N), a consumer of the trunk_details extension has still access to all the information required to assemble a configuration layout for guest consumption and setup. So where exactly do you see Neutron having a gap to be filled?

[Bence Romsics (bence-romsics)]

Armando, thanks for the quick reaction. Yes, the information is there in trunk_details, but how does the guest OS access it? Usually the guest does not have direct access to Neutron API. That's why we have metadata.

My first idea was that it is nova's responsibility to collect the data that will be exposed in metadata.json. Started thinking about the details. Nova grabs trunk_details from the parent port. But if a subport is added or removed then nova needs to know if trunk_details was updated. So neutron needs to send an os-external notification to nova about updates to trunk_details.

But if neutron already has to send something to nova isn't it easier cut the number of turnarounds and set a meta value directly from neutron?

So I guess, I'm kind of debating if neutron should or should not be involved in exposing metadata. Does that make sense?

[Bence Romsics (bence-romsics)]

In reply to #2: I do not want to duplicate PORT0-MAC-ADDRESS in port0's trunk_details attribute. I would only want to include it when that information is propagated into a nova meta field.

[Armando Migliaccio (armando-migliaccio)]

There are a couple of issues right now:

a) If a trunk dynamically changes after a consumer of the Neutron API access trunk-details, the consumer won't know unless it polls.

b) From within the guest there is no good way of knowing whether a trunk is dynamically changing after boot. Mind you, it's doable but it's not clean.

While I think a) may be worth addressing in the form of a os-external notification (which should not require an RFE), I don't see how Neutron should be in the business of dealing with b). Shortcutting components together is usually a manifestation of tight coupling, and no-one wants that.

[Armando Migliaccio (armando-migliaccio)]

Bence: thanks for clearing this up.

I feel that the initial problem statement was not well formulated. I tried to reformulate it in #6. Please let me know if this is what you really intended.

I took the liberty of marking this as WONTFIX in the meantime because I suspect that we can only reasonably tackle a), and that does not need an RFE (in fact, I had it on my backlog ;))

[Bence Romsics (bence-romsics)]

Correct me if I'm reading it wrong, but I think your point (b) is the same as problem (d) in the original post. Which is explicitly and intentionally not addressed here. I'm not arguing for that.

I find your point (a) obviously true. However if we don't expose the (up-to-date) trunk_details over the metadata API then the guest OS does not even have the chance to poll. I'm a bit baffled: Why did we add the option to the trunk port feature to add/remove subports to a running vm, if we don't make the up-to-date information required to bring up a subport available where it is needed?

Are you arguing against the use case and/or against the implementation proposal?

If you accept the use case, then either nova has to be a neutron client reading trunk_details, or neutron has to be a nova client setting some meta-like field. None of these couplings are new. Nova is a heavy neutron client, but AFAICT neutron is also a nova client already when it uses os-external notifications. So I don't see how this proposal would make the nova-neutron coupling stronger than it is now.

[Armando Migliaccio (armando-migliaccio)]

> Correct me if I'm reading it wrong, but I think your point (b) is the same as problem (d) in the original post. Which is explicitly and intentionally not addressed here. I'm not arguing for that.

If you leave your (d) my (b) aside, then this report should be solely about ensuring my point (a). And the title and the whole description does not make that clear.

As I said in #6, I think that (a) is worth addressing, but we should be able to leverage the existing os-external notifications for port updates. Having said that, it is not clear at this stage how Nova is/will be consuming the trunk-details extension, and thus I have not only left this aspect willingly unimplemented, but I think that we should wait for Nova to consume the extension first, before we can consider optimizing the experience.

[Armando Migliaccio (armando-migliaccio)]

My suggestion for you would be to revise/simplify the problem statement, but I feel that until we get to close the circle on the Nova side, there's hardly anything we can do on the Neutron side.

[Armando Migliaccio (armando-migliaccio)]

Putting back to Incomplete now that the problem statement is a little more clear.

[Bence Romsics (bence-romsics)]

Okay, I'm starting to see your approach. Just to double-check, it would be neutron sending notifications to nova, in reaction to that nova consuming the trunk_details attribute.

I'm not against that. I actually started with that way of thinking but ended up trying to optimize it away to reduce the back and forth between nova and neutron. That state is represented in my original proposal. The optimization may have been premature.

I also haven't went down that lane, because I was thinking neutron is not sending os-external notifications for all updates of the parent port's trunk_details. So nova would see some changes in trunk_details, but missing others. But this is not something I have checked in the code yet. Is this correct?

This way of thinking raises other questions for me too. Do we need to increase the number of notifications sent? Do we need a single port_update type of notification, or do we need to introduce a trunk_details_update type of notification?

Regarding the process I would like to avoid waiting for nova to do something in itself. Previously when we did not have a joint discussion of features affecting both nova and neutron we have burned ourselves and had to start from scratch again.

So nova folks please join and share your opinion!

[Armando Migliaccio (armando-migliaccio)]

The problem of alerting a consumer such as Nova of resource changes on the Neutron side (whether it's for example the port status, fixed or floating IPs, MAC, etc), is well known and established. All I am saying is that trunk_details can be considered as yet another element that Nova might want to be aware of and be alerted of, and therefore we could augment the existing nova notification logic (on the Neutron side) to make sure that changes in trunk details trigger a notification that Nova may want to care about. In fact, we do this already for port status and floating IPs.

Having said that, I think that a precondition to talking about emitting notifications on trunk_details changes is the use of trunk details from a consumer like Nova, and guests respectively. Until then, I see no point in putting time and effort into this from a Neutron point of view.

[Armando Migliaccio (armando-migliaccio)]

My suggestion is to first focus on figuring out how trunk_details is consumed by Nova and guests as if there were immutable. Making sure that this information is current and changes to it are propagated to the interested parties is something that can be followed up once the first element is addressed.

[Bence Romsics (bence-romsics)]

Understood. One more thought from me before I turn in for today, it's getting late here. AFAICT the existing places where nova is a client to neutron are about pieces of information about which nova actually cares. For example when to resume the paused vm right after boot. However in the use case discussed here I tend to think about nova(-metadata) as just the conduit between neutron and the guest OS. I'm not sure if this is a good way of thinking about this use case, but it leads me to a question. Do we have another use case for nova not just to pass trunk_details from one party to another, but actually look at it?

[Armando Migliaccio (armando-migliaccio)]

I am not sure I am well positioned to answer question #15 right now, but your comment reminded me that another party interested in trunk details might as well be Ironic.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]