[RFE] Ironic needs to synchronize external events with Neutron

[Adam Gandelman]

Related to https://review.openstack.org/#/c/84361/ and https://bugs.launchpad.net/ironic/+bug/1300589

Updates to Neutron resources via its API are processed asynchronously on its backend. This exposes potential races with Ironic. Example: an API request from Ironic to update a port's DHCP settings will return successfully long before the associated dnsmasq config has been updated and the server restarted. There is a small potential for a race condition where Ironic will boot a machine before its DHCP has been properly configured, especially if the machine boots very quickly (ie a local VM)

Though none are used by Ironic (yet?), other Neutron operations are dependent on some other state in Neutron. For instance, a firewall will stay PENDING until an associated router and router interface have been created and are ACTIVE.

We need a way to synchronize these events. During Icehouse, Nova solved almost identical issues regarding orchestration between Nova and Neutron via an admin API endpoint Neutron can use to post back notifications. Ironic's Neutron usage is relatively limited ATM, but providing a framework similar to Nova's for this type of orchestration would solve the current issues and allow future drivers to take advantage of other Neutron features.

[John Stafford (john-stafford)]

Hi Shraddha,

Are you still working this bug?

[Vasyl Saienko (vsaienko)]

Neutron port remains in down state for Ironic instances, we should fix it first: https://bugs.launchpad.net/neutron/+bug/1599836

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/339489

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/345963

[Ruby Loo (rloo)]

Proposed spec: https://review.openstack.org/#/c/343684/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vasyl Saienko (<email address hidden>) on branch: master
Review: https://review.openstack.org/339489

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vasyl Saienko (<email address hidden>) on branch: master
Review: https://review.openstack.org/345963

[Julia Kreger (juliaashleykreger)]

So this is still an outstanding item, and stalled on the question of "how to know when to resume".

So to start off, where are we:

1) We can get events! https://github.com/openstack/ironic/blob/master/ironic/api/controllers/v1/event.py#L107
2) People even already configure it! But when you look at the link above you feel sad.
3) We *still* have a sleep in place deep inside of the ironic's networking code.
https://github.com/openstack/ironic/blob/268b28f52782d20cd3f7bf27ead36438695b786a/ironic/dhcp/neutron.py#L160

I thought there was another sleep someplace, but we'll have to look for it.

Anyway, the issue is neutron sometimes takes time to complete binding, and we should wait some period of time for a callback from neutron.

4) So, ideally, what we could do is setup a database table to append the events to, with a periodic to delete any events older than say 15 minutes. We have some prior art here with the node history table.

5) We can then swap the sleep code around to look for an event in the new events table, and then unblock the flow once the event has been observed.

This would require an RPC change, to allow the inbound event to be submitted over RPC, as conductors are the database writers.

And the other conductor thread would just poll the events to determine what is required.

Event would remain in the table until the periodic purges it.

The sleep(s) for configuration would then be updated to be the upper bounds "how long to wait" for network configuration to complete.