Booting Ironic instance, neutron port remains in DOWN state

[Vasyl Saienko]

When booting ironic instance in flat network, neutron port always remain in down state since it is not bound.

stack@vsaienko-ironic-neutron-poller:~$ neutron port-show 6cabc468-8828-4ca3-89e3-4d99a9018f03
+-----------------------+----------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+----------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | vsaienko-ironic-neutron-poller |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| created_at | 2016-07-07T07:32:05 |
| description | |
| device_id | 0a03a565-f3dd-4ad1-94e1-e1754b718ea0 |
| device_owner | compute:None |
| extra_dhcp_opts | {"opt_value": "undionly.kpxe", "ip_version": 4, "opt_name": "tag:!ipxe,bootfile-name"} |
| | {"opt_value": "10.11.0.51", "ip_version": 4, "opt_name": "tftp-server"} |
| | {"opt_value": "10.11.0.51", "ip_version": 4, "opt_name": "server-ip-address"} |
| | {"opt_value": "http://10.11.0.51:3928/boot.ipxe", "ip_version": 4, "opt_name": "tag:ipxe,bootfile-name"} |
| fixed_ips | {"subnet_id": "17ab9d45-2b7e-4d71-8bb8-f76d5edbdce0", "ip_address": "10.20.30.12"} |
| id | 6cabc468-8828-4ca3-89e3-4d99a9018f03 |
| mac_address | 52:54:00:d9:9e:d8 |
| name | |
| network_id | b5587303-c13c-4245-9cb0-04de3443b84b |
| port_security_enabled | True |
| security_groups | edd667ef-3806-47d9-b33b-1ee8af5d100d |
| status | DOWN |
| tenant_id | fd9f6a00f3a849b9bbd80ced82749c16 |
| updated_at | 2016-07-07T07:32:07 |
+-----------------------+----------------------------------------------------------------------------------------------------------+

stack@vsaienko-ironic-neutron-poller:~$ grep 6cabc468-8828-4ca3-89e3-4d99a9018f03 new/q-agt.log
2016-07-07 03:32:08.086 7799 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-79c54446-c81a-4f11-ba96-1ec2697b102c neutron -] port_update message processed for port 6cabc468-8828-4ca3-89e3-4d99a9018f03 port_update /opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:418
2016-07-07 03:32:15.215 7799 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-f1621527-14a9-4adf-830a-196e0ceb33fc admin -] port_update message processed for port 6cabc468-8828-4ca3-89e3-4d99a9018f03 port_update /opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:418

q-svc.log http://paste.openstack.org/show/526918/:

2016-07-07 03:32:06.931 7807 DEBUG neutron.db.provisioning_blocks [req-80057833-83bd-4295-8102-b9a0eb1307c3 - -] Provisioning complete for port 6cabc468-8828-4ca3-89e3-4d99a9018f03 provisioning_complete /opt/stack/neutron/neutron/db/provisioning_blocks.py:149
2016-07-07 03:32:06.959 7807 DEBUG neutron.plugins.ml2.plugin [req-80057833-83bd-4295-8102-b9a0eb1307c3 - -] Port 6cabc468-8828-4ca3-89e3-4d99a9018f03 cannot update to ACTIVE because it is not bound. _port_provisioned

The reason why it is happening is next.

1. Nova allocates networks for instance. Neutron port is created with vnic_type: normal
2. Neutron ovs agent that responds for binding this vnic type can't bind port, since de facto instance lives on dedicated hardware server.
As result port remains unbound.
3. Neutron don't move port to ACTIVE, since it is unbound.

How to fix it:

1. Create neutron port for Ironic instance with vnic_type: baremetal
2. Implement neutron driver, that will perform fake binding of baremetal port. (If network if flat just set that port is bound, no need to perform any operations, since hardware server is already plugged to this network)

[John Garbutt (johngarbutt)]

Hmm, this seems to be exposing lots of details to the end user of the API, I am curious why it works like this.

[John Garbutt (johngarbutt)]

So I should be clear, if you need a custom VNIC_TYPE, the current solution is that you must create your port upfront, and then pass that into Nova.

What I am not sure about, is should we default to baremetal type when using ironic (somehow), and if its possible to update the vnic_type after creating the port.

[John Garbutt (johngarbutt)]

Hmm, seems it doesn't allow type to get updated:
https://github.com/openstack/neutron/blob/master/neutron/extensions/portbindings.py#L105

Doesn't seem like we would be able to default the port create to the correct type inside Nova (unless it was via a horrid config system that doesn't really make much sense). Ewww.

[John Garbutt (johngarbutt)]

oops, thats VIF_TYPE, we can update VNIC_TYPE: https://github.com/openstack/neutron/blob/master/neutron/extensions/portbindings.py#L113

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/339124

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/339129

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/339143

[Vasyl Saienko (vsaienko)]

@John, thank you for fast look and replies. I think that we should get vnic type from nova virt driver. Could you please have look at the: https://review.openstack.org/#/c/339124/ The patch is WIP, but I'm interested in your suggestions. Thanks in advance!

[John Garbutt (johngarbutt)]

Ah, so I have a similar patch almost ready above

[John Garbutt (johngarbutt)]

@vsaienko happy for you to take over the patch I started, just going to tidy up a few loose ends on it that I couldn't get done yesterday.

[Vasyl Saienko (vsaienko)]

@John, I have uploaded my patch earlier. I don't mind if you will drive it since your version look more finished but please add me to Co-Autors thanks :). Thanks.

[John Garbutt (johngarbutt)]

PS we should remove neutron and ironic from this bug, its purely a nova issue.

[Vasyl Saienko (vsaienko)]

It is not only Nova issue since, we need to bind baremetal port on Neutron side: https://review.openstack.org/#/c/339129/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vasyl Saienko (<email address hidden>) on branch: master
Review: https://review.openstack.org/339124
Reason: abandon in flavor of: https://review.openstack.org/#/c/339143/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/339143
Reason: This patch has been sitting unchanged for more than 12 weeks. I am therefore going to abandon it to keep the nova review queue sane. Please feel free to restore the change if you're still working on it.

[Sean Dague (sdague)]

No longer in progress, patch is in merge conflict

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: master
Review: https://review.openstack.org/339129
Reason: Either way it's misplaced, and I don't see a reply from the author for some time. I will abandon the patch. Feel free to restore if you feel like the discussion is not closed.

[Vladyslav Drok (vdrok)]

I don't think this affects neutron anymore, we now have the networking-baremetal project.

[Ruby Loo (rloo)]

I don't know if there is anything to be done in nova to address this. As long as the networking-baremetal plugin is used in neutron, this isn't a problem. (Is that a temporary solution or *the* solution, that is the question!) I'm going to close this for now.

We have it documented in ironic: https://docs.openstack.org/ironic/latest/admin/multitenancy.html#flat-network-interface.

[Ruby Loo (rloo)]

the fix wasn't in nova so I changed the status to invalid. No idea what it ought to be...