Booting Ironic instance, neutron port remains in DOWN state

Bug #1599836 reported by Vasyl Saienko on 2016-07-07
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Unassigned
neutron
Undecided
Vasyl Saienko

Bug Description

When booting ironic instance in flat network, neutron port always remain in down state since it is not bound.

stack@vsaienko-ironic-neutron-poller:~$ neutron port-show 6cabc468-8828-4ca3-89e3-4d99a9018f03
+-----------------------+----------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+----------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | vsaienko-ironic-neutron-poller |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| created_at | 2016-07-07T07:32:05 |
| description | |
| device_id | 0a03a565-f3dd-4ad1-94e1-e1754b718ea0 |
| device_owner | compute:None |
| extra_dhcp_opts | {"opt_value": "undionly.kpxe", "ip_version": 4, "opt_name": "tag:!ipxe,bootfile-name"} |
| | {"opt_value": "10.11.0.51", "ip_version": 4, "opt_name": "tftp-server"} |
| | {"opt_value": "10.11.0.51", "ip_version": 4, "opt_name": "server-ip-address"} |
| | {"opt_value": "http://10.11.0.51:3928/boot.ipxe", "ip_version": 4, "opt_name": "tag:ipxe,bootfile-name"} |
| fixed_ips | {"subnet_id": "17ab9d45-2b7e-4d71-8bb8-f76d5edbdce0", "ip_address": "10.20.30.12"} |
| id | 6cabc468-8828-4ca3-89e3-4d99a9018f03 |
| mac_address | 52:54:00:d9:9e:d8 |
| name | |
| network_id | b5587303-c13c-4245-9cb0-04de3443b84b |
| port_security_enabled | True |
| security_groups | edd667ef-3806-47d9-b33b-1ee8af5d100d |
| status | DOWN |
| tenant_id | fd9f6a00f3a849b9bbd80ced82749c16 |
| updated_at | 2016-07-07T07:32:07 |
+-----------------------+----------------------------------------------------------------------------------------------------------+

stack@vsaienko-ironic-neutron-poller:~$ grep 6cabc468-8828-4ca3-89e3-4d99a9018f03 new/q-agt.log
2016-07-07 03:32:08.086 7799 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-79c54446-c81a-4f11-ba96-1ec2697b102c neutron -] port_update message processed for port 6cabc468-8828-4ca3-89e3-4d99a9018f03 port_update /opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:418
2016-07-07 03:32:15.215 7799 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-f1621527-14a9-4adf-830a-196e0ceb33fc admin -] port_update message processed for port 6cabc468-8828-4ca3-89e3-4d99a9018f03 port_update /opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:418

q-svc.log http://paste.openstack.org/show/526918/:

2016-07-07 03:32:06.931 7807 DEBUG neutron.db.provisioning_blocks [req-80057833-83bd-4295-8102-b9a0eb1307c3 - -] Provisioning complete for port 6cabc468-8828-4ca3-89e3-4d99a9018f03 provisioning_complete /opt/stack/neutron/neutron/db/provisioning_blocks.py:149
2016-07-07 03:32:06.959 7807 DEBUG neutron.plugins.ml2.plugin [req-80057833-83bd-4295-8102-b9a0eb1307c3 - -] Port 6cabc468-8828-4ca3-89e3-4d99a9018f03 cannot update to ACTIVE because it is not bound. _port_provisioned

The reason why it is happening is next.

1. Nova allocates networks for instance. Neutron port is created with vnic_type: normal
2. Neutron ovs agent that responds for binding this vnic type can't bind port, since de facto instance lives on dedicated hardware server.
As result port remains unbound.
3. Neutron don't move port to ACTIVE, since it is unbound.

How to fix it:

1. Create neutron port for Ironic instance with vnic_type: baremetal
2. Implement neutron driver, that will perform fake binding of baremetal port. (If network if flat just set that port is bound, no need to perform any operations, since hardware server is already plugged to this network)

Vasyl Saienko (vsaienko) on 2016-07-07
description: updated
John Garbutt (johngarbutt) wrote :

Hmm, this seems to be exposing lots of details to the end user of the API, I am curious why it works like this.

tags: added: neutron
Changed in nova:
assignee: nobody → John Garbutt (johngarbutt)
importance: Undecided → High
status: New → Triaged
John Garbutt (johngarbutt) wrote :

So I should be clear, if you need a custom VNIC_TYPE, the current solution is that you must create your port upfront, and then pass that into Nova.

What I am not sure about, is should we default to baremetal type when using ironic (somehow), and if its possible to update the vnic_type after creating the port.

John Garbutt (johngarbutt) wrote :

Hmm, seems it doesn't allow type to get updated:
https://github.com/openstack/neutron/blob/master/neutron/extensions/portbindings.py#L105

Doesn't seem like we would be able to default the port create to the correct type inside Nova (unless it was via a horrid config system that doesn't really make much sense). Ewww.

Fix proposed to branch: master
Review: https://review.openstack.org/339124

Changed in nova:
assignee: John Garbutt (johngarbutt) → Vasyl Saienko (vsaienko)
status: Triaged → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/339143

Changed in nova:
assignee: Vasyl Saienko (vsaienko) → John Garbutt (johngarbutt)
Vasyl Saienko (vsaienko) wrote :

@John, thank you for fast look and replies. I think that we should get vnic type from nova virt driver. Could you please have look at the: https://review.openstack.org/#/c/339124/ The patch is WIP, but I'm interested in your suggestions. Thanks in advance!

John Garbutt (johngarbutt) wrote :

Ah, so I have a similar patch almost ready above

John Garbutt (johngarbutt) wrote :

@vsaienko happy for you to take over the patch I started, just going to tidy up a few loose ends on it that I couldn't get done yesterday.

Vasyl Saienko (vsaienko) wrote :

@John, I have uploaded my patch earlier. I don't mind if you will drive it since your version look more finished but please add me to Co-Autors thanks :). Thanks.

John Garbutt (johngarbutt) wrote :

PS we should remove neutron and ironic from this bug, its purely a nova issue.

Vasyl Saienko (vsaienko) wrote :

It is not only Nova issue since, we need to bind baremetal port on Neutron side: https://review.openstack.org/#/c/339129/

no longer affects: ironic

Change abandoned by Vasyl Saienko (<email address hidden>) on branch: master
Review: https://review.openstack.org/339124
Reason: abandon in flavor of: https://review.openstack.org/#/c/339143/

Changed in nova:
assignee: John Garbutt (johngarbutt) → Vasyl Saienko (vsaienko)
Vasyl Saienko (vsaienko) on 2016-07-20
Changed in neutron:
assignee: nobody → Vasyl Saienko (vsaienko)
tags: added: ironic
description: updated

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/339143
Reason: This patch has been sitting unchanged for more than 12 weeks. I am therefore going to abandon it to keep the nova review queue sane. Please feel free to restore the change if you're still working on it.

Sean Dague (sdague) wrote :

No longer in progress, patch is in merge conflict

Changed in nova:
assignee: Vasyl Saienko (vsaienko) → nobody
importance: High → Medium
status: In Progress → Confirmed

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: master
Review: https://review.openstack.org/339129
Reason: Either way it's misplaced, and I don't see a reply from the author for some time. I will abandon the patch. Feel free to restore if you feel like the discussion is not closed.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers