[pci-passthrough] nova-compute fails to start

[Joe Talerico]

Created a guest using nova with a passthrough device, shutdown that guest, and disabled nova-compute (openstack-service stop). Went to turn things back on, and nova-compute fails to start.

The trace:
2014-10-20 16:06:45.734 48553 ERROR nova.openstack.common.threadgroup [-] PCI device request ({'requests': [InstancePCIRequest(alias_name='rook',count=2,is_new=False,request_id=None,spec=[{product_id='10fb',vendor_id='8086'}])], 'code': 500}equests)s failed
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 125, in wait
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup x.wait()
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 173, in wait
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 293, in switch
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 492, in run_service
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup service.start()
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/service.py", line 181, in start
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup self.manager.pre_start_hook()
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1152, in pre_start_hook
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup self.update_available_resource(nova.context.get_admin_context())
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5949, in update_available_resource
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup rt.update_available_resource(context)
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 332, in update_available_resource
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup return self._update_available_resource(context, resources)
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 272, in inner
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup return f(*args, **kwargs)
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 349, in _update_available_resource
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup self._update_usage_from_instances(context, resources, instances)
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 708, in _update_usage_from_instances
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup self._update_usage_from_instance(context, resources, instance)
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 668, in _update_usage_from_instance
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup self.pci_tracker.update_pci_for_instance(context, instance)
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/pci/pci_manager.py", line 212, in update_pci_for_instance
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup devs = self._claim_instance(context, instance)
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/pci/pci_manager.py", line 157, in _claim_instance
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup devs = self.stats.consume_requests(pci_requests.requests)
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/pci/pci_stats.py", line 156, in consume_requests
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup raise exception.PciDeviceRequestFailed(requests=pci_requests)
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup PciDeviceRequestFailed: PCI device request ({'requests': [InstancePCIRequest(alias_name='rook',count=2,is_new=False,request_id=None,spec=[{product_id='10fb',vendor_id='8086'}])], 'code': 500}equests)s failed
2014-10-20 16:06:45.734 48553 TRACE nova.openstack.common.threadgroup

The resolution was to delete the guest via nova, then virsh undefine the guest. After the nova delete and virsh undefine, I could start nova-compute without issue.

[Dan Smith (danms)]

This prevents nova-compute from starting, seemingly just because we have PCI devices allocated. This is a pretty major issue, IMHO.

[Baodong (Robert) Li (baoli)]

Joe, I was not able to reproduce it by doing the following:

        -- start a VM with sr-iov interface
        -- suspend it with 'nova suspend' (in your bug description, you shut your vm down, but I don't see a nova shutdown command)
        -- kill the nova compute service (I don't have openstack-service command on my testbed to stop the service, thought this should be something equivalent)
        -- start the nova compute service

I repeated the above with a VM that has pci alias requests in the flavor.

In both cases, the nova compute service was restarted successfully and I was able to resume and log into the VM.

Let me know what I did wrong.

I'm running ubuntu 14.04. Is openstack-service available on Redhat?

[Baodong (Robert) Li (baoli)]

how many PCI devices did you have available for PCI passthrough on the compute node?

[Joe Talerico (jtaleric)]

Robert,
I have a couple to passthrough.`:
01:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
01:00.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
02:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
02:00.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
05:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
05:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
41:00.0 Ethernet controller [0200]: Intel Corporation Ethernet 10G 2P X520 Adapter [8086:154d] (rev 01)
41:00.1 Ethernet controller [0200]: Intel Corporation Ethernet 10G 2P X520 Adapter [8086:154d] (rev 01)

However, I passed an entire 10G nic, I was not using SRIOV.

The test rebooted the host, so you could simply reboot the host, or try stop openstack services and virsh destroy instance-000000XX to recreate

The machine I was testing on has something running on it as we speak. I was going to recreate to see if I can get more information for you.

[Joe Talerico (jtaleric)]

I cannot consistently recreate this, which is troubling. Is it due to the number of passthrough devices available?

[Dan Smith (danms)]

Yeah, that's troubling. However, there is really _no_ situation in which restarting nova-compute can be permitted to fail because of a reason like this. If we need to set the instance state to error, or even remove the PCI device forcibly, then we need to do that. Failing on startup is _not_ an option.

Do we at least have enough information from the original report to prevent whatever was happening from killing the startup of nova-compute?

[Baodong (Robert) Li (baoli)]

I am able to identify the issues in the resource tracker and pci_manager in the case of restarting the nova compute service. The issue has been there for a long time. Refer to these bugs and associated patches:

https://review.openstack.org/#/c/82206/
https://review.openstack.org/#/c/102298/

The issue is that the pci tracker (basically PCI devices on this node, their usage/stats, etc) is not initialized correctly, and when it worked, it really worked by accident.

Basically the PCI device uses compute_node_id to keep track of PCI devices available on the node. And _update_availabe_resources() gets the compute_node_id in its last step of actions. Reconciliation of PCI devices is supposed to be done during init time. However node_id is not available during init time, and DB access is not possible to get PCI devices previously on this node.

The solution to it requires further discussion. 82206 tries to defer the PCI tracker init the second time that _update_available_resources() is called (which is called periodically), which doesn't seem to be good. 102298 tries to get the compute node id during PCI tracker init. But it seems to be extra code considering _sync_compute_node() will get the node id anyway.

Also update_pci_for_instance() in pci_manager.py accepts instance that is active already, or suspended, and this is how sometimes it works during restart.

Ideally, compute node id should be available before initialize the PCI tracker or other resources on this node

[Yongli He (yongli-he)]

Joe,
i discuss this with yunhong-jiang. this patch is likely a solution.
https://review.openstack.org/#/c/102298/

after yunhong update it to new version, you might could try that.

[jiang, yunhong (yunhong-jiang)]

Hi, Robert, although the compute node_id is sure to be a bug, but it should not cause the compute service failure. Yes, it will make the status be wrong, but it should not be claim failure, because the set_hvdevs() will repopulate the pci devices information, thus the pci_manager.update_pci_for_instance() will still have devices to assign, unless something wrong in the set_hvdevs().

[jiang, yunhong (yunhong-jiang)]

Hi, Joe, can you please provide more log of the the nova compute service? Thanks.

[jiang, yunhong (yunhong-jiang)]

I mean the log from compute service to the error point. It should not be too much since it failed in early stage.

[Joe Talerico (jtaleric)]

Sure

      1 2014-10-20 13:16:35.024 28308 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __new__ /usr/lib/python2.7/site-packages/nova/servicegroup/api.py:65
      2 2014-10-20 13:16:35.207 28308 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
      3 2014-10-20 13:16:35.238 28308 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver'
      4 2014-10-20 13:16:35.305 28308 INFO oslo.messaging._drivers.impl_rabbit [req-b5d13d52-7529-4f61-ab02-eb6489445f77 ] Connecting to AMQP server on 10.16.29.115:5672
      5 2014-10-20 13:16:35.328 28308 INFO oslo.messaging._drivers.impl_rabbit [req-b5d13d52-7529-4f61-ab02-eb6489445f77 ] Connected to AMQP server on 10.16.29.115:5672
      6 2014-10-20 13:16:35.332 28308 INFO oslo.messaging._drivers.impl_rabbit [req-b5d13d52-7529-4f61-ab02-eb6489445f77 ] Connecting to AMQP server on 10.16.29.115:5672
      7 2014-10-20 13:16:35.344 28308 INFO oslo.messaging._drivers.impl_rabbit [req-b5d13d52-7529-4f61-ab02-eb6489445f77 ] Connected to AMQP server on 10.16.29.115:5672
      8 2014-10-20 13:16:35.366 28308 DEBUG nova.openstack.common.service [-] Full set of CONF: _wait_for_exit_or_signal /usr/lib/python2.7/site-packages/nova/openstack/common/service.py:167
      9 2014-10-20 13:16:35.367 28308 DEBUG nova.openstack.common.service [-] ******************************************************************************** log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1983
     10 2014-10-20 13:16:35.367 28308 DEBUG nova.openstack.common.service [-] Configuration options gathered from: log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1984
     11 2014-10-20 13:16:35.367 28308 DEBUG nova.openstack.common.service [-] command line args: [] log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1985
     12 2014-10-20 13:16:35.367 28308 DEBUG nova.openstack.common.service [-] config files: ['/usr/share/nova/nova-dist.conf', '/etc/nova/nova.conf'] log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1986
     13 2014-10-20 13:16:35.368 28308 DEBUG nova.openstack.common.service [-] ================================================================================ log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1987
     14 2014-10-20 13:16:35.368 28308 DEBUG nova.openstack.common.service [-] allow_instance_snapshots = True log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1996
     15 2014-10-20 13:16:35.368 28308 DEBUG nova.openstack.common.service [-] allow_migrate_to_same_host = False log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1996
     16 2014-10-20 13:16:35.368 28308 DEBUG nova.openstack.common.service [-] allow_resize_to_same_host = False log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1996
     17 2014-10-20 13:16:35.369 28308 DEBUG nova.openstack.common.service [-] allow_same_net_traffic = True log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1996
     18 2014-10-20 13:16:35.369 28308 DEBUG nova.openstack.common.ser...

[Baodong (Robert) Li (baoli)]

Hi, Yunhong, I agree with you that in the restart case, and given the flow of _update_available_resource() in RT, pci_manager.update_pci_for_instance() shouldn't have bailed out if it were not something wrong in pci_manager.set_hvdevs(). A possible explanation is that there was truly no SR-IOV devices available on the node at the time of restart. This could have happened if the node itself was restarted and the SR-IOV config was not persisted across node restart.

Joe, do you recall the exact steps that led to the failure? Also, was it just a trace back in the nova-compute service log or the nova-compute simply exited itself?

[Joe Talerico (jtaleric)]

Robert - This machine was rebooted by a workmate of mine - openstack services were shutdown before the reboot, but the guest was not. After the reboot, nova-compute would exit, and fail to start.

Once the PciDeviceRequestFailed error pops up, nova-compute fails and I attempt to restart it, seen here :
    530 2014-10-20 13:16:36.077 28308 TRACE nova.openstack.common.threadgroup PciDeviceRequestFailed: PCI device request ({'requests': [InstancePCIRequest(alias_name='rook',count=2,is_new=False,request_id=None,spec=[{product_id='10fb' 530 ,vendor_id='8086'}])], 'code': 500}equests)s failed
    531 2014-10-20 13:16:36.077 28308 TRACE nova.openstack.common.threadgroup
    532 2014-10-20 13:16:36.649 28936 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __new__ /usr/lib/python2.7/site-packages/nova/servicegroup/api.py:65
    533 2014-10-20 13:16:36.743 28936 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
    534 2014-10-20 13:16:36.760 28936 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver'
    535 2014-10-20 13:16:36.794 28936 INFO oslo.messaging._drivers.impl_rabbit [req-4c00e797-b170-4dd4-9f9f-bfea37f4796f ] Connecting to AMQP server on 10.16.29.115:5672
    536 2014-10-20 13:16:36.807 28936 INFO oslo.messaging._drivers.impl_rabbit [req-4c00e797-b170-4dd4-9f9f-bfea37f4796f ] Connected to AMQP server on 10.16.29.115:5672
    537 2014-10-20 13:16:36.809 28936 INFO oslo.messaging._drivers.impl_rabbit [req-4c00e797-b170-4dd4-9f9f-bfea37f4796f ] Connecting to AMQP server on 10.16.29.115:5672
    538 2014-10-20 13:16:36.817 28936 INFO oslo.messaging._drivers.impl_rabbit [req-4c00e797-b170-4dd4-9f9f-bfea37f4796f ] Connected to AMQP server on 10.16.29.115:5672
    539 2014-10-20 13:16:36.831 28936 DEBUG nova.openstack.common.service [-] Full set of CONF: _wait_for_exit_or_signal /usr/lib/python2.7/site-packages/nova/openstack/common/service.py:167
 540 2014-10-20 13:16:36.831 28936 DEBUG nova.openstack.common.service [-] ******************************************************************************** log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1983
    541 2014-10-20 13:16:36.831 28936 DEBUG nova.openstack.common.service [-] Configuration options gathered from: log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1984
    542 2014-10-20 13:16:36.832 28936 DEBUG nova.openstack.common.service [-] command line args: [] log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1985
    543 2014-10-20 13:16:36.832 28936 DEBUG nova.openstack.common.service [-] config files: ['/usr/share/nova/nova-dist.conf', '/etc/nova/nova.conf'] log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1986
    544 2014-10-20 13:16:36.832 28936 DEBUG nova.openstack.common.service [-] ================================================================================ log_opt_values /usr/lib/python2.7/site-packages/oslo/config/cfg.py:1987

[Baodong (Robert) Li (baoli)]

Joe, thanks for the log file. It's clear from the log file what had happened:

Hypervisor: assignable PCI devices: [{"dev_id": "pci_0000_05_00_0", "product_id": "10fb", "address": "0000:05:00.0", "dev_type": "type-PCI", "vendor_id": "8086", "label": "label_8086_10fb"}]

The above line indicates that only one device was assignable.

PCI device request ({'requests': [InstancePCIRequest(alias_name='rook',count=2,is_new=False,request_id=None,spec=[{product_id='10fb',vendor_id='8086'}])], 'code': 500}equests)s failed

The above inline indicates that two devices were requested. Since only one is available, thus the failure.

I think that the error log may have caused it to exit.

Can you please check how many PCI devices were assignable initially (prior to the restart)

[jiang, yunhong (yunhong-jiang)]

Joe, thanks for you log. Just a bit curios of your "openstack services were shutdown before the reboot, but the guest was not.". Do you mean when shutdown the host, the guest is still running?

[Joe Talerico (jtaleric)]

Yunhong-Jiang - openstack services were properly shut down, but my workmate did not issue a nova stop <guest>.

[jiang, yunhong (yunhong-jiang)]

Joe, thanks for your reply. Can you please check the pci devices that were assignable before restart as Robert's suggestion?

[Joe Talerico (jtaleric)]

Yunhong-Jiang - The PCI devices where the same (two intel nics with different product_ids)... However, I recently swapped out one of the NICs to have two of the same vendor & product_id.

[Yongli He (yongli-he)]

Joe,

original you have 2 nic one "product_id": "10fb", another is not. then you you have 2 10fb nic on host as you sadi, could you lspci in your host? and dump you nova.conf?
it seems there is no 2 nic with same product id, you need confirm this first.

[Joe Talerico (jtaleric)]

Previously
05:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:154d] (rev 01)
05:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:154d] (rev 01)
41:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
41:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)

Now
05:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
05:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
41:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
41:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)

[Yongli He (yongli-he)]

Hi, Joe
thanks. now we found , as boli sadi, hypervisor see only one pci device, this is wired:

1. to verify libvirt works fine, you need run:
vish nodedevs-list

lspci | grep PCI

and keep the pci address when you put the output here.

2. another thing about your env is the VM's extra spec, cause the VM already request the 2 new PCI device when try to recover compute node. so what's you configuration steps to do this?

3.
nova.conf
extra-spec of the VM
full Nova compute/schedule log is help.
could you uploading them ?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/131321

[Thierry Carrez (ttx)]

Apparently nobody is actively working on it

[Yongli He (yongli-he)]

i does work on it. wo got a +2, there are some discuss about the solution. i summarize the main focus here:

1 ) Concern : not completely resolve the problem the db init sync up problem
      Answer: that issue had another bug for that:
           https://review.openstack.org/#/c/148904/

2) retrial stats from db is not good
      wo need trust the information from the instance: which it holded, it's in use. this way, make resize simplify, make it align to vcpu/mem resource management.

 3) there might be compete opportunity: when pci reported free device, but actually there is no, same device might assign to 2 vm:

   answer:
        there is no such risk. because after reported the stats, the local pci manager allocation pool been updated. new devices will assign to new VM.

[John Garbutt (johngarbutt)]

@yongli-he so is the issue described here still a problem?

You seem to suggest there are some bits that still need work, but is this still a critical bug? we should open new bugs for any other issues.

[John Garbutt (johngarbutt)]

if you still are working on a patch, please can you link that in these comments, it seems to be missing right now.

[John Garbutt (johngarbutt)]

My bad, I see you are now working on https://review.openstack.org/#/c/131321/ again.

[John Garbutt (johngarbutt)]

reducing priority as this does not affect all users of Nova.

[Yongli He (yongli-he)]

@John Garbutt

days ago, Nikola Dipanov given good comments about the way to fix the bug. his comment been addressed, refer to the new patch:
https://review.openstack.org/#/c/131321/21

[Sylvain Bauza (sylvain-bauza)]

Given the complexity of the patch for being reviewed, I sincerely doubt it can be merged in time for RC1. I propose it for kilo-rc-potential to see if there is still an opportunity to merge it by Kilo

[John Garbutt (johngarbutt)]

Seems like we might be able to get this one in:
https://review.openstack.org/#/c/131321

But its going to be a push.

[John Garbutt (johngarbutt)]

Hmm, this might be a trigger for RC2...

[John Garbutt (johngarbutt)]

lots of comments on the patch, lets aim to get this in early next week, and backport it into RC2 if we have to

[Nikola Đipanov (ndipanov)]

So reading back through this , and then digging into the code - I _think_ most of this bug is solved by:

https://review.openstack.org/#/c/148904/

Since the pci_manager will be properly initiated after the restart with all the database information. Patch at

https://review.openstack.org/#/c/131321/

is definitely a nice to have as it will get rid of a number of edge cases I believe.

As for this bug - AFAICT according to comment #15 - it was caused by removing a device an existing instance had ownership of (would be great if Joe could confirm that this is what happened. This seems to me is caused solely by improper initialization of the pci tracker, as the tracker (once initalized properly), would actually notice the device has been removed and not attempt to claim it again (though this logic is very convoluted).

So I propose to not block Kilo on landing

https://review.openstack.org/#/c/131321/

as

https://review.openstack.org/#/c/148904/

is already in Kilo and should be sufficient to avoid nova compute crashes, unless we get a reproducer of this bug with RC1.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/173226
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6c4f6fb466cce6b02087418055ff56c41679db79
Submitter: Jenkins
Branch: stable/juno

commit 6c4f6fb466cce6b02087418055ff56c41679db79
Author: Paul Murray <email address hidden>
Date: Tue Aug 26 18:31:30 2014 +0200

    Move ComputeNode creation at init stage in ResourceTracker

    This is the first step in a series refactoring code that uses
    ResourceTracker.compute_node so that it can later be changed
    to be a ComputeNode object. Note that in this patch compute_node
    is still a dict.

    The main refactor in this patch is to move initialization of the
    compute_node property to a new method called _init_compute_node()
    That is called near the beginning of update_available_resource().

    At the moment the update methods use a method parameter called
    resources that is either the resources data structure obtained from
    the virt driver or the compute_node property, depending on how it is
    called. The result is always copied into the compute_node property.
    This change initialises the compute_node property and creates the
    compute node record as an initialisation step.

    Moving the initialization of the compute_node property to the start
    of update_available_resources() paves the way for the next patch in
    this series to use it consistently in the update methods. The
    code will then be ready to introduce the ComputeNode object.

    This patch also fixes bug 1415768

    Co-Authored-By: Sylvain Bauza <email address hidden>
    Co-Authored-By: Ed Leafe <email address hidden>

    (cherry picked from commit c3ffcd18d9fb1d999d6fa360a811b4c7fdcaba13)

    Conflicts:
     nova/compute/resource_tracker.py
     nova/tests/pci/test_pci_manager.py
     nova/tests/unit/compute/test_tracker.py

    Closes-bug #1415768
    Closes-bug #1383465

    Change-Id: Ic04af76c3835a5bf63a42163d0335d6c7e26d68a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/131321
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=58fbfdb9bf48fa18f2f7909b32d35340acd7e9dd
Submitter: Jenkins
Branch: master

commit 58fbfdb9bf48fa18f2f7909b32d35340acd7e9dd
Author: Yongli he <email address hidden>
Date: Tue Oct 28 10:04:10 2014 +0800

    resource tracker style pci resource management

    pci device resource should be managed via a method very like
    vcpu/mem. allocte device in the Claim, release them when fail
    or instance deleted.

    Change-Id: Ia17a9f6bbaec358d244d4492798869b0e0a6a8cc
    Closes-Bug: #1383465