OpenStack Compute (nova)

Instance directory does not exist: Unable to pre-create chardev file console.log

Bug #984996 reported by Julien Danjou on 2012-04-18

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	High	Matt Riedemann	OpenStack Compute (nova) 12.0.0 "liberty"
	Kilo	Fix Released	High	Unassigned	OpenStack Compute (nova) 2015.1.0 "kilo"

Bug Description

When asking for reboot of an instance, I see the following backtrace in nova-compute.log (Nova 2012.1).

2012-04-18 12:29:33 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-04-18 12:29:33 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in _process_data
2012-04-18 12:29:33 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-04-18 12:29:33 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-04-18 12:29:33 TRACE nova.rpc.amqp return f(*args, **kw)
2012-04-18 12:29:33 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 153, in decorated_function
2012-04-18 12:29:33 TRACE nova.rpc.amqp function(self, context, instance_uuid, *args, **kwargs)
2012-04-18 12:29:33 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in decorated_function
2012-04-18 12:29:33 TRACE nova.rpc.amqp sys.exc_info())
2012-04-18 12:29:33 TRACE nova.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-04-18 12:29:33 TRACE nova.rpc.amqp self.gen.next()
2012-04-18 12:29:33 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 171, in decorated_function
2012-04-18 12:29:33 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs)
2012-04-18 12:29:33 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 898, in reboot_instance
2012-04-18 12:29:33 TRACE nova.rpc.amqp reboot_type)
2012-04-18 12:29:33 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-04-18 12:29:33 TRACE nova.rpc.amqp return f(*args, **kw)
2012-04-18 12:29:33 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 707, in reboot
2012-04-18 12:29:33 TRACE nova.rpc.amqp return self._hard_reboot(instance, network_info)
2012-04-18 12:29:33 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 762, in _hard_reboot
2012-04-18 12:29:33 TRACE nova.rpc.amqp self._create_new_domain(xml)
2012-04-18 12:29:33 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 1557, in _create_new_domain
2012-04-18 12:29:33 TRACE nova.rpc.amqp domain.createWithFlags(launch_flags)
2012-04-18 12:29:33 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/libvirt.py", line 547, in createWithFlags
2012-04-18 12:29:33 TRACE nova.rpc.amqp if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2012-04-18 12:29:33 TRACE nova.rpc.amqp libvirtError: Unable to pre-create chardev file '/var/lib/nova/instances/instance-0000047b/console.log': No such file or directory
2012-04-18 12:29:33 TRACE nova.rpc.amqp

Tags:

Revision history for this message

Chuck Short (zulcss) wrote on 2012-05-01:

Is this with kvm?

Changed in nova:
status:	New → Incomplete

Revision history for this message

David Kranz (david-kranz) wrote on 2012-05-01:

I just saw this in a reboot stress test. Using kvm and 2012.1 release.

Revision history for this message

David Kranz (david-kranz) wrote on 2012-05-01:

I also saw this:

/var/log/nova/nova-compute.log:2012-05-01 15:22:25 ERROR nova.virt.libvirt.connection [-] Getting disk size of instance-00000056: [Errno 2] No such file or directory: '/var/lib/nova/instances/instance-00000056/disk'

Revision history for this message

Julien Danjou (jdanjou) wrote on 2012-05-02:

Chuck: yes, libvirt and KVM.

Thierry Carrez (ttx) on 2012-06-07

Changed in nova:
status:	Incomplete → New

Revision history for this message

Tomasz Kłosiński (tomasz-michal-klosinski) wrote on 2012-08-10:

Download full text (4.7 KiB)

I have the same issue. After stress reboot test, nova-compute can't start because of this error:

2012-08-10 11:35:45 CRITICAL nova [-] Unable to pre-create chardev file '/var/lib/nova/instances/instance-00000049/console.log': No such file or directory
2012-08-10 11:35:45 TRACE nova Traceback (most recent call last):
2012-08-10 11:35:45 TRACE nova File "/usr/bin/nova-compute", line 49, in <module>
2012-08-10 11:35:45 TRACE nova service.wait()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 413, in wait
2012-08-10 11:35:45 TRACE nova _launcher.wait()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 131, in wait
2012-08-10 11:35:45 TRACE nova service.wait()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
2012-08-10 11:35:45 TRACE nova return self._exit_event.wait()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2012-08-10 11:35:45 TRACE nova return hubs.get_hub().switch()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2012-08-10 11:35:45 TRACE nova return self.greenlet.switch()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
2012-08-10 11:35:45 TRACE nova result = function(*args, **kwargs)
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 101, in run_server
2012-08-10 11:35:45 TRACE nova server.start()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in start
2012-08-10 11:35:45 TRACE nova self.manager.init_host()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 253, in init_host
2012-08-10 11:35:45 TRACE nova self.reboot_instance(context, instance['uuid'])
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-08-10 11:35:45 TRACE nova return f(*args, **kw)
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 159, in decorated_function
2012-08-10 11:35:45 TRACE nova function(self, context, instance_uuid, *args, **kwargs)
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 183, in decorated_function
2012-08-10 11:35:45 TRACE nova sys.exc_info())
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-08-10 11:35:45 TRACE nova self.gen.next()
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in decorated_function
2012-08-10 11:35:45 TRACE nova return function(self, context, instance_uuid, *args, **kwargs)
2012-08-10 11:35:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 904, in reboot_instance
2012-08-10 11:35:45 TRACE nova reboot_type)
2012-08-10 11:35:45 TRACE nova File "/usr/lib...

I have the same issue. After stress reboot test, nova-compute can't start because of this error:

2012-08-10 11:35:45 CRITICAL nova [-] Unable to pre-create chardev file '/var/lib/nova/instances/instance-00000049/console.log': No such file or directory
2012-08-10 11:35:45 TRACE nova Traceback (most recent call last):
2012-08-10 11:35:45 TRACE nova   File "/usr/bin/nova-compute", line 49, in <module>
2012-08-10 11:35:45 TRACE nova     service.wait()
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 413, in wait
2012-08-10 11:35:45 TRACE nova     _launcher.wait()
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 131, in wait
2012-08-10 11:35:45 TRACE nova     service.wait()
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
2012-08-10 11:35:45 TRACE nova     return self._exit_event.wait()
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2012-08-10 11:35:45 TRACE nova     return hubs.get_hub().switch()
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2012-08-10 11:35:45 TRACE nova     return self.greenlet.switch()
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
2012-08-10 11:35:45 TRACE nova     result = function(*args, **kwargs)
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 101, in run_server
2012-08-10 11:35:45 TRACE nova     server.start()
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 162, in start
2012-08-10 11:35:45 TRACE nova     self.manager.init_host()
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 253, in init_host
2012-08-10 11:35:45 TRACE nova     self.reboot_instance(context, instance['uuid'])
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-08-10 11:35:45 TRACE nova     return f(*args, **kw)
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 159, in decorated_function
2012-08-10 11:35:45 TRACE nova     function(self, context, instance_uuid, *args, **kwargs)
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 183, in decorated_function
2012-08-10 11:35:45 TRACE nova     sys.exc_info())
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-08-10 11:35:45 TRACE nova     self.gen.next()
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in decorated_function
2012-08-10 11:35:45 TRACE nova     return function(self, context, instance_uuid, *args, **kwargs)
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 904, in reboot_instance
2012-08-10 11:35:45 TRACE nova     reboot_type)
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-08-10 11:35:45 TRACE nova     return f(*args, **kw)
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 721, in reboot
2012-08-10 11:35:45 TRACE nova     if self._soft_reboot(instance):
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 757, in _soft_reboot
2012-08-10 11:35:45 TRACE nova     dom.create()
2012-08-10 11:35:45 TRACE nova   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 551, in create
2012-08-10 11:35:45 TRACE nova     if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
2012-08-10 11:35:45 TRACE nova libvirtError: Unable to pre-create chardev file '/var/lib/nova/instances/instance-00000049/console.log': No such file or directory
2012-08-10 11:35:45 TRACE nova

It seems that in nova db the instance is exists, but on the host it doesn't. Libvirt sees this instance as shut off.

root@KVM7:~# virsh list --all
 Id Name                 State
----------------------------------
  - instance-00000047    shut off
  - instance-00000048    shut off
  - instance-00000049    shut off
  - instance-00000057    shut off
  - instance-000000f9    shut off

root@KVM7:~# virsh start instance-00000049
error: Failed to start domain instance-00000049
error: Unable to pre-create chardev file '/var/lib/nova/instances/instance-00000049/console.log': No such file or directory

Is it possible to recreate the /var/lib/nova/instances/instance-00000049 somehow?

Revision history for this message

Tomasz Kłosiński (tomasz-michal-klosinski) wrote on 2012-08-10:

Here is mentioned instance record (exported to json format - for readability) form nova.instance table:

[
{
  'created_at' : '2012-07-24 15:15:12',
  'updated_at' : '2012-07-24 15:26:42',
  'deleted_at' : '2012-07-24 15:26:42',
  'deleted' : 1,
  'id' : 2,
  'internal_id' : NULL,
  'user_id' : '1f5039d64f624c9c936177d04f1bc480',
  'project_id' : 'f4ac28a96c0747008142aa0a6ba87169',
  'image_ref' : '9e4e2f95-ccaa-40e7-9079-c7d7c22b8de2',
  'kernel_id' : '',
  'ramdisk_id' : '',
  'server_name' : NULL,
  'launch_index' : 1,
  'key_name' : 'mykey',
  'key_data' : 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtg+MlE5i8BJCcjrgOAE+/NGwBi7k+nREWQnqJ9tOmZ4KJP8UeqIGm52i9fV49d1zvUnK2Ssw8RkVpei/D/9oCtyMdx+mZiFkvOzyhawPypxCmzWBgOTnW5kSnUaMzawIPa7rvBsXpBgjRecCX2X0VNPBkArzidvdGMD1Bz2Lh4Jea1rH+ieAPCG6JU9lRgnmyOBKS9HaQOoTu/YcuZy5Ic2cbQOWk9Vv8z9I8IOJEgOAa+0bOxryrJE1ndftOcqgTx2hGTzOPHKUhuNq4iE8BesQ66Wwc8Vqaf7BNONh+ksFf0qhqKCssCOxUjOD6+yWdoShTqGFZKMC42T86o9vb root@KVM7',
  'power_state' : 1,
  'vm_state' : 'deleted',
  'memory_mb' : 2048,
  'vcpus' : 1,
  'hostname' : '10',
  'host' : 'KVM7',
  'user_data' : '',
  'reservation_id' : 'r-bcv08dl8',
  'scheduled_at' : '2012-07-24 15:15:13',
  'launched_at' : '2012-07-24 15:15:30',
  'terminated_at' : '2012-07-24 15:26:42',
  'display_name' : '10',
  'display_description' : '10',
  'availability_zone' : NULL,
  'locked' : 0,
  'os_type' : NULL,
  'launched_on' : 'KVM7',
  'instance_type_id' : 5,
  'vm_mode' : NULL,
  'uuid' : '3ccd3b88-3ad3-41bf-a074-de94b191bbcb',
  'architecture' : NULL,
  'root_device_name' : '/dev/vda',
  'access_ip_v4' : NULL,
  'access_ip_v6' : NULL,
  'config_drive' : '',
  'task_state' : NULL,
  'default_ephemeral_device' : '/dev/vdb',
  'default_swap_device' : NULL,
  'progress' : 0,
  'auto_disk_config' : NULL,
  'shutdown_terminate' : 1,
  'disable_terminate' : 0,
  'root_gb' : 10,
  'ephemeral_gb' : 20,
  'cell_name' : NULL
}
]

Revision history for this message

Thierry Carrez (ttx) wrote on 2012-09-07:

Any sighting of this error in Folsom ? Looks like its due to a DB skew between libvirt and nova...

summary:	- Unable to pre-create chardev file + Instance directory does not exist: Unable to pre-create chardev file + console.log
Changed in nova:
importance:	Undecided → Medium
status:	New → Confirmed

Revision history for this message

Davanum Srinivas (DIMS) (dims-v) wrote on 2013-02-12:

is https://bugs.launchpad.net/nova/+bug/1123274 related to this?

Revision history for this message

Thang Pham (thang-pham) wrote on 2014-03-31:

Is this still a problem in the latest release of nova (icehouse or havana)?

Revision history for this message

Justin Shepherd (jshepher) wrote on 2014-12-16:

#10

Marking this bug as invalid.

1. It was filed against folsom
2. Bug has not been updated in over 8 months, since last update

Changed in nova:
status:	Confirmed → Invalid

Revision history for this message

Matt Riedemann (mriedem) wrote on 2015-04-13:

#11

We're also seeing this in the aiopcpu job during block live migration:

http://logs.openstack.org/14/172614/13/experimental/check-tempest-dsvm-aiopcpu-full/599095f/logs/10.209.0.151-subnode/screen-n-cpu.txt.gz?level=TRACE#_2015-04-11_19_32_41_959

The error was also mentioned with live migrate on ask.openstack.org:

https://ask.openstack.org/en/question/45282/issue-solvedunable-to-pre-create-chardev-file-optstackdatanovainstanceconsolelog-in-remote-compute-node/

But the solution there was shared file storage.

Changed in nova:
status:	Invalid → New
tags:	added: libvirt

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-13: Related fix proposed to nova (master)

#12

Related fix proposed to branch: master
Review: https://review.openstack.org/173042

Revision history for this message

Matt Riedemann (mriedem) wrote on 2015-04-13:

#13

Marking for kilo-rc-potential since it appears we've regressed the live migration job since 4/9, possibly with the RPC version caps.

Changed in nova:
status:	New → Confirmed
importance:	Medium → High
tags:	added: kilo-rc-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-13: Fix proposed to nova (master)

#14

Fix proposed to branch: master
Review: https://review.openstack.org/173099

Changed in nova:
assignee:	nobody → Matt Riedemann (mriedem)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-13: Fix proposed to nova (proposed/kilo)

#15

Fix proposed to branch: proposed/kilo
Review: https://review.openstack.org/173100

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-14: Related fix merged to nova (master)

#16

Reviewed: https://review.openstack.org/173042
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f25fbd6c15ba4f2a82f7cec112c2f10995dcc741
Submitter: Jenkins
Branch: master

commit f25fbd6c15ba4f2a82f7cec112c2f10995dcc741
Author: Matt Riedemann <email address hidden>
Date: Mon Apr 13 11:44:46 2015 -0700

libvirt: add debug logging to pre_live_migration

    There are several conditions checked within pre_live_migration with
    little logging, so add some debug logging to see what we're doing while
    running through this method. This focuses mainly on non-shared storage
    block migration since that's what we're currently testing in the aiopcpu
    job.

Related-Bug: #984996

Change-Id: Ia331c967e46e7d1ade42afc1ee37e6de7a246631

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-14: Fix merged to nova (master)

#17

Reviewed: https://review.openstack.org/173099
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4e515ec2269a1c3187ee9ffad3a6be059ec74b0b
Submitter: Jenkins
Branch: master

commit 4e515ec2269a1c3187ee9ffad3a6be059ec74b0b
Author: Matt Riedemann <email address hidden>
Date: Mon Apr 13 14:47:20 2015 -0700

Pass migrate_data to pre_live_migration

    Commit ebfa09fa197a1d88d1b3ab1f308232c3df7dc009 added an RPC proxy but
    as part of that was passing migrate_data=None for pre_live_migration
    which breaks live block migration when not using shared storage.

Closes-Bug: #984996

Change-Id: I2a83f1fb0e4468f9a6c67a188af725c3406139d1

Changed in nova:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-15: Fix proposed to nova (stable/kilo)

#18

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/174079

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-15: Change abandoned on nova (proposed/kilo)

#19

Change abandoned by Doug Hellmann (<email address hidden>) on branch: proposed/kilo
Review: https://review.openstack.org/173100
Reason: replaced by https://review.openstack.org/174079 in stable/kilo

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-21: Fix merged to nova (stable/kilo)

#20

Reviewed: https://review.openstack.org/174079
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=33ba90240a2ad3165274d9e54ceb156273404c9a
Submitter: Jenkins
Branch: stable/kilo

commit 33ba90240a2ad3165274d9e54ceb156273404c9a
Author: Matt Riedemann <email address hidden>
Date: Mon Apr 13 14:47:20 2015 -0700

Pass migrate_data to pre_live_migration

Closes-Bug: #984996

Change-Id: I2a83f1fb0e4468f9a6c67a188af725c3406139d1
(cherry picked from commit 4e515ec2269a1c3187ee9ffad3a6be059ec74b0b)

Thierry Carrez (ttx) on 2015-04-23

tags:

removed: kilo-rc-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-30: Fix proposed to nova (master)

#21

Fix proposed to branch: master
Review: https://review.openstack.org/179284

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-06: Fix merged to nova (master)

#22

Download full text (18.1 KiB)

Reviewed: https://review.openstack.org/179284
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5228d4e418734164ffa5ccd91d2865d9cc659c00
Submitter: Jenkins
Branch: master

commit 906ab9d6522b3559b4ad36d40dec3af20397f223
Author: He Jie Xu <email address hidden>
Date: Thu Apr 16 07:09:34 2015 +0800

Update rpc version aliases for kilo

    Update all of the rpc client API classes to include a version alias
    for the latest version implemented in Kilo. This alias is needed when
    doing rolling upgrades from Kilo to Liberty. With this in place, you can
    ensure all services only send messages that both Kilo and Liberty will
    understand.

Closes-Bug: #1444745

Conflicts:
nova/conductor/rpcapi.py

NOTE(alex_xu): The conflict is due to there are some logs already added
into the master.

Change-Id: I2952aec9aae747639aa519af55fb5fa25b8f3ab4
(cherry picked from commit 78a8b5802ca148dcf37c5651f75f2126d261266e)

commit f191a2147a21c7e50926b288768a96900cf4c629
Author: Hans Lindgren <email address hidden>
Date: Fri Apr 24 13:10:39 2015 +0200

Add security group calls missing from latest compute rpc api version bump

The recent compute rpc api version bump missed out on the security group
related calls that are part of the api.

    One possible reason is that both compute and security group client side
    rpc api:s share a single target, which is of little value and only cause
    mistakes like this.

This change eliminates future problems like this by combining them into
one to get a 1:1 relationship between client and server api:s.

    Change-Id: I9207592a87fab862c04d210450cbac47af6a3fd7
    Closes-Bug: #1448075
    (cherry picked from commit bebd00b117c68097203adc2e56e972d74254fc59)

commit a2872a9262985bd0ee2c6df4f7593947e0516406
Author: Dan Smith <email address hidden>
Date: Wed Apr 22 09:02:03 2015 -0700

Fix migrate_flavor_data() to catch instances with no instance_extra rows

    The way the query was being performed previously, we would not see any
    instances that didn't have a row in instance_extra. This could happen if
    an instance hasn't been touched for several releases, or if the data
    set is old.

    The fix is a simple change to use outerjoin instead of join. This patch
    includes a test that ensures that instances with no instance_extra rows
    are included in the migration. If we query an instance without such a
    row, we create it before doing a save on the instance.

    Closes-Bug: #1447132
    Change-Id: I2620a8a4338f5c493350f26cdba3e41f3cb28de7
    (cherry picked from commit 92714accc49e85579f406de10ef8b3b510277037)

commit e3a7b83834d1ae2064094e9613df75e3b07d77cd
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Apr 23 02:18:41 2015 +0000

Updated from global requirements

Change-Id: I5d4acd36329fe2dccb5772fed3ec55b442597150

commit 8c9b5e620eef3233677b64cd234ed2551e6aa182
Author: Divya <email address hidden>
Date: Tue Apr 21 08:26:29 2015 +0200

Control create/delete flavor api permissions using policy.json

The permissions of ...

Reviewed:  https://review.openstack.org/179284
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5228d4e418734164ffa5ccd91d2865d9cc659c00
Submitter: Jenkins
Branch:    master

commit 906ab9d6522b3559b4ad36d40dec3af20397f223
Author: He Jie Xu <hejie.xu@intel.com>
Date:   Thu Apr 16 07:09:34 2015 +0800

Update rpc version aliases for kilo
    
    Update all of the rpc client API classes to include a version alias
    for the latest version implemented in Kilo.  This alias is needed when
    doing rolling upgrades from Kilo to Liberty.  With this in place, you can
    ensure all services only send messages that both Kilo and Liberty will
    understand.
    
    Closes-Bug: #1444745
    
    Conflicts:
    	nova/conductor/rpcapi.py
    
    NOTE(alex_xu): The conflict is due to there are some logs already added
    into the master.
    
    Change-Id: I2952aec9aae747639aa519af55fb5fa25b8f3ab4
    (cherry picked from commit 78a8b5802ca148dcf37c5651f75f2126d261266e)

commit f191a2147a21c7e50926b288768a96900cf4c629
Author: Hans Lindgren <hanlind@kth.se>
Date:   Fri Apr 24 13:10:39 2015 +0200

Add security group calls missing from latest compute rpc api version bump
    
    The recent compute rpc api version bump missed out on the security group
    related calls that are part of the api.
    
    One possible reason is that both compute and security group client side
    rpc api:s share a single target, which is of little value and only cause
    mistakes like this.
    
    This change eliminates future problems like this by combining them into
    one to get a 1:1 relationship between client and server api:s.
    
    Change-Id: I9207592a87fab862c04d210450cbac47af6a3fd7
    Closes-Bug: #1448075
    (cherry picked from commit bebd00b117c68097203adc2e56e972d74254fc59)

commit a2872a9262985bd0ee2c6df4f7593947e0516406
Author: Dan Smith <dansmith@redhat.com>
Date:   Wed Apr 22 09:02:03 2015 -0700

Fix migrate_flavor_data() to catch instances with no instance_extra rows
    
    The way the query was being performed previously, we would not see any
    instances that didn't have a row in instance_extra. This could happen if
    an instance hasn't been touched for several releases, or if the data
    set is old.
    
    The fix is a simple change to use outerjoin instead of join. This patch
    includes a test that ensures that instances with no instance_extra rows
    are included in the migration. If we query an instance without such a
    row, we create it before doing a save on the instance.
    
    Closes-Bug: #1447132
    Change-Id: I2620a8a4338f5c493350f26cdba3e41f3cb28de7
    (cherry picked from commit 92714accc49e85579f406de10ef8b3b510277037)

commit e3a7b83834d1ae2064094e9613df75e3b07d77cd
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Apr 23 02:18:41 2015 +0000

Updated from global requirements
    
    Change-Id: I5d4acd36329fe2dccb5772fed3ec55b442597150

commit 8c9b5e620eef3233677b64cd234ed2551e6aa182
Author: Divya <dikonoor@in.ibm.com>
Date:   Tue Apr 21 08:26:29 2015 +0200

Control create/delete flavor api permissions using policy.json
    
    The permissions of create/delete flavor api is currently broken
    and expects the user to be always an admin, instead of controlling
    the permissions by the rules defined in the nova policy.json.
    
    Change-Id: Ide3c9ec2fa674b4fe3ea9d935cd4f7848914b82e
    Closes-Bug: 1445335
    (cherry picked from commit ced60b1d1b1608dc8229741b207a95498bc0b212)

commit bf79742d26ae66886bcdc55eeaf27e1d7ce24be5
Author: Przemyslaw Czesnowicz <przemyslaw.czesnowicz@intel.com>
Date:   Tue Apr 14 16:28:57 2015 +0100

Fix handling of pci_requests in consume_from_instance.
    
    Properly retrieve requests from pci_requests in consume_from_instance.
    Without this the call to numa_fit_instance_to_host will fail because
    it expects the request list.
    And change the order in which apply_requests and numa_fit_instance_to_host
    are called. Calling apply_requests first will remove devices from pools
    and  may make numa_fit_instance_to_host fail.
    
    Change-Id: I41cf4e8e5c1dea5f91e5261a8f5e88f46c7994ef
    Closes-bug: #1444021
    (cherry picked from commit 0913e799e9ce3138235f5ea6f80159f468ad2aaa)

commit c2d7060b480608d9773340f51d6496fadf97b667
Author: Przemyslaw Czesnowicz <przemyslaw.czesnowicz@intel.com>
Date:   Thu Apr 16 17:10:06 2015 +0100

Use list of requests in InstancePCIRequests.obj_from_db.
    
    InstancePCIRequests.obj_from_db assumes it's called with with a dict
    of values from instances_extra table, but in some cases it's called
    with just the value of pci_requests column.
    This changes obj_from_db to be used with just the value of pci_requests column.
    
    Change-Id: I7bed733c845c365081719a70b8a2f0cc9a58370c
    Closes-bug: #1445040
    (cherry picked from commit a074d7b4465b45730a5171e024c5c39a66a9c927)

commit 7a609f153808f7cee1edbbb36accc292fa8df0d0
Author: Przemyslaw Czesnowicz <przemyslaw.czesnowicz@intel.com>
Date:   Tue Apr 7 16:31:05 2015 +0100

Add numa_node field to PciDevicePool
    
    Without this field, PciDevicePool.from_dict will treat numa_node key in
    the dict as a tag, which in turn means that the scheduler client will
    drop it when converting stats to objects before reporting.
    
    Converting it back to dicts on the scheduler side thus will not have
    access to the numa_node information which would cause any requests that
    will look for the exact match between the device and instance NUMA nodes
    in the NUMATopologyFilter to fail.
    
    Closes-Bug: #1441169
    (cherry picked from commit 7db1ebc66c59205f78829d1e9cd10dcc1201d798)
    
    Conflicts:
    	nova/tests/unit/objects/test_objects.py
    
    Change-Id: I7381f909620e8e787178c0be9a362f8d3eb9ff7d

commit 880a356e40d327c0af4ce94b5a08fe0cd6fcab5d
Author: Nikola Dipanov <ndipanov@redhat.com>
Date:   Tue Apr 7 20:53:32 2015 +0100

scheduler: re-calculate NUMA on consume_from_instance
    
    This patch narrows down the race window between the filter running and
    the consumption of resources from the instance after the host has been
    chosen.
    
    It does so by re-calculating the fitted NUMA topology just before consuming it
    from the chosen host. Thus we avoid any locking, but also make sure that
    the host_state is kept as up to date as possible for concurrent
    requests, as there is no opportunity for switching threads inside a
    consume_from_instance.
    
    Several things worth noting:
      * Scheduler being lock free (and thus racy) does not really affect
      resources other than PCI and NUMA topology this badly - this is due
      to complexity of said resources. In order for scheduler decesions to not
      be based on basically guessing, in case of those two we will likely need
      to introduce either locking or special heuristics.
    
      * There is a lot of repeated code between the 'consume_from_instance'
      method and the actual filters. This situation should really be fixed but
      is out of scope for this bug fix (which is about preventing valid
      requests failing because of races in the scheduler).
    
    Change-Id: If0c7ad20506c9dddf4dec1eb64c9d6dd4fb75633
    Closes-bug: #1438238
    (cherry picked from commit d6b3156a6c89ddff9b149452df34c4b32c50b6c3)

commit a4e9a146c3993f5775501716a21632f34a63a3ad
Author: Rajesh Tailor <rajesh.tailor@nttdata.com>
Date:   Wed Apr 15 06:59:04 2015 -0700

Fix kwargs['migration'] KeyError in @errors_out_migration decorator
    
    @errors_out_migration decorator is used in the compute manager on
    resize_instance and finish_resize methods of ComputeManager class.
    It is decorated via @utils.expects_func_args('migration') to check
    'migration' is a parameter to the decorator method, however, that
    only ensures there is a migration argument, not that it's in args or
    kwargs (either is fine for what expects_func_args checks).
    The errors_out_migration decorator can get a KeyError when checking
    kwargs['migration'] and fails to set the migration status to 'error'.
    
    This fixes the KeyError in the decorator by normalizing the args/kwargs
    list into a single dict that we can pull the migration from.
    
    Change-Id: I774ac9b749b21085f4fbcafa4965a78d68eec9c7
    Closes-Bug: 1444300
    (cherry picked from commit 3add7923fc16c050d4cfaef98a87886c6b6a589c)

commit 389368bcfe498323b369f68682babb92a5b0ca54
Author: Gary Kotton <gkotton@vmware.com>
Date:   Wed Apr 15 05:14:42 2015 -0700

Resource tracker: unable to restart nova compute
    
    The resource tracker calculates its used resources. In certain cases
    of failed migrations and an instance being deleted the resource tracker
    causes an exception in nova compute. If this situation arises then nova
    compute may not even be able to restart.
    
    Change-Id: I4a154e0cae3b8e22bd59ed05ba708e07eed8dea7
    Closes-bug: #1444439
    (cherry picked from commit ee7a7446cc6947a6bacacb6cb514934cc22e5782)

commit bd6a40fecde943a3ded0124481a12c27dbb167de
Author: Andreas Jaeger <aj@suse.de>
Date:   Mon Apr 20 11:01:22 2015 +0200

Release Import of Translations from Transifex
    
    Manual import of Translations from Transifex. This change also removes
    all po files that are less than 66 per cent translated since such
    partially translated files will not help users.
    
    This updates also recreates all pot (translation source files) to
    reflect the state of the repository.
    
    This change needs to be done manually since the automatic import does
    not handle the proposed branches and we need to sync with latest
    translations.
    
    Change-Id: I0e9ef00182a2229602d23b8a67a02f0be62ee239

commit 8ebd515aa94ed399074a3b55bd36fd8cd579a499
Author: Matt Riedemann <mriedem@us.ibm.com>
Date:   Thu Apr 16 11:08:50 2015 -0700

Use kwargs from compute v4 proxy change_instance_metadata
    
    The args were passed to the compute manager method in the wrong order.
    We noticed this in the gate with KeyError: 'uuid' in the logs because of
    the LOG.debug statement in change_instance_metadata. Just use kwargs
    like rpcapi would normally.
    
    There isn't a unit test for this since the v4 proxy code goes away in
    liberty, this is for getting it into stable/kilo.
    
    Closes-Bug: #1444728
    
    Change-Id: Ic988f48d99e626ee5773c97904e09dbf00c5414a
    (cherry picked from commit e55f746ea8590cce7c2b07a023197f369251a7ef)

commit b19764d2c6a8160102a806c1d6811c4182a8bac8
Author: Matt Riedemann <mriedem@us.ibm.com>
Date:   Wed Apr 15 11:51:26 2015 -0700

compute: stop handling virt lifecycle events in cleanup_host()
    
    When rebooting a compute host, guest VMs can be getting shutdown
    automatically by the hypervisor and the virt driver is sending events to
    the compute manager to handle them. If the compute service is still up
    while this happens it will try to call the stop API to power off the
    instance and update the database to show the instance as stopped.
    
    When the compute service comes back up and events come in from the virt
    driver that the guest VMs are running, nova will see that the vm_state
    on the instance in the nova database is STOPPED and shut down the
    instance by calling the stop API (basically ignoring what the virt
    driver / hypervisor tells nova is the state of the guest VM).
    
    Alternatively, if the compute service shuts down after changing the
    intance task_state to 'powering-off' but before the stop API cast is
    complete, the instance can be in a strange vm_state/task_state
    combination that requires the admin to manually reset the task_state to
    recover the instance.
    
    Let's just try to avoid some of this mess by disconnecting the event
    handling when the compute service is shutting down like we do for
    neutron VIF plugging events. There could still be races here if the
    compute service is shutting down after the hypervisor (e.g. libvirtd),
    but this is at least a best attempt to do the mitigate the potential
    damage.
    
    Closes-Bug: #1444630
    Related-Bug: #1293480
    Related-Bug: #1408176
    
    Change-Id: I1a321371dff7933cdd11d31d9f9c2a2f850fd8d9
    (cherry picked from commit d1fb8d0fbdd6cb95c43b02f754409f1c728e8cd0)

commit 75e9de5d572578520c217b540aa2a40726f137f0
Author: Roman Podoliaka <rpodolyaka@mirantis.com>
Date:   Wed Mar 4 17:27:06 2015 +0200

Forbid booting of QCOW2 images with virtual_size > root_gb
    
    Currently, it's possible to boot an instance from a QCOW2 image,
    which has virtual_size bigger than one allowed by the given flavor
    (root_gb).
    
    The issue is caused by two different problems in the code:
    
    1) typo in get_disk_size() has made it always return None and
       effectively disabled verify_base_size() checks
    
    2) Rbd image backend skips the verify_base_size() step for
       'cached' images (the one with base files), so it is possible to
       boot an instance using a larger flavor once and then use smaller
       flavors to boot the same image, even if allowed root_gb size is
       smaller than the image virtual size
    
    Closes-Bug: #1429093
    
    Change-Id: I383130e5f8cc288f4b428ed43fe4d3aba7169473
    (cherry picked from commit c1f9ed27af64e6893d9d0153a964df5aba99b8f0)

commit 33ba90240a2ad3165274d9e54ceb156273404c9a
Author: Matt Riedemann <mriedem@us.ibm.com>
Date:   Mon Apr 13 14:47:20 2015 -0700

Pass migrate_data to pre_live_migration
    
    Commit ebfa09fa197a1d88d1b3ab1f308232c3df7dc009 added an RPC proxy but
    as part of that was passing migrate_data=None for pre_live_migration
    which breaks live block migration when not using shared storage.
    
    Closes-Bug: #984996
    
    Change-Id: I2a83f1fb0e4468f9a6c67a188af725c3406139d1
    (cherry picked from commit 4e515ec2269a1c3187ee9ffad3a6be059ec74b0b)

commit dea6116723f22632c2e478e00bb0aafcd2febdc9
Author: Timofey Durakov <tdurakov@mirantis.com>
Date:   Fri Apr 10 19:38:33 2015 +0300

Fixed order of arguments during execution live_migrate()
    
    order of arguments that passed to
    ComputeManager.live_,migration() differs in ComputeManager and
    _ComputeV4Proxy classes
    
    Change-Id: I23c25d219e9cdd0673ae6a12250219680fb7bda9
    Closes-Bug:#1442656
    (cherry picked from commit ba521fa53711774e0718808fe333aca676de57ae)

commit 22d7547c6b62fb9dabd861e4941edd34eedabfc6
Author: Doug Hellmann <doug@doughellmann.com>
Date:   Wed Apr 15 19:58:17 2015 +0000

update .gitreview for stable/kilo
    
    Change-Id: I6356513ac42b79402dbde8ee5e75cbbd1aee7eef

commit 68d6f924037f3b931add2ce5d0d433913e720ca6
Author: Ken'ichi Ohmichi <oomichi@mxs.nes.nec.co.jp>
Date:   Wed Apr 15 03:13:43 2015 +0000

Add min/max of API microversions to version API
    
    As nova-spec api-microversions, versions API needs to expose minimum
    and maximum microversions to version API, because clients need to
    know available microversions through the API. That is very important
    for the interoperability.
    This patch adds these versions as the nova-spec mentioned.
    
    Note:
      As v2(not v2.1) API change manner, we have added new extensions if
      changing API. However, this patch doesn't add a new extension even
      if adding new parameters "version" and "min_version" because version
      API is independent from both v2 and v2.1 APIs.
    
    Change-Id: Id464a07d624d0e228fe0aa66a04c8e51f292ba0c
    Closes-Bug: #1443375
    (cherry picked from commit 1830870718fe7472b47037f3331cfe59b5bdda07)
    (cherry picked from commit 853671e912c6ad9a4605acad2575417911875cdd)

commit d82d492c67db546d01addc5fead9708760fb6abd
Author: Sabari Kumar Murugesan <smurugesan@vmware.com>
Date:   Mon Apr 6 20:36:48 2015 -0700

VMware: Fix attribute error in resize
    
    The class DatastorePath was recently removed from ds_util as it's
    available in oslo.vmware. One of the reference was missed during
    the refactor.
    
    Change-Id: Idc5825c304a99e83cbf36e93751148d6f995131a
    Closes-Bug: #1440968
    (cherry picked from commit ab4a5a5300179a79f7a67688f0e9f3fc280c0efa)

commit 3cff2c673c6cdf487c2a1eb2a5c6c89c6de80d11
Author: jichenjc <jichenjc@cn.ibm.com>
Date:   Fri Mar 20 08:36:37 2015 +0800

Release bdm constraint source and dest type
    
    https://bugs.launchpad.net/nova/+bug/1377958 fixed a problem
    that source_type: image, destination_type: local is not
    supported for boot instance, exception should be raised to
    reject the param otherwise it will lead to instance become
    ERROR state.
    
    However the fix introduced a problem on nova client
    https://bugs.launchpad.net/python-novaclient/+bug/1418484
    The fix of the bug leads to following command become invalid
    
    nova boot test-vm --flavor m1.medium --image centos-vm-32
    --nic net-id=c3f40e33-d535-4217-916b-1450b8cd3987 --block-device
    id=26b7b917-2794-452a-95e5-2efb2ca6e32d,bus=sata,source=volume,bootindex=1
    
    So we need to release the original constraint to allow
    the above special case pass the validation check then
    we can revert the nova client exception
    (https://review.openstack.org/#/c/165932/)
    
    This patch checks the boot_index and whether image param is
    given after we found the bdm has source_type: image,
    destination_type: local, if this is the special case, then
    no exception will be raised.
    
    Closes-Bug: #1433609
    
    Change-Id: If43faae95169bc3864449a8364975f5c887aac14
    (cherry picked from commit cadbcc440a2fcfb8532f38111999a06557fbafc2)

commit 97145ba175291d6522afa079860c72220e43024a
Author: Dan Smith <dansmith@redhat.com>
Date:   Fri Apr 10 07:10:52 2015 -0700

Fix check_can_live_migrate_destination() in ComputeV4Proxy
    
    There was a mismatch in the V4 proxy in the call signatures of this
    function. This was missed because the "destination" parameter is passed
    in the rpcapi as the host to contact, which is consumed by the rpc
    layer and not passed. Since it was not called one of the standard
    names (either host if to be not passed, or host_param if to be passed),
    this was missed.
    
    Change-Id: Idf2160934dade650ed02b672f3b64cb26247f8e6
    Closes-Bug: #1442602
    (cherry picked from commit 0c08f7f2ef070f7c6172d7742f9789e0a8bda91a)

Thierry Carrez (ttx) on 2015-06-24

Changed in nova:
milestone:	none → liberty-1
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2015-10-15

Changed in nova:
milestone:	liberty-1 → 12.0.0

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

OpenStack Compute (nova)

Instance directory does not exist: Unable to pre-create chardev file console.log

Bug Description

Other bug subscribers

Related questions

Remote bug watches